diff options
Diffstat (limited to 'doc')
573 files changed, 14557 insertions, 7152 deletions
diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index 84d1bab3cc6..7b20887f53f 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -11,6 +11,7 @@ link: https://about.gitlab.com/handbook/communication/#top-misused-terms level: warning ignorecase: true swap: + air(?:-| )?gapped: offline environment click: select code base: codebase config: configuration diff --git a/doc/.vale/gitlab/Uppercase.yml b/doc/.vale/gitlab/Uppercase.yml index f1b06e10fe6..6093565fd7c 100644 --- a/doc/.vale/gitlab/Uppercase.yml +++ b/doc/.vale/gitlab/Uppercase.yml @@ -22,6 +22,7 @@ exceptions: - ARN - ASCII - AWS + - BMP - BSD - CAS - CDN @@ -69,6 +70,7 @@ exceptions: - GID - GIF - GKE + - GLFM - GNU - GPG - GPL @@ -85,6 +87,7 @@ exceptions: - IAM - IANA - IBM + - ICO - IDE - IID - IMAP @@ -124,9 +127,9 @@ exceptions: - PEM - PEP - PGP + - PHP - PID - PKCS - - PHP - PNG - POSIX - POST @@ -136,6 +139,7 @@ exceptions: - RBAC - RDP - RDS + - RDS - REST - RFC - RHEL @@ -143,7 +147,6 @@ exceptions: - RPM - RPS - RSA - - RDS - RSS - RTC - RVM @@ -161,6 +164,7 @@ exceptions: - SFTP - SHA - SLA + - SLI - SMS - SMTP - SOC @@ -168,6 +172,7 @@ exceptions: - SPDX - SPF - SQL + - SRE - SSD - SSG - SSH @@ -185,9 +190,9 @@ exceptions: - TOML - TOTP - TTL - - UID - UDP - UID + - UID - UNIX - URI - URL @@ -198,6 +203,7 @@ exceptions: - VCS - VPC - VPN + - WEBP - WIP - WSL - XML diff --git a/doc/.vale/gitlab/Wordy.yml b/doc/.vale/gitlab/Wordy.yml index 716fdc6c38d..7888d16dadb 100644 --- a/doc/.vale/gitlab/Wordy.yml +++ b/doc/.vale/gitlab/Wordy.yml @@ -14,3 +14,4 @@ swap: needs? to: "Rewrite the sentence, or use 'must', instead of" note that: "Be concise: rewrite the sentence to not use" please: "Remove this word from the sentence: " + respectively: "Rewrite the sentence to be more precise, instead of using " diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index 6dc0704963b..76e8f99b5db 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -90,6 +90,7 @@ Caddy callstack callstacks Camo +canonicalization canonicalized captcha CentOS @@ -517,6 +518,7 @@ reinitialize reinitializing relicensing remediations +renderers replicables repmgr repmgrd @@ -524,6 +526,7 @@ repurposing requeue requeued requeues +requeuing Restlet resync resynced @@ -576,7 +579,6 @@ severities sharded sharding shfmt -Shibboleth Shimo Shopify Sidekiq @@ -594,6 +596,7 @@ Sobelow Solargraph Solarized Sourcegraph +spammable sparkline sparklines spidering @@ -645,6 +648,8 @@ subtrees sudo supercookie supercookies +superset +supersets supertype supertypes swappiness diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index ca59dff7ef4..6cf630ad079 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -23,11 +23,11 @@ GitLab can stream a single event more than once to the same destination. Use the ## Add a new event streaming destination WARNING: -Event streaming destinations will receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. +Event streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. -### Add event streaming destination using GitLab UI +### Use the GitLab UI -Users with at least the Owner role of a group can add event streaming destinations for it: +Users with at least the Owner role for a group can add event streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** @@ -41,10 +41,10 @@ Event streaming is enabled if: - No warning is shown. - The added endpoint is displayed in the UI. -### Add event streaming destination using the API +### Use the API -To enable event streaming, a group owner must add a new event streaming destination using the `externalAuditEventDestinationCreate` mutation -in the GraphQL API. +To enable event streaming and add a destination, users with at least the Owner role for a group must use the +`externalAuditEventDestinationCreate` mutation in the GraphQL API. ```graphql mutation { @@ -66,19 +66,24 @@ Event streaming is enabled if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## List currently enabled streaming destinations +## List streaming destinations -### List currently enabled streaming destination using GitLab UI +Users with at least the Owner role for a group can list event streaming destinations. -Users with at least the Owner role of a group can list event streaming destinations: +### Use the GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. + +Users with at least the Owner role for a group can list event streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** 1. On the main area, select **Streams** tab. -### List currently enabled streaming destinations using the API +### Use the API -Group owners can view a list of event streaming destinations at any time using the `externalAuditEventDestinations` query type. +Users with at least the Owner role for a group can view a list of event streaming destinations at any time using the +`externalAuditEventDestinations` query type. ```graphql query { @@ -97,27 +102,34 @@ query { If the resulting list is empty, then audit event streaming is not enabled for that group. -## Delete currently enabled streaming destination +## Delete streaming destinations -Group Owners can delete event streaming destinations at any time using the `deleteAuditEventDestinations` mutation type. +Users with at least the Owner role for a group can delete event streaming destinations using the +`deleteAuditEventDestinations` mutation type. + +When the last destination is successfully deleted, event streaming is disabled for the group. -### Delete currently enabled streaming using GitLab UI +### Use the GitLab UI -Uses with at least the Owner role of a group can delete event streaming destination. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. + +Users with at least the Owner role for a group can delete event streaming destinations. 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** 1. On the main area, select **Streams** tab. 1. Select **{remove}** at the right side of each item. -The external streaming destination is delete when: +The external streaming destination is deleted when: - No warning is shown. - The deleted endpoint is not displayed in the UI. -### Delete currently enabled streaming using the API +### Use the API -You can delete an event streaming destination by specifying an ID. Get the required ID by [listing the details](audit_event_streaming.md#list-currently-enabled-streaming-destinations-using-the-api) of event streaming destinations. +Delete an event streaming destination by specifying an ID. Get the required ID by +[listing the details](audit_event_streaming.md#use-the-api-1) of event +streaming destinations. ```graphql @@ -134,8 +146,6 @@ Destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. -When the last destination is successfully deleted, event streaming is disabled for the group. - ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -144,7 +154,7 @@ Each streaming destination has a unique verification token (`verificationToken`) token is generated when the event destination is created and cannot be changed. Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against -the destination's value when [listing streaming destinations](#list-currently-enabled-streaming-destinations). +the destination's value when [listing streaming destinations](#list-streaming-destinations). ## Audit event streaming on Git operations @@ -155,7 +165,7 @@ On self-managed GitLab, by default this feature is not available. To make it ava Streaming audit events can be sent when signed-in users push or pull a project's remote Git repositories: -- [Using SSH](../ssh/index.md). +- [Using SSH](../user/ssh.md). - Using HTTP or HTTPS. - Using the **Download** button (**{download}**) in GitLab UI. @@ -163,9 +173,9 @@ Audit events are not captured for users that are not signed in. For example, whe To configure streaming audit events for Git operations, see [Add a new event streaming destination](#add-a-new-event-streaming-destination). -### Request headers +### Headers -Request headers are formatted as follows: +Headers are formatted as follows: ```plaintext POST /logs HTTP/1.1 @@ -174,7 +184,7 @@ Content-Type: application/x-www-form-urlencoded X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> ``` -### Example responses for SSH events +### Example payloads for SSH events Fetch: @@ -236,7 +246,7 @@ Push: } ``` -### Example responses for HTTP and HTTPS events +### Example payloads for HTTP and HTTPS events Fetch: @@ -298,7 +308,7 @@ Push: } ``` -### Example responses for events from GitLab UI download button +### Example payloads for events from GitLab UI download button Fetch: @@ -333,9 +343,9 @@ Fetch: Stream audit events that relate to merge approval actions performed within a project. -### Request headers +### Headers -Request headers are formatted as follows: +Headers are formatted as follows: ```plaintext POST /logs HTTP/1.1 @@ -344,7 +354,7 @@ Content-Type: application/x-www-form-urlencoded X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> ``` -### Example request body +### Example payload ```json { diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 7c89f34ce61..b3304fd1cbd 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -61,8 +61,7 @@ To create an Auditor user: 1. On the left sidebar, select **Overview > Users**. 1. Create a new user or edit an existing one, and in the **Access** section select Auditor. -1. Select **Create user** or **Save changes** if you created a new user or - edited an existing one respectively. +1. If you created a user, select **Create user**. For an existing user, select **Save changes**. To revoke Auditor permissions from a user, make them a Regular user by following the previous steps. diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index d09344629e2..fca4f163075 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -31,7 +31,6 @@ providers: - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** -- [Shibboleth](../../integration/saml.md) - [Smartcard](smartcard.md) **(PREMIUM SELF)** - [Twitter](../../integration/twitter.md) diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 2a396c4d53a..a7e070b755a 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -518,7 +518,7 @@ GitLab doesn't support TLS client authentication. Complete these steps on your L The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be authenticated with the TLS protocol. -## Deleting users +## Users deleted from LDAP Users deleted from the LDAP server: @@ -531,6 +531,16 @@ However, these users can continue to use Git with SSH until the next time the To delete the account immediately, you can manually [block the user](../../../user/admin_area/moderate_users.md#block-a-user). +## Updating user email addresses + +Email addresses on the LDAP server are considered the source of truth for users when LDAP is used to sign in. Updating user email +addresses must be done on the LDAP server that manages the user. The email address for GitLab is updated either: + +- When the user next signs in. +- When the next [user sync](ldap_synchronization.md#user-sync) is run. + +The updated user's previous email address becomes the secondary email address to preserve that user's commit history. + ## Google Secure LDAP > Introduced in GitLab 11.9. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index b8391bec72f..5c5d5aaffe8 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -71,7 +71,28 @@ options = { filter: Net::LDAP::Filter.eq('cn', '*'), # :attributes is optional - # the attributes we want to get returnedk + # the attributes we want to get returned + attributes: %w(dn cn memberuid member submember uniquemember memberof) +} +adapter.ldap_search(options) +``` + +When using OIDs in the filter, replace `Net::LDAP::Filter.eq` with `Net::LDAP::Filter.construct`: + +```ruby +adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') +options = { + # :base is required + # use .base or .group_base + base: adapter.config.base, + + # :filter is optional + # This filter includes OID 1.2.840.113556.1.4.1941 + # It will search for all direct and nested members of the group gitlab_grp in the LDAP directory + filter: Net::LDAP::Filter.construct("(memberOf:1.2.840.113556.1.4.1941:=CN=gitlab_grp,DC=example,DC=com)"), + + # :attributes is optional + # the attributes we want to get returned attributes: %w(dn cn memberuid member submember uniquemember memberof) } adapter.ldap_search(options) diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index a099b9c76f8..561cbd1b3ae 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -471,7 +471,7 @@ For your app, complete the following steps on Casdoor: 1. Add your GitLab redirect URL. For example, if your GitLab domain is `gitlab.example.com`, ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab/) for more details. Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index e5c371b9d40..82bb1a35e02 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -116,7 +116,7 @@ If you get the following error message: The project specified by the manifest (`root/kas-manifest001`) doesn't exist or the project where the manifest is kept is private. To fix this issue, -ensure the project path is correct and that the project's visibility is [set to public](../../public_access/public_access.md). +ensure the project path is correct and that the project's visibility is [set to public](../../user/public_access.md). ### Configuration file not found diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 7c3f36c71c2..99c291379f1 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -82,7 +82,7 @@ These features can help provide visibility into GitLab and audit what is happeni These features can also help with compliance requirements: -- [**Email all users of a project, group, or entire server**](../tools/email.md) +- [**Email all users of a project, group, or entire server**](../user/admin_area/email_from_gitlab.md) (for instances): An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index e041f1e11c6..4ddca58c774 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -15,12 +15,12 @@ without modifying the GitLab source code. A file hook runs on each event. You can filter events or projects in a file hook's code, and create many file hooks as you need. Each file hook is triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks](../system_hooks/system_hooks.md) documentation. +see the [system hooks](system_hooks.md) documentation. NOTE: File hooks must be configured on the file system of the GitLab server. Only GitLab server administrators can complete these tasks. Explore -[system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) +[system hooks](system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have file system access. Instead of writing and supporting your own file hook, you can also make changes @@ -52,7 +52,7 @@ Follow the steps below to set up a custom hook: language type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. 1. The data to the file hook is provided as JSON on STDIN. It is exactly the - same as for [system hooks](../system_hooks/system_hooks.md). + same as for [system hooks](system_hooks.md). That's it! Assuming the file hook code is properly implemented, the hook fires as appropriate. The file hooks file list is updated for each event, there is no diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 939b4aec968..2a21bda23d7 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -42,6 +42,38 @@ Repository-centric strategies for using `rsync` effectively can be found in the [moving repositories](../../operations/moving_repositories.md) documentation; these strategies can be adapted for use with any other file-based data, such as [GitLab Pages](../../pages/index.md#change-storage-path). +### Container registry + +By default, the container registry is not automatically replicated to secondary +sites and this needs to be manually configured, see [Docker Registry for a secondary site](../replication/docker_registry.md). + +If you are using local storage on your current primary site for the container +registry, you can `rsync` the container registry objects to the secondary +site you are about to failover to: + +```shell +# Run from the secondary site +rsync --archive --perms --delete root@<geo-primary>:/var/opt/gitlab/gitlab-rails/shared/registry/. /var/opt/gitlab/gitlab-rails/shared/registry +``` + +Alternatively, you can [back up](../../../raketasks/backup_restore.md#back-up-gitlab) +the container registry on the primary site and restore it onto the secondary +site: + +1. On your primary site, back up only the registry and [exclude specific directories +from the backup](../../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup): + + ```shell + # Create a backup in the /var/opt/gitlab/backups folder + sudo gitlab-backup create SKIP=db,uploads,builds,artifacts,lfs,terraform_state,pages,repositories,packages + ``` + +1. Copy the backup tarball generated from your primary site to the `/var/opt/gitlab/backups` folder +on your secondary site. + +1. On your secondary site, restore the registry following the [Restore GitLab](../../../raketasks/backup_restore.md#restore-gitlab) +documentation. + ## Preflight checks NOTE: diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 70b3eaf5fea..e4432c41ff1 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -208,4 +208,16 @@ The following are additional validation tests we performed. - Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). - Outcome: When using Azure based replication the average time for an image to replicate from the primary object storage to the secondary was recorded as 40 seconds, the longest replication time was 70 seconds and the quickest was 11 seconds. When using GitLab based replication the average time for replication to complete was 5 seconds, the longest replication time was 10 seconds and the quickest was 3 seconds. - Follow up issue: - - [Test and validate object storage replication performance on reference architectures](https://gitlab.com/gitlab-org/gitlab/-/issues/347314) + - [Validate Cross Region Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/358154) + +### April 2022 + +[Validate Object storage replication using AWS based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351463): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located within the same region and when they are further apart (Europe to America). When using Geo managed replication within the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds. + +[Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual or single region based. This means that the bucket will automatically store replicas in a region based on the option chosen. Even when using multi region, this will still only replicate within a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating within the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 8f55ce99787..871d6041066 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -88,29 +88,49 @@ node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** s sudo gitlab-rake geo:status ``` -Example output: +The output includes: + +- a count of "failed" items if any failures occurred +- the percentage of "succeeded" items, relative to the "total" + +Example: ```plaintext http://secondary.example.com/ ----------------------------------------------------- - GitLab Version: 11.10.4-ee + GitLab Version: 14.9.2-ee Geo Role: Secondary Health Status: Healthy - Repositories: 289/289 (100%) - Verified Repositories: 289/289 (100%) - Wikis: 289/289 (100%) - Verified Wikis: 289/289 (100%) - LFS Objects: 8/8 (100%) - Attachments: 5/5 (100%) - CI job artifacts: 0/0 (0%) - Repositories Checked: 0/289 (0%) + Repositories: succeeded 12345 / total 12345 (100%) + Verified Repositories: succeeded 12345 / total 12345 (100%) + Wikis: succeeded 6789 / total 6789 (100%) + Verified Wikis: succeeded 6789 / total 6789 (100%) + Attachments: succeeded 4 / total 4 (100%) + CI job artifacts: succeeded 0 / total 0 (0%) + Design repositories: succeeded 1 / total 1 (100%) + LFS Objects: failed 1 / succeeded 2 / total 3 (67%) + Merge Request Diffs: succeeded 0 / total 0 (0%) + Package Files: failed 1 / succeeded 2 / total 3 (67%) + Terraform State Versions: failed 1 / succeeded 2 / total 3 (67%) + Snippet Repositories: failed 1 / succeeded 2 / total 3 (67%) + Group Wiki Repositories: succeeded 4 / total 4 (100%) + Pipeline Artifacts: failed 3 / succeeded 0 / total 3 (0%) + Pages Deployments: succeeded 0 / total 0 (0%) + Repositories Checked: failed 5 / succeeded 0 / total 5 (0%) + Package Files Verified: succeeded 0 / total 10 (0%) + Terraform State Versions Verified: succeeded 0 / total 10 (0%) + Snippet Repositories Verified: succeeded 99 / total 100 (99%) + Pipeline Artifacts Verified: succeeded 0 / total 10 (0%) Sync Settings: Full Database replication lag: 0 seconds - Last event ID seen from primary: 10215 (about 2 minutes ago) - Last event ID processed by cursor: 10215 (about 2 minutes ago) - Last status report was: 2 minutes ago + Last event ID seen from primary: 12345 (about 2 minutes ago) + Last event ID processed by cursor: 12345 (about 2 minutes ago) + Last status report was: 1 minute ago ``` +To find more details about failed items, check +[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors) + ### Check if PostgreSQL replication is working To check if PostgreSQL replication is working, check if: diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index b1183e56cd0..d96bfae78a1 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -27,7 +27,7 @@ Everything up-to-date ``` NOTE: -If you're using HTTPS instead of [SSH](../../../ssh/index.md) to push to the secondary, +If you're using HTTPS instead of [SSH](../../../user/ssh.md) to push to the secondary, you can't store credentials in the URL like `user:password@URL`. Instead, you can use a [`.netrc` file](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html) for Unix-like operating systems or `_netrc` for Windows. In that case, the credentials diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index d3bc54775f4..b0797445890 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -12,7 +12,7 @@ for updating Geo sites. ## Updating to 14.9 -**DO NOT** update to GitLab 14.9.0. +**DO NOT** update to GitLab 14.9.0. We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue will be fixed in the next patch release. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index e6a2e833af3..b2bdd876499 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -20,7 +20,7 @@ Authentication is the first step in making your installation secure. This one-time secret code is an additional safeguard that keeps intruders out, even if they have your password. - Add a backup email. If you lose access to your account, the GitLab Support team can help you more quickly. - Save or print your recovery codes. If you can't access your authentication device, you can use these recovery codes to sign in to your GitLab account. - - Add [an SSH key](../ssh/index.md) to your profile. You can generate new recovery codes as needed with SSH. + - Add [an SSH key](../user/ssh.md) to your profile. You can generate new recovery codes as needed with SSH. - Enable [personal access tokens](../user/profile/personal_access_tokens.md). When using 2FA, you can use these tokens to access the GitLab API. ## Projects and groups @@ -62,9 +62,10 @@ You may need to import projects from external sources like GitHub, Bitbucket, or ### Popular project imports -- [GitHub Enterprise to self-managed GitLab](../integration/github.md): Enabling OAuth makes it easier for developers to find and import their projects. -- [Bitbucket Server](../user/project/import/bitbucket_server.md#limitations): There are certain data limitations. - For assistance with these data types, contact your GitLab account manager or GitLab Support about our professional migration services. +- [GitHub Enterprise to self-managed GitLab](../integration/github.md) +- [Bitbucket Server](../user/project/import/bitbucket_server.md) + +For assistance with these data types, contact your GitLab account manager or GitLab Support about our professional migration services. ## GitLab instance security diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 88efd1885db..0fb285c50d6 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -300,7 +300,7 @@ disable enforcement. For more information, see the documentation on configuring ```toml listen_addr = '0.0.0.0:8075' - internal_socket_dir = '/var/opt/gitlab/gitaly' + runtime_dir = '/var/opt/gitlab/gitaly' [logging] format = 'json' @@ -308,6 +308,9 @@ disable enforcement. For more information, see the documentation on configuring dir = '/var/log/gitaly' ``` + For GitLab 14.9 and earlier, set `internal_socket_dir = '/var/opt/gitlab/gitaly'` instead + of `runtime_dir`. + 1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: On `gitaly1.internal`: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f88a1aa2b4d..8e4464bba43 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -43,23 +43,26 @@ repository storage is either: - Read requests are distributed between multiple Gitaly nodes, which can improve performance. - Write requests are broadcast to repository replicas. -## Guidance regarding Gitaly Cluster +## Before deploying Gitaly Cluster -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please review existing technical limitations and considerations prior to deploying Gitaly Cluster. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. +Before deploying Gitaly Cluster, please review: -- [Known issues](#known-issues) +- Existing [known issues](#known-issues). - [Snapshot limitations](#snapshot-backup-and-recovery-limitations). +- [Configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make + sure that Gitaly Cluster is the best setup for you. -Please also review the [configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best set-up for you. Finally, refer to the following guidance: +If you have: -- If you have not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the - service you are using. NFS is supported in 14.x releases. -- If you have not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options - a sharded Gitaly instance or Gitaly Cluster. -- If you have migrated to Gitaly Cluster and the limitations and tradeoffs are not suitable for your environment, your options are: - 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) back to your NFS solution - 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) to NFS solution or to a sharded Gitaly instance. +- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. NFS is + supported in 14.x releases but is [deprecated](../../update/deprecations.md#nfs-for-git-repository-storage). +Support for storing Git repository data on NFS will end for all versions of GitLab with the release of 15.0. +- Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options: + - A sharded Gitaly instance. + - Gitaly Cluster. -Reach out to your Technical Account Manager or customer support if you have any questions. +Contact your Technical Account Manager or customer support if you have any questions. ### Known issues @@ -69,19 +72,20 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage that do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | Don't directly change repositories on any Gitaly Cluster node at this time. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations -Gitaly Cluster does not support snapshot backups because these can cause issues where the Praefect -database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication -metadata of Gitaly disk information during a restore, we recommend using the -[official backup and restore Rake tasks](../../raketasks/backup_restore.md). If you are unable to use this method, please contact customer support for restoration help. +Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes +out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information +during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). -To track progress on work on a solution for manually re-synchronizing the Praefect database with -disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). +If you are unable to use this method, please contact customer support for restoration help. + +We are tracking in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351383) improvements to the +[official backup and restore Rake tasks](../../raketasks/backup_restore.md) to add support for incremental backups. For +more information, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/2094). ### What to do if you are on Gitaly Cluster experiencing an issue or limitation @@ -361,7 +365,8 @@ For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluste ## Migrating to Gitaly Cluster -Please see [current guidance on Gitaly Cluster](#guidance-regarding-gitaly-cluster). The basic process for migrating to Gitaly Cluster involves: +See the [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster) section before continuing. The basic process +for migrating to Gitaly Cluster involves: 1. Create the required storage. Refer to [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). @@ -378,16 +383,14 @@ NOTE: GitLab requires a `default` repository storage to be configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). -### Migrate off Gitaly Cluster +## Migrate off Gitaly Cluster -If you have repositories stored on a Gitaly Cluster, but you'd like to migrate -them back to direct Gitaly storage: +If the limitations and tradeoffs of Gitaly Cluster are found to be not suitable for your environment, you can Migrate +off Gitaly Cluster to a sharded Gitaly instance: -1. Create and configure a new - [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). -1. [Move the repositories](../operations/moving_repositories.md#move-repositories) - to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over - multiple Gitaly servers. +1. Create and configure a new [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). +1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can + move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. ## Monitor Gitaly and Gitaly Cluster diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index d7a06994b6c..7dee12a6690 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -231,6 +231,16 @@ instructions only work on Omnibus-provided PostgreSQL: The database used by Praefect is now configured. +You can now configure Praefect to use the database: + +```ruby +praefect['database_host'] = POSTGRESQL_HOST +praefect['database_port'] = 5432 +praefect['database_user'] = 'praefect' +praefect['database_password'] = PRAEFECT_SQL_PASSWORD +praefect['database_dbname'] = 'praefect_production' +``` + If you see Praefect database errors after configuring PostgreSQL, see [troubleshooting steps](troubleshooting.md#relation-does-not-exist-errors). @@ -619,7 +629,7 @@ Note the following: environment variable so that the Gitaly certificate is trusted. For example: ```shell - sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes + sudo SSL_CERT_DIR=/etc/gitlab/trusted-certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes ``` - You can configure Praefect servers with both an unencrypted listening address @@ -1197,7 +1207,7 @@ You can configure: ``` If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new -node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. ## Automatic failover and primary election strategies diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 516af4ca469..1be0bf23ed5 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -16,7 +16,8 @@ Before troubleshooting, see the Gitaly and Gitaly Cluster The following sections provide possible solutions to Gitaly errors. -See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings. +See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings, +and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent). ### Check versions when using standalone Gitaly servers @@ -614,7 +615,7 @@ Possible solutions: ## Profiling Gitaly Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening -on port `9236` of the GitLab server: +on port `9236` of the GitLab server: - Get a list of running `goroutines` and their backtraces: diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 4de48aa3f14..71b8439f70a 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -40,8 +40,6 @@ The following housekeeping options are available: - **Enable automatic repository housekeeping**: Regularly run `git repack` and `git gc`. If you keep this setting disabled for a long time, Git repository access on your GitLab server becomes slower and your repositories use more disk space. -- **Enable Git pack file bitmap creation**: Create pack file bitmaps which accelerates `git clone` - performance. Makes housekeeping take a little longer. - **Incremental repack period**: Number of Git pushes after which an incremental `git repack` is run. - **Full repack period**: Number of Git pushes after which a full `git repack` is run. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index caade32b7c2..63f8ce3c4cb 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -564,7 +564,7 @@ to receive sub-addressed mail. To enable sub-addressing: -1. Download and install the `ExchangeOnlineManagement` module from the [PowerShell gallery](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/). +1. Download and install the `ExchangeOnlineManagement` module from the [PowerShell gallery](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/2.0.5). 1. In PowerShell, run the following commands: ```powershell @@ -766,6 +766,8 @@ This example for Omnibus GitLab assumes you're using the following mailbox: `inc ##### Configure Microsoft Graph +> Alternative Azure deployments [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5978) in GitLab 14.9. + ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -788,4 +790,19 @@ gitlab_rails['incoming_email_inbox_options'] = { } ``` +For Microsoft Cloud for US Government or [other Azure deployments](https://docs.microsoft.com/en-us/graph/deployments), configure the `azure_ad_endpoint` and `graph_endpoint` settings. + +- Example for Microsoft Cloud for US Government: + +```ruby +gitlab_rails['incoming_email_inbox_options'] = { + 'azure_ad_endpoint': 'https://login.microsoftonline.us', + 'graph_endpoint': 'https://graph.microsoft.us', + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional +} +``` + The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. diff --git a/doc/administration/index.md b/doc/administration/index.md index 50dc2bc905a..73ea4a8e1d0 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -32,7 +32,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Install](../install/index.md): Requirements, directory structures, and installation methods. - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. +- [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. - [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. - [Add License](../user/admin_area/license.md): Add a license at install time to unlock features that are in paid tiers of GitLab. @@ -40,7 +40,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Configuring GitLab - [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab. -- [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. +- [System hooks](system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/index.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. - [Global user settings](user_settings.md): Configure instance-wide user permissions. @@ -126,7 +126,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sync LDAP](auth/ldap/index.md) - [Kerberos authentication](../integration/kerberos.md) - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from GitLab. +- [Email users](../user/admin_area/email_from_gitlab.md): Email GitLab users from GitLab. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit events](audit_events.md): View the changes made on the GitLab server for: - Groups and projects. @@ -146,7 +146,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. - [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab. - [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project. -- [Restrict the use of public or internal projects](../public_access/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. +- [Restrict the use of public or internal projects](../user/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. - [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. ## Package Registry administration @@ -218,6 +218,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) - [Navigating GitLab via Rails console](troubleshooting/navigating_gitlab_via_rails_console.md) - [GitLab application limits](instance_limits.md) +- [Responding to security incidents](../security/responding_to_security_incidents.md) ### Support Team Docs diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 11a34f5b5f8..0a0c538caa2 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -240,7 +240,7 @@ Activity history for projects and individuals' profiles was limited to one year > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14939) in GitLab 12.7. -There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for performance reasons. +There is a limit when embedding metrics in GitLab Flavored Markdown (GLFM) for performance reasons. - **Max limit**: 100 embeds. @@ -383,6 +383,20 @@ Plan.default.actual_limits.update!(ci_active_jobs: 500) Set the limit to `0` to disable it. +### Maximum time jobs can run + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16777) in GitLab 12.3. + +The default maximum time that jobs can run for is 60 minutes. Jobs that run for +more than 60 minutes time out. + +You can change the maximum time a job can run before it times out: + +- At the project-level in the [project's CI/CD settings](../ci/pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run) + for a given project. This limit must be between 10 minutes and 1 month. +- At the [runner level](../ci/runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + This limit must be 10 minutes or longer. + ### Maximum number of deployment jobs in a pipeline > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46931) in GitLab 13.7. @@ -858,7 +872,7 @@ See the [documentation about Snippets settings](snippets/index.md). ## Design Management limits -See the [Design Management Limitations](../user/project/issues/design_management.md#limitations) section. +See the limits in the [Add a design to an issue](../user/project/issues/design_management.md#add-a-design-to-an-issue) section. ## Push Event Limits @@ -962,3 +976,16 @@ retarget merge requests pointing to the now-merged branch. To learn more, read ## CDN-based limits on GitLab.com In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://about.gitlab.com/handbook/communication/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations. + +## Container Repository tag deletion limit + +Container repository tags are in the Container Registry and, as such, each tag deletion will trigger network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. + +## Project-level Secure Files API limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8. + +The [secure files API](../api/secure_files.md) enforces the following limits: + +- Files must be smaller than 5 MB. +- Projects cannot have more than 100 secure files. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 008d33c6c94..0f02e3783a5 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12. When [Kroki](https://kroki.io) integration is enabled and configured in -GitLab you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. +GitLab, you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. ## Kroki Server diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 5a9699b3a39..cdf6d48ad41 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -213,7 +213,7 @@ After configuring your local PlantUML server, you're ready to enable the PlantUM 1. On the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** checkbox. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, - and click **Save changes**. + and select **Save changes**. Depending on your PlantUML and GitLab version numbers, you may also need to take these steps: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 544a5973052..03a0162d526 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -307,33 +307,6 @@ To migrate back to local storage: ## Expiring artifacts -> [In GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76504), we improved the performance of removing expired artifacts, introduced [with a flag](feature_flags.md) named `ci_destroy_all_expired_service`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to -[enable the feature flag](feature_flags.md) named `ci_destroy_all_expired_service`. The feature is not ready for -production use. -On GitLab.com, this feature is not available. - -### Removing expired job artifacts on GitLab self-managed instances - -In the process of migrating old artifacts for our SaaS customers, we are working to resolve any potential unrecoverable data loss for self-managed customers for artifacts that they may not want deleted yet. Before we can use the more performant way of cleaning up expired artifacts, we need to do some remediation to make sure customers don't lose their data, which is part of our effort in [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/7097). - -Two options are available: - -- If you don't need any artifacts created before 2020-06-23, an Administrator can enable the worker for removing expired CI/CD artifacts: - - ```ruby - Feature.enable(:ci_destroy_all_expired_service) - ``` - -- If you want to keep any artifacts (including job logs) before 2020-06-23, follow the [progress of the migration effort](https://gitlab.com/groups/gitlab-org/-/epics/7097) where we work on a resolution to have this flag fully enabled in a future release. - -Alternatively, Administrators can also run commands in the Rails console to -[delete artifacts from completed jobs prior to a specific date](#delete-job-artifacts-from-jobs-completed-before-a-specific-date). - -### Usage details - If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). @@ -641,4 +614,4 @@ WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab. FATAL: invalid argument ``` -If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. +If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 09cef51d14c..6f2e3cce0fa 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -8,18 +8,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default. > - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 15.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 16.0. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) -for use in GitLab 14.9, and is planned for removal in GitLab 15.0. +in GitLab 14.9, and is planned for removal in GitLab 16.0. GitLab provides administrators insights into the health of their GitLab instance. To provide a native experience (similar interacting with an application deployed using GitLab), a project called **Monitoring** is created: -- With [internal visibility](../../../public_access/public_access.md#internal-projects-and-groups). +- With [internal visibility](../../../user/public_access.md#internal-projects-and-groups). - Under a group called **GitLab Instance**. The project is created specifically for visualizing and configuring the monitoring of your GitLab diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index f8a66f6f48f..6cd20132092 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) -for use in GitLab 14.8, and is planned for removal in GitLab 15.0. +in GitLab 14.8, and is planned for removal in GitLab 15.0. To profile a request: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index a1bfae678c7..08f7d5095da 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -88,7 +88,6 @@ The following metrics are available: | `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | | `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | | `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | -| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | | `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | | `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 50c8102f366..ef5d26ac845 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -246,7 +246,7 @@ use the `hard` option, because (from the man page): > use the soft option only when client responsiveness is more important than data integrity Other vendors make similar recommendations, including -[System Applications and Products in Data Processing (SAP)](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's +[System Applications and Products in Data Processing (SAP)](https://wiki.scn.sap.com/wiki/display/Basis/Recommended+mount+options+for+read-write+directories) and NetApp's [knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), they highlight that if the NFS client driver caches data, `soft` means there is no certainty if writes by GitLab are actually on disk. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 8ea4ad9019e..6ea433f95e9 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -17,7 +17,7 @@ GitLab has been tested by vendors and customers on a number of object storage pr - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 8877b1266e0..477a11dd899 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -31,7 +31,8 @@ able to accept a fingerprint. Check the version of OpenSSH on your server with ` ## Fast lookup is required for Geo **(PREMIUM)** -By default, GitLab manages an `authorized_keys` file that is located in the +Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), Omnibus GitLab by default +manages an `authorized_keys` file that is located in the `git` user's home directory. For most installations, this will be located under `/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system: @@ -83,7 +84,7 @@ file (start the line with a `#` to comment it), and from your local machine, att ssh -T git@gitlab.example.com ``` -A successful pull or [welcome message](../../ssh/index.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, +A successful pull or [welcome message](../../user/ssh.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. NOTE: diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 9e828b39f46..c12f75989c3 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -108,6 +108,12 @@ To change the worker timeout to 600 seconds: ## Disable Puma clustered mode in memory-constrained environments +WARNING: +This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature +is not ready for production use. If you want to use this feature, we recommend testing +with non-production data first. See the [known issues](#puma-single-mode-known-issues) +for additional details. + In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma [clustered mode](https://github.com/puma/puma#clustered-mode). @@ -131,6 +137,8 @@ For details on Puma worker and thread settings, see the [Puma requirements](../. The downside of running Puma in this configuration is the reduced throughput, which can be considered a fair tradeoff in a memory-constrained environment. +### Puma single mode known issues + When running Puma in single mode, some features are not supported: - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index e707cb70187..c80437221c4 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -26,8 +26,8 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 14.1 | 12.6, 13.3 | 12.6 | 12.6 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). -| 14.0 | 12.6 | 12.6 | 12.6 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | +| 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). +| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | | 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | | 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 547b8bf8658..647620858d9 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -24,14 +24,15 @@ The following lists the currently supported OSs and their possible EOL dates. | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | 2026 | <https://wiki.debian.org/LTS> | | OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | May 2024 | <https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates> | | SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs> | -| Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | 2024 | <https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/> | NOTE: -CentOS 8 will be EOL on December 31, 2021. In GitLab 14.5 and later, +CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, [CentOS builds work in AlmaLinux](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/954#note_730198505). We will officially support all distributions that are binary compatible with Red Hat Enterprise Linux. This gives users a path forward for their CentOS 8 builds at its end of life. @@ -78,7 +79,7 @@ release for them can be found below: | Ubuntu 14.04 | [April 2019](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.10&dist=ubuntu%2Ftrusty) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.10&dist=ubuntu%2Ftrusty) 11.10 | | OpenSUSE 42.3 | [July 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.1&dist=opensuse%2F42.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.1&dist=opensuse%2F42.3) 12.1 | | OpenSUSE 15.0 | [December 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.5&dist=opensuse%2F15.0) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.5&dist=opensuse%2F15.0) 12.5 | -| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | +| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.3&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 7bf7362c68b..d6f5588ebc9 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -903,6 +903,10 @@ You can also [run cleanup on a schedule](../../user/packages/container_registry/ ## Container Registry garbage collection +NOTE: +Retention policies within your object storage provider, such as Amazon S3 Lifecycle, may prevent +objects from being properly deleted. + Container Registry can use considerable amounts of disk space. To clear up some unused layers, the registry includes a garbage collect command. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 93c6585230f..4c50cfeeb0f 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -264,3 +264,29 @@ Feature.disable(:dependency_proxy_for_private_groups) # Re-enable the authentication Feature.enable(:dependency_proxy_for_private_groups) ``` + +## Changing the JWT expiration + +The Dependency Proxy follows the [Docker v2 token authentication flow](https://docs.docker.com/registry/spec/auth/token/), +issuing the client a JWT to use for the pull requests. The token expiration time is a configurable +using the application setting `container_registry_token_expire_delay`. It can be changed from the +rails console: + +```ruby +# update the JWT expiration to 30 minutes +ApplicationSetting.update(container_registry_token_expire_delay: 30) +``` + +The default expiration and the expiration on GitLab.com is 15 minutes. + +## Using the dependency proxy behind a proxy + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines: + + ```ruby + gitlab_workhorse['env'] = { + "http_proxy" => "http://USERNAME:PASSWORD@example.com:8080", + "https_proxy" => "http://USERNAME:PASSWORD@example.com:8080" + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index abf5c46114b..b122cb9db90 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -10,9 +10,7 @@ GitLab Packages allows organizations to use GitLab as a private repository for a variety of common package managers. Users are able to build and publish packages, which can be easily consumed as a dependency in downstream projects. -The Packages feature allows GitLab to act as a repository for the following: - -The Package Registry supports the following formats: +The Packages feature allows GitLab to act as a repository and supports the following formats: | Package type | GitLab version | |-------------------------------------------------------------------|----------------| diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 56ccf70d169..114894c01f0 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: 'Learn how to administer GitLab Pages.' --- @@ -243,6 +243,7 @@ control over how the Pages daemon runs and serves content in your environment. | `enable` | Enable or disable GitLab Pages on the current system. | | `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | | `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | +| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30s). | | `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). | | `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). | | `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600s). | @@ -292,6 +293,10 @@ control over how the Pages daemon runs and serves content in your environment. | `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | | `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. | | `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | +| `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` | +| `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` | +| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `5m` | +| `server_keep_alive` | The `Keep-Alive` period for network connections accepted by this listener. If `0`, `Keep-Alive` is enabled if supported by the protocol and operating system. If negative, `Keep-Alive` is disabled. Default: `15s` | ## Advanced configuration @@ -1392,15 +1397,15 @@ in all of your GitLab Pages instances. Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). This can happen to GitLab instances with multiple servers -running both the core GitLab application and GitLab Pages. This can also happen when a single +running both the core GitLab application and GitLab Pages. This can also happen when a single container is running both the core GitLab application and GitLab Pages. AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) to resolve this issue. -Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) -may resolve this issue when the core GitLab application and GitLab Pages run on the same host or -container. +Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) +may resolve this issue when the core GitLab application and GitLab Pages run on the same host or +container. ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index c4b1756d8a1..6c148387d7d 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md deleted file mode 100644 index 457077462a6..00000000000 --- a/doc/administration/raketasks/doctor.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'check.md#verify-database-values-can-be-decrypted-using-the-current-secrets' -remove_date: '2022-03-04' ---- - -This document was moved to [another location](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -<!-- This redirect file can be deleted after 2022-03-04. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index d66f3b1ed35..82ebdcaab42 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -183,9 +183,12 @@ Redis version >= 2.0.0? ... yes Checking GitLab ... Finished ``` -## Rebuild authorized_keys file +## Rebuild `authorized_keys` file -In some case it is necessary to rebuild the `authorized_keys` file. To do this, run: +In some cases it is necessary to rebuild the `authorized_keys` file, +for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../ssh/index.md) +and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs.md#gitlab-shelllog). +To rebuild `authorized_keys`, run: **Omnibus Installation** diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index fcce44f62b2..2ca79bbeae4 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested daily with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1157,8 +1157,8 @@ are supported and can be added if needed. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1455,10 +1455,12 @@ To configure the Praefect nodes, on each one: ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1531,6 +1533,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1768,6 +1790,7 @@ To configure the Sidekiq nodes, on each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1914,6 +1937,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2093,6 +2117,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2119,7 +2167,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2136,7 +2184,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2236,11 +2285,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2250,18 +2299,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|-----------------------|------------------|--------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2269,7 +2318,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 0d0e7681ffd..7213a1eb92b 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -21,7 +21,7 @@ many organizations. > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you > can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). -> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index c08fe985b40..2a1d344508e 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** @@ -24,7 +24,7 @@ full list of reference architectures, see | Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -86,7 +86,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } @@ -1163,8 +1163,8 @@ fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1459,10 +1459,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1535,6 +1537,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1772,6 +1794,7 @@ To configure the Sidekiq nodes, on each one: # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1920,6 +1943,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2098,6 +2122,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2123,7 +2171,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2140,7 +2188,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2234,11 +2283,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|-----------------|---------------------------------| +| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 223 vCPU, 206.5 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2248,18 +2297,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|------------------------|------------------|--------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2267,7 +2316,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -2312,7 +2361,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 6f6c02c309a..f417d1e2ab5 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,9 +13,9 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). -> - **Estimated Costs:** [See cost table](index.md#cost-to-run) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** @@ -397,11 +397,13 @@ are supported and can be added if needed. ## Configure Gitaly -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5TB of data. Although this -reference architecture includes a single Gitaly server node, you may require -additional nodes depending on your repository storage requirements. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data size, +specifically the number of projects and those projects' sizes. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or switching to Gitaly Cluster +may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -665,6 +667,7 @@ On each node perform the following: # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -886,7 +889,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -903,7 +906,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -1000,11 +1004,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|------------------------|-----------------|-----------------------------| -| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | 23.7 vCPU, 16.9 GB memory | -| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|------------------------|-----------------|--------------|---------------------------------| +| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | 23.7 vCPU, 16.9 GB memory | +| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | 1.9 vCPU, 5.5 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -1014,12 +1018,12 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Object storage<sup>3</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|----------------------------|-------|------------------------|-----------------|-------------| +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Object storage<sup>3</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 76f81e65580..9d8be3e90b6 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -24,7 +24,7 @@ For a full list of reference architectures, see > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** @@ -51,7 +51,7 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1104,8 +1104,8 @@ The following IPs will be used as an example: In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1399,10 +1399,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1475,6 +1477,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1696,6 +1718,7 @@ To configure the Sidekiq nodes, one each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1882,6 +1905,7 @@ On each node perform the following: # Object storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2018,6 +2042,30 @@ running [Prometheus](../monitoring/prometheus/index.md) and retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.21:9188", + "10.6.0.22:9188", + "10.6.0.23:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2058,7 +2106,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2075,7 +2123,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2157,6 +2206,7 @@ but with smaller performance requirements, several modifications can be consider - Reducing the node counts: Some node types do not need consensus and can run with fewer nodes (but more than one for redundancy). This will also lead to reduced performance. - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. + - PostgreSQL and PgBouncer: A quorum is not strictly necessary. Two PostgreSQL nodes and two PgBouncer nodes are enough for redundancy. - Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or Amazon RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. @@ -2193,11 +2243,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 31.8 vCPU, 24.8 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 31.8 vCPU, 24.8 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2207,17 +2257,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|-------|-----------------------|-----------------|-------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2225,7 +2275,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index dfa963d1ad0..6c4e2227b18 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1170,8 +1170,8 @@ Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1468,10 +1468,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1544,6 +1546,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1781,6 +1803,7 @@ To configure the Sidekiq nodes, on each one: # Object storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1936,6 +1959,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2114,6 +2138,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2139,7 +2187,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2156,7 +2204,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2250,12 +2299,12 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | - +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `m5.8xlarge` | 510 vCPU, 472 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | + - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. @@ -2264,18 +2313,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|------------------------|------------------|---------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2283,7 +2332,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index f2463afbf3b..dd7209a3a3a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -21,7 +21,7 @@ costly-to-operate environment by using the > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** @@ -48,7 +48,7 @@ costly-to-operate environment by using the 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1101,8 +1101,8 @@ The following IPs will be used as an example: In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1397,10 +1397,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1473,6 +1475,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1693,6 +1715,7 @@ To configure the Sidekiq nodes, one each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1867,6 +1890,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2018,6 +2042,30 @@ running [Prometheus](../monitoring/prometheus/index.md) and retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2058,7 +2106,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2075,7 +2123,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2169,11 +2218,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 79.5 vCPU, 62 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2183,17 +2232,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|-------|-----------------------|-----------------|--------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2201,7 +2250,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 3fcd6d7ae4e..bb741c39c08 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -18,20 +18,6 @@ you scale GitLab accordingly.  <!-- Internal link: https://docs.google.com/spreadsheets/d/1obYP4fLKkVVDOljaI3-ozhmCiPtEeMblbBKkf2OADKs/edit#gid=1403207183 --> -Testing on these reference architectures was performed with the -[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) -at specific coded workloads, and the throughputs used for testing were -calculated based on sample customer data. Select the -[reference architecture](#available-reference-architectures) that matches your scale. - -Each endpoint type is tested with the following number of requests per second (RPS) -per 1,000 users: - -- API: 20 RPS -- Web: 2 RPS -- Git (Pull): 2 RPS -- Git (Push): 0.4 RPS (rounded to nearest integer) - For GitLab instances with less than 2,000 users, it's recommended that you use the [default setup](#automated-backups) by [installing GitLab](../../install/index.md) on a single machine to minimize @@ -48,7 +34,8 @@ When scaling GitLab, there are several factors to consider: - A load balancer is added in front to distribute traffic across the application nodes. - The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. -NOTE: +## Available reference architectures + Depending on your workflow, the following recommended reference architectures may need to be adapted accordingly. Your workload is influenced by factors including how active your users are, how much automation you use, mirroring, @@ -57,12 +44,10 @@ provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-ty For different cloud vendors, attempt to select options that best match the provided architecture. -## Available reference architectures - -The following reference architectures are available. - ### GitLab package (Omnibus) +The following reference architectures, where the GitLab package is used, are available: + - [Up to 1,000 users](1k_users.md) - [Up to 2,000 users](2k_users.md) - [Up to 3,000 users](3k_users.md) @@ -87,17 +72,53 @@ to get assistance from Support with troubleshooting the [2,000 users](2k_users.m and higher reference architectures. [Read more about our definition of scaled architectures](https://about.gitlab.com/support/#definition-of-scaled-architecture). -### Validation and test results +## Validation and test results + +The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) +does regular smoke and performance tests for the reference architectures to ensure they +remain compliant. + +### Why we perform the tests + +The Quality Department has a focus on measuring and improving the performance +of GitLab, as well as creating and validating reference architectures that +self-managed customers can rely on as performant configurations. + +For more information, see our [handbook page](https://about.gitlab.com/handbook/engineering/quality/performance-and-scalability/). + +### How we perform the tests + +Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: + +- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. +- The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. + +Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. + +We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. + +The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). + +Testing on these reference architectures is performed with the +[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) +at specific coded workloads, and the throughputs used for testing are +calculated based on sample customer data. Select the +[reference architecture](#available-reference-architectures) that matches your scale. -The [Quality Engineering - Enablement team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) does regular smoke and performance tests for the reference architectures to ensure they remain compliant. +Each endpoint type is tested with the following number of requests per second (RPS) +per 1,000 users: -- Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: - - The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. - - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. -- Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. -- We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. -- Testing is done publicly and all results are shared. -- For more information about performance testing at GitLab, read [how our QA team leverages GitLab’s performance testing tool (and you can too)](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). +- API: 20 RPS +- Web: 2 RPS +- Git (Pull): 2 RPS +- Git (Push): 0.4 RPS (rounded to nearest integer) + +### How to interpret the results + +NOTE: +Read our blog post on [how our QA team leverages GitLab’s performance testing tool](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). + +Testing is done publicly and all results are shared. The following table details the testing done against the reference architectures along with the frequency and results. Additional testing is continuously evaluated, and the table is updated accordingly. @@ -192,9 +213,7 @@ table.test-coverage th { </tr> </table> -The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). - -### Cost to run +## Cost to run <table class="test-coverage"> <col> @@ -217,61 +236,61 @@ The Standard Reference Architectures are designed to be platform agnostic, with <th scope="row">1k</th> <td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> </tr> <tr> <th scope="row">2k</th> <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/3f973040ebc14023933d35f576c89846">Calculated cost</a></td> </tr> </table> -### Recommended cloud providers and services +## Recommended cloud providers and services NOTE: The following lists are non exhaustive. Generally, other cloud providers not listed @@ -347,7 +366,7 @@ The following specific cloud provider services have been found to have issues in - [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead. - [Azure Database for PostgreSQL Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality. -- [AWS Aurora Database](https://aws.amazon.com/rds/aurora) is not recommended due to compatibility issues. +- [AWS Aurora Database](https://aws.amazon.com/rds/aurora/) is not recommended due to compatibility issues. NOTE: As a general rule we unfortunately don't recommend Azure Services at this time. @@ -411,7 +430,7 @@ to any of the [available reference architectures](#available-reference-architect > - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems GitLab supports [zero-downtime upgrades](../../update/zero_downtime.md). -Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/zero_downtime.md#single-node-deployment). +Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/index.md#upgrade-based-on-installation-method). To avoid this, we recommend to separate GitLab into several application nodes. As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 6dd8faac65b..c2233f70a9a 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -34,7 +34,7 @@ When enabled, GitLab periodically runs a repository check on all project reposit repositories to detect possible data corruption. A project is checked no more than once per month. Administrators can configure the frequency of repository checks. To edit the frequency: -- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in +- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in `/etc/gitlab/gitlab.rb`. - For source-based installations, edit `[gitlab.cron_jobs.repository_check_worker]` in `/home/git/gitlab/config/gitlab.yml`. diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md index 2ed736bac2c..ae02711f46f 100644 --- a/doc/administration/sidekiq_health_check.md +++ b/doc/administration/sidekiq_health_check.md @@ -21,7 +21,7 @@ The readiness probe checks whether the Sidekiq workers are ready to process jobs GET /readiness ``` -Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +If you set Sidekiq's address as `localhost` and port as `8092`, here's an example request: ```shell @@ -44,7 +44,7 @@ Checks whether the Sidekiq cluster is running. GET /liveness ``` -Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +If you set Sidekiq's address as `localhost` and port as `8092`, here's an example request: ```shell diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md new file mode 100644 index 00000000000..71d7e7f1426 --- /dev/null +++ b/doc/administration/system_hooks.md @@ -0,0 +1,771 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# System hooks **(FREE SELF)** + +Your GitLab instance can perform HTTP POST requests on the following events: + +- `group_create` +- `group_destroy` +- `group_rename` +- `key_create` +- `key_destroy` +- `project_create` +- `project_destroy` +- `project_rename` +- `project_transfer` +- `project_update` +- `repository_update` +- `user_add_to_group` +- `user_add_to_team` +- `user_create` +- `user_destroy` +- `user_failed_login` +- `user_remove_from_group` +- `user_remove_from_team` +- `user_rename` +- `user_update_for_group` +- `user_update_for_team` + +The triggers for most of these are self-explanatory, but `project_update` and +`project_rename` deserve some clarification: `project_update` is fired any time +an attribute of a project is changed (including name, description, and tags) +_unless_ the `path` attribute is also changed. In that case, a `project_rename` +is triggered instead (so that, for instance, if all you care about is the +repository URL, you can just listen for `project_rename`). + +`user_failed_login` is sent whenever a _blocked_ user attempts to sign in and is +denied access. + +System hooks can be used, for example, for logging or changing information in an +LDAP server. + +In addition to these default events, you can enable triggers for other events, +such as push events, and disable the `repository_update` event +when you create a system hook. + +NOTE: +We follow the same structure and deprecations as [Webhooks](../user/project/integrations/webhooks.md) +for Push and Tag events, but we never display commits. + +## Create a system hook + +To create a system hook: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **System Hooks**. +1. Provide the **URL** and **Secret Token**. +1. Select the checkbox next to each optional **Trigger** you want to enable. +1. Select **Enable SSL verification**, if desired. +1. Click **Add system hook**. + +## Hooks request example + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Project created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_create", + "name": "StoreCloud", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private" +} +``` + +**Project destroyed:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_destroy", + "name": "Underscore", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "underscore", + "path_with_namespace": "jsmith/underscore", + "project_id": 73, + "project_visibility": "internal" +} +``` + +**Project renamed:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_rename", + "name": "Underscore", + "path": "underscore", + "path_with_namespace": "jsmith/underscore", + "project_id": 73, + "owner_name": "John Smith", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "project_visibility": "internal", + "old_path_with_namespace": "jsmith/overscore" +} +``` + +Note that `project_rename` is not triggered if the namespace changes. +Please refer to `group_rename` and `user_rename` for that case. + +**Project transferred:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_transfer", + "name": "Underscore", + "path": "underscore", + "path_with_namespace": "scores/underscore", + "project_id": 73, + "owner_name": "John Smith", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "project_visibility": "internal", + "old_path_with_namespace": "jsmith/overscore" +} +``` + +**Project updated:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_update", + "name": "StoreCloud", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private" +} +``` + +**New Team Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_add_to_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**Team Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_remove_from_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**Team Member Updated:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_update_for_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**User created:** + +```json +{ + "created_at": "2012-07-21T07:44:07Z", + "updated_at": "2012-07-21T07:38:22Z", + "email": "js@gitlabhq.com", + "event_name": "user_create", + "name": "John Smith", + "username": "js", + "user_id": 41 +} +``` + +**User removed:** + +```json +{ + "created_at": "2012-07-21T07:44:07Z", + "updated_at": "2012-07-21T07:38:22Z", + "email": "js@gitlabhq.com", + "event_name": "user_destroy", + "name": "John Smith", + "username": "js", + "user_id": 41 +} +``` + +**User failed login:** + +```json +{ + "event_name": "user_failed_login", + "created_at": "2017-10-03T06:08:48Z", + "updated_at": "2018-01-15T04:52:06Z", + "name": "John Smith", + "email": "user4@example.com", + "user_id": 26, + "username": "user4", + "state": "blocked" +} +``` + +If the user is blocked via LDAP, `state` is `ldap_blocked`. + +**User renamed:** + +```json +{ + "event_name": "user_rename", + "created_at": "2017-11-01T11:21:04Z", + "updated_at": "2017-11-01T14:04:47Z", + "name": "new-name", + "email": "best-email@example.tld", + "user_id": 58, + "username": "new-exciting-name", + "old_username": "old-boring-name" +} +``` + +**Key added** + +```json +{ + "event_name": "key_create", + "created_at": "2014-08-18 18:45:16 UTC", + "updated_at": "2012-07-21T07:38:22Z", + "username": "root", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", + "id": 4 +} +``` + +**Key removed** + +```json +{ + "event_name": "key_destroy", + "created_at": "2014-08-18 18:45:16 UTC", + "updated_at": "2012-07-21T07:38:22Z", + "username": "root", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", + "id": 4 +} +``` + +**Group created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "group_create", + "name": "StoreCloud", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group removed:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "group_destroy", + "name": "StoreCloud", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group renamed:** + +```json +{ + "event_name": "group_rename", + "created_at": "2017-10-30T15:09:00Z", + "updated_at": "2017-11-01T10:23:52Z", + "name": "Better Name", + "path": "better-name", + "full_path": "parent-group/better-name", + "group_id": 64, + "old_path": "old-name", + "old_full_path": "parent-group/old-name" +} +``` + +**New Group Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_add_to_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +**Group Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_remove_from_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +**Group Member Updated:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_update_for_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +## Push events + +Triggered when you push to the repository, except when pushing tags. +It generates one event per modified branch. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "push", + "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", + "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "ref": "refs/heads/master", + "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "user_id": 4, + "user_name": "John Smith", + "user_email": "john@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 15, + "project":{ + "name":"Diaspora", + "description":"", + "web_url":"http://example.com/mike/diaspora", + "avatar_url":null, + "git_ssh_url":"git@example.com:mike/diaspora.git", + "git_http_url":"http://example.com/mike/diaspora.git", + "namespace":"Mike", + "visibility_level":0, + "path_with_namespace":"mike/diaspora", + "default_branch":"master", + "homepage":"http://example.com/mike/diaspora", + "url":"git@example.com:mike/diaspora.git", + "ssh_url":"git@example.com:mike/diaspora.git", + "http_url":"http://example.com/mike/diaspora.git" + }, + "repository":{ + "name": "Diaspora", + "url": "git@example.com:mike/diaspora.git", + "description": "", + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 + }, + "commits": [ + { + "id": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "message": "Add simple search to projects in public area", + "timestamp": "2013-05-13T18:18:08+00:00", + "url": "https://dev.gitlab.org/gitlab/gitlabhq/commit/c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "author": { + "name": "Example User", + "email": "user@example.com" + } + } + ], + "total_commits_count": 1 +} +``` + +## Tag events + +Triggered when you create (or delete) tags to the repository. +It generates one event per modified tag. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "tag_push", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "ref": "refs/tags/v1.0.0", + "checkout_sha": "5937ac0a7beb003549fc5fd26fc247adbce4a52e", + "user_id": 1, + "user_name": "John Smith", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project":{ + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "repository":{ + "name": "Example", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + +## Merge request events + +Triggered when a new merge request is created, an existing merge request was +updated/merged/closed or a commit is added in the source branch. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +```json +{ + "object_kind": "merge_request", + "event_type": "merge_request", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project": { + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" + }, + "object_attributes": { + "id": 99, + "target_branch": "master", + "source_branch": "ms-viewport", + "source_project_id": 14, + "author_id": 51, + "assignee_id": 6, + "title": "MS-Viewport", + "created_at": "2013-12-03T17:23:34Z", + "updated_at": "2013-12-03T17:23:34Z", + "milestone_id": null, + "state": "opened", + "merge_status": "unchecked", + "target_project_id": 14, + "iid": 1, + "description": "", + "source": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "target": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "last_commit": { + "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "message": "fixed readme", + "timestamp": "2012-01-03T23:36:29+02:00", + "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "author": { + "name": "GitLab dev user", + "email": "gitlabdev@dv6700.(none)" + } + }, + "work_in_progress": false, + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open", + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + } + }, + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "changes": { + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, + "labels": { + "previous": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "current": [{ + "id": 205, + "title": "Platform", + "color": "#123123", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "Platform related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + } + } +} +``` + +## Repository Update events + +Triggered only once when you push to the repository (including tags). + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "repository_update", + "user_id": 1, + "user_name": "John Smith", + "user_email": "admin@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project": { + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "changes": [ + { + "before":"8205ea8d81ce0c6b90fbe8280d118cc9fdad6130", + "after":"4045ea7a3df38697b3730a20fb73c8bed8a3e69e", + "ref":"refs/heads/master" + } + ], + "refs":["refs/heads/master"] +} +``` + +## Local requests in system hooks + +[Requests to local network by system hooks](../security/webhooks.md) can be allowed +or blocked by an administrator. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 1c948771f5b..54d934c8986 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -257,6 +257,13 @@ ProjectDestroyWorker.perform_async(project.id, user.id, {}) # or Projects::DestroyService.new(project, user).execute ``` +If this fails, display why it doesn't work with: + +```ruby +project = Project.find_by_full_path('<project_path>') +project.delete_error +``` + ### Remove fork relationship manually ```ruby diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index a30ade058f6..c6a102e87ee 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -20,6 +20,7 @@ They may then set up a test configuration of the desired identity provider. We i This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - [Azure Active Directory](#azure-active-directory) +- [Google Workspace](#google-workspace) - [Okta](#okta) - [OneLogin](#onelogin) @@ -47,6 +48,20 @@ Group Sync:  +## Google Workspace + +Basic SAML app configuration: + + + +User claims and attributes: + + + +IdP links and certificate: + + + ## Okta Basic SAML app configuration: diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png Binary files differnew file mode 100644 index 00000000000..bc11e18fb6f --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png Binary files differnew file mode 100644 index 00000000000..78bb1725e9c --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png Binary files differnew file mode 100644 index 00000000000..e665e23058c --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index c5443c564f4..c5b1d302db2 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -11,6 +11,11 @@ but if they are not available you can still quickly parse [GitLab logs](../logs.md) in JSON format (the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). +NOTE: +Spefically for summarising error events and basic usage statistics, +the GitLab Support Team provides the specialised +[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). + ## What is JQ? As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples @@ -18,6 +23,10 @@ include use cases targeted for parsing GitLab log files. ## Parsing Logs +The examples listed below address their respective log files by +their relative Omnibus paths and default filenames. +Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog). + ### General Commands #### Pipe colorized `jq` output into `less` @@ -61,7 +70,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R ``` -### Parsing `production_json.log` and `api_json.log` +### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log` #### Find all requests with a 5XX status code @@ -111,7 +120,7 @@ jq 'select(.queue_duration > 10000)' <FILE> jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> ``` -### Parsing `production_json.log` +### Parsing `gitlab-rails/production_json.log` #### Print the top three controller methods by request volume and their three longest durations @@ -127,7 +136,7 @@ CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57 CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39 ``` -### Parsing `api_json.log` +### Parsing `gitlab-rails/api_json.log` #### Print top three routes with request count and their three longest durations @@ -143,8 +152,22 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` +### Parsing `gitlab-rails/geo.log` + +#### Find most common Geo sync errors + +If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task) +repeatedly reports that some items never reach 100%, +the following command helps to focus on the most common errors. + +```shell +jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail +``` + ### Parsing `gitaly/current` +The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md). + #### Find all Gitaly requests sent from web UI ```shell @@ -185,7 +208,7 @@ jq --raw-output --slurp ' .[2]."grpc.time_ms", .[0]."grpc.request.glProjectPath" ] - | @sh' /var/log/gitlab/gitaly/current \ + | @sh' current \ | awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' ``` @@ -203,12 +226,12 @@ jq --raw-output --slurp ' #### Find all projects affected by a fatal Git problem ```shell -grep "fatal: " /var/log/gitlab/gitaly/current | \ +grep "fatal: " current | \ jq '."grpc.request.glProjectPath"' | \ sort | uniq ``` -### Parsing `gitlab-shell.log` +### Parsing `gitlab-shell/gitlab-shell.log` For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367). @@ -226,7 +249,7 @@ jq --raw-output --slurp ' | sort_by(-length) | limit(20; .[]) | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - /var/log/gitlab/gitlab-shell/gitlab-shell.log + gitlab-shell.log ``` Find the top 20 calls by project, user, and command: @@ -244,5 +267,5 @@ jq --raw-output --slurp ' | sort_by(-length) | limit(20; .[]) | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - /var/log/gitlab/gitlab-shell/gitlab-shell.log + gitlab-shell.log ``` diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index eabaa4217b5..f6d1e554aae 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -21,76 +21,77 @@ See also: The following API resources are available in the project context: -| Resource | Available endpoints | -|:------------------------------------------------------------------------|:--------------------| -| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Access tokens](resource_access_tokens.md) | `/projects/:id/access_tokens` (also available for groups) | -| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | -| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | -| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | -| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | -| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | -| [Debian distributions](packages/debian_project_distributions.md) | `/projects/:id/debian_distributions` (also available for groups) | -| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | -| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | -| [Deploy tokens](deploy_tokens.md) | `/projects/:id/deploy_tokens` (also available for groups and standalone) | -| [Deployments](deployments.md) | `/projects/:id/deployments` | +| Resource | Available endpoints | +|:------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | +| [Access tokens](resource_access_tokens.md) | `/projects/:id/access_tokens` (also available for groups) | +| [Agents](cluster_agents.md) | `/projects/:id/cluster_agents` | +| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | +| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | +| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | +| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | +| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | +| [Debian distributions](packages/debian_project_distributions.md) | `/projects/:id/debian_distributions` (also available for groups) | +| [Dependencies](dependencies.md) **(ULTIMATE)** | `/projects/:id/dependencies` | +| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Deploy tokens](deploy_tokens.md) | `/projects/:id/deploy_tokens` (also available for groups and standalone) | +| [Deployments](deployments.md) | `/projects/:id/deployments` | | [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | -| [Environments](environments.md) | `/projects/:id/environments` | -| [Error Tracking](error_tracking.md) | `/projects/:id/error_tracking/settings` | -| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | -| [Feature Flag User Lists](feature_flag_user_lists.md) | `/projects/:id/feature_flags_user_lists` | -| [Feature Flags](feature_flags.md) | `/projects/:id/feature_flags` | -| [Freeze Periods](freeze_periods.md) | `/projects/:id/freeze_periods` | -| [Integrations](integrations.md) (Formerly "services") | `/projects/:id/integrations` | -| [Invitations](invitations.md) | `/projects/:id/invitations` (also available for groups) | -| [Issue boards](boards.md) | `/projects/:id/boards` | -| [Issue links](issue_links.md) | `/projects/:id/issues/.../links` | -| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | -| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | -| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | -| [Labels](labels.md) | `/projects/:id/labels` | -| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | -| [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | -| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | -| [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | -| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | -| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | -| [Packages](packages.md) | `/projects/:id/packages` | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | -| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | -| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | -| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | -| [Project badges](project_badges.md) | `/projects/:id/badges` | -| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | -| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | -| [Project milestones](milestones.md) | `/projects/:id/milestones` | -| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | -| [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/templates` | -| [Project wikis](wikis.md) | `/projects/:id/wikis` | -| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | -| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | -| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | -| [Protected environments](protected_environments.md) | `/projects/:id/protected_environments` | -| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | -| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | -| [Releases](releases/index.md) | `/projects/:id/releases` | -| [Remote mirrors](remote_mirrors.md) | `/projects/:id/remote_mirrors` | -| [Repositories](repositories.md) | `/projects/:id/repository` | -| [Repository files](repository_files.md) | `/projects/:id/repository/files` | -| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | -| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | -| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | -| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Tags](tags.md) | `/projects/:id/repository/tags` | -| [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | -| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | -| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | -| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | -| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | +| [Environments](environments.md) | `/projects/:id/environments` | +| [Error Tracking](error_tracking.md) | `/projects/:id/error_tracking/settings` | +| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | +| [Feature Flag User Lists](feature_flag_user_lists.md) | `/projects/:id/feature_flags_user_lists` | +| [Feature Flags](feature_flags.md) | `/projects/:id/feature_flags` | +| [Freeze Periods](freeze_periods.md) | `/projects/:id/freeze_periods` | +| [Integrations](integrations.md) (Formerly "services") | `/projects/:id/integrations` | +| [Invitations](invitations.md) | `/projects/:id/invitations` (also available for groups) | +| [Issue boards](boards.md) | `/projects/:id/boards` | +| [Issue links](issue_links.md) | `/projects/:id/issues/.../links` | +| [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | +| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Iterations](iterations.md) **(PREMIUM)** | `/projects/:id/iterations` (also available for groups) | +| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | +| [Labels](labels.md) | `/projects/:id/labels` | +| [Managed licenses](managed_licenses.md) **(ULTIMATE)** | `/projects/:id/managed_licenses` | +| [Members](members.md) | `/projects/:id/members` (also available for groups) | +| [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | +| [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | +| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | +| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | +| [Packages](packages.md) | `/projects/:id/packages` | +| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | +| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | +| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | +| [Project badges](project_badges.md) | `/projects/:id/badges` | +| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | +| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | +| [Project milestones](milestones.md) | `/projects/:id/milestones` | +| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | +| [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/templates` | +| [Project wikis](wikis.md) | `/projects/:id/wikis` | +| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | +| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | +| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | +| [Protected environments](protected_environments.md) | `/projects/:id/protected_environments` | +| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | +| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | +| [Releases](releases/index.md) | `/projects/:id/releases` | +| [Remote mirrors](remote_mirrors.md) | `/projects/:id/remote_mirrors` | +| [Repositories](repositories.md) | `/projects/:id/repository` | +| [Repository files](repository_files.md) | `/projects/:id/repository/files` | +| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | +| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | +| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | +| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | +| [Tags](tags.md) | `/projects/:id/repository/tags` | +| [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | +| [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | +| [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | +| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | +| [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | ## Group resources @@ -103,10 +104,10 @@ The following API resources are available in the group context: | [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | | [Debian distributions](packages/debian_group_distributions.md) | `/groups/:id/-/packages/debian` (also available for projects) | | [Deploy tokens](deploy_tokens.md) | `/groups/:id/deploy_tokens` (also available for projects and standalone) | -| [Discussions](discussions.md) (threaded comments) **(ULTIMATE)** | `/groups/:id/epics/.../discussions` (also available for projects) | -| [Epic issues](epic_issues.md) **(ULTIMATE)** | `/groups/:id/epics/.../issues` | -| [Epic links](epic_links.md) **(ULTIMATE)** | `/groups/:id/epics/.../epics` | -| [Epics](epics.md) **(ULTIMATE)** | `/groups/:id/epics` | +| [Discussions](discussions.md) (comments and threads) | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **(PREMIUM)** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **(PREMIUM)** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **(PREMIUM)** | `/groups/:id/epics` | | [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | | [Group badges](group_badges.md) | `/groups/:id/badges` | | [Group issue boards](group_boards.md) | `/groups/:id/boards` | @@ -114,6 +115,7 @@ The following API resources are available in the group context: | [Group labels](group_labels.md) | `/groups/:id/labels` | | [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | | [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Group releases](group_releases.md) | `/groups/:id/releases`| | [Group wikis](group_wikis.md) **(PREMIUM)** | `/groups/:id/wikis` | | [Invitations](invitations.md) | `/groups/:id/invitations` (also available for projects) | | [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | diff --git a/doc/api/bulk_imports.md b/doc/api/bulk_imports.md index 2b71c83b224..cae23b35fbe 100644 --- a/doc/api/bulk_imports.md +++ b/doc/api/bulk_imports.md @@ -58,11 +58,12 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla GET /bulk_imports ``` -| Attribute | Type | Required | Description | -|:-----------|:--------|:---------|:---------------------------------------| -| `per_page` | integer | no | Number of records to return per page. | -| `page` | integer | no | Page to retrieve. | -| `status` | string | no | Import status. | +| Attribute | Type | Required | Description | +|:-----------|:--------|:---------|:--------------------------------------------------------------------------------------------| +| `per_page` | integer | no | Number of records to return per page. | +| `page` | integer | no | Page to retrieve. | +| `sort` | string | no | Return GitLab migration sorted in `asc` or `desc` order by creation date. Default is `desc` | +| `status` | string | no | Import status. | The status can be one of the following: @@ -100,11 +101,12 @@ curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab GET /bulk_imports/entities ``` -| Attribute | Type | Required | Description | -|:-----------|:--------|:---------|:---------------------------------------| -| `per_page` | integer | no | Number of records to return per page. | -| `page` | integer | no | Page to retrieve. | -| `status` | string | no | Import status. | +| Attribute | Type | Required | Description | +|:-----------|:--------|:---------|:-----------------------------------------------------------------------------------------------------| +| `per_page` | integer | no | Number of records to return per page. | +| `page` | integer | no | Page to retrieve. | +| `sort` | string | no | Return GitLab migration entities sorted in `asc` or `desc` order by creation date. Default is `desc` | +| `status` | string | no | Import status. | The status can be one of the following: @@ -184,11 +186,12 @@ curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab GET /bulk_imports/:id/entities ``` -| Attribute | Type | Required | Description | -|:-----------|:--------|:---------|:---------------------------------------| -| `per_page` | integer | no | Number of records to return per page. | -| `page` | integer | no | Page to retrieve. | -| `status` | string | no | Import status. | +| Attribute | Type | Required | Description | +|:-----------|:--------|:---------|:--------------------------------------------------------------------------------------------| +| `per_page` | integer | no | Number of records to return per page. | +| `page` | integer | no | Page to retrieve. | +| `sort` | string | no | Return GitLab migration sorted in `asc` or `desc` order by creation date. Default is `desc` | +| `status` | string | no | Import status. | The status can be one of the following: diff --git a/doc/api/cluster_agents.md b/doc/api/cluster_agents.md new file mode 100644 index 00000000000..37cc4a24342 --- /dev/null +++ b/doc/api/cluster_agents.md @@ -0,0 +1,238 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Agents API **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83270) in GitLab 14.10. + +Use the Agents API to work with the GitLab agent for Kubernetes. + +## List the agents for a project + +Returns the list of agents registered for the project. + +You must have at least the Developer role to use this endpoint. + +```plaintext +GET /projects/:id/cluster_agents +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-----------|-------------------|-----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) maintained by the authenticated user | + +Response: + +The response is a list of agents with the following fields: + +| Attribute | Type | Description | +|--------------------------------------|----------|------------------------------------------------------| +| `id` | integer | ID of the agent | +| `name` | string | Name of the agent | +| `config_project` | object | Object representing the project the agent belongs to | +| `config_project.id` | integer | ID of the project | +| `config_project.description` | string | Description of the project | +| `config_project.name` | string | Name of the project | +| `config_project.name_with_namespace` | string | Full name with namespace of the project | +| `config_project.path` | string | Path to the project | +| `config_project.path_with_namespace` | string | Full path with namespace to the project | +| `config_project.created_at` | string | ISO8601 datetime when the project was created | +| `created_at` | string | ISO8601 datetime when the agent was created | +| `created_by_user_id` | integer | ID of the user who created the agent | + +Example request: + +```shell +curl --header "Private-Token: <your_access_token>" "https://gitlab.example.com/api/v4/projects/20/cluster_agents" +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "agent-1", + "config_project": { + "id": 20, + "description": "", + "name": "test", + "name_with_namespace": "Administrator / test", + "path": "test", + "path_with_namespace": "root/test", + "created_at": "2022-03-20T20:42:40.221Z" + }, + "created_at": "2022-04-20T20:42:40.221Z", + "created_by_user_id": 42 + }, + { + "id": 2, + "name": "agent-2", + "config_project": { + "id": 20, + "description": "", + "name": "test", + "name_with_namespace": "Administrator / test", + "path": "test", + "path_with_namespace": "root/test", + "created_at": "2022-03-20T20:42:40.221Z" + }, + "created_at": "2022-04-20T20:42:40.221Z", + "created_by_user_id": 42 + } +] +``` + +## Get details about an agent + +Gets a single agent details. + +You must have at least the Developer role to use this endpoint. + +```shell +GET /projects/:id/cluster_agents/:agent_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) maintained by the authenticated user | +| `agent_id` | integer | yes | ID of the agent | + +Response: + +The response is a single agent with the following fields: + +| Attribute | Type | Description | +|--------------------------------------|---------|------------------------------------------------------| +| `id` | integer | ID of the agent | +| `name` | string | Name of the agent | +| `config_project` | object | Object representing the project the agent belongs to | +| `config_project.id` | integer | ID of the project | +| `config_project.description` | string | Description of the project | +| `config_project.name` | string | Name of the project | +| `config_project.name_with_namespace` | string | Full name with namespace of the project | +| `config_project.path` | string | Path to the project | +| `config_project.path_with_namespace` | string | Full path with namespace to the project | +| `config_project.created_at` | string | ISO8601 datetime when the project was created | +| `created_at` | string | ISO8601 datetime when the agent was created | +| `created_by_user_id` | integer | ID of the user who created the agent | + +Example request: + +```shell +curl --header "Private-Token: <your_access_token>" "https://gitlab.example.com/api/v4/projects/20/cluster_agents/1" +``` + +Example response: + +```json +{ + "id": 1, + "name": "agent-1", + "config_project": { + "id": 20, + "description": "", + "name": "test", + "name_with_namespace": "Administrator / test", + "path": "test", + "path_with_namespace": "root/test", + "created_at": "2022-03-20T20:42:40.221Z" + }, + "created_at": "2022-04-20T20:42:40.221Z", + "created_by_user_id": 42 +} +``` + +## Register an agent with a project + +Registers an agent to the project. + +You must have at least the Maintainer role to use this endpoint. + +```shell +POST /projects/:id/cluster_agents +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-----------|-------------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) maintained by the authenticated user | +| `name` | string | yes | Name for the agent | + +Response: + +The response is the new agent with the following fields: + +| Attribute | Type | Description | +|--------------------------------------|---------|------------------------------------------------------| +| `id` | integer | ID of the agent | +| `name` | string | Name of the agent | +| `config_project` | object | Object representing the project the agent belongs to | +| `config_project.id` | integer | ID of the project | +| `config_project.description` | string | Description of the project | +| `config_project.name` | string | Name of the project | +| `config_project.name_with_namespace` | string | Full name with namespace of the project | +| `config_project.path` | string | Path to the project | +| `config_project.path_with_namespace` | string | Full path with namespace to the project | +| `config_project.created_at` | string | ISO8601 datetime when the project was created | +| `created_at` | string | ISO8601 datetime when the agent was created | +| `created_by_user_id` | integer | ID of the user who created the agent | + +Example request: + +```shell +curl --header "Private-Token: <your_access_token>" "https://gitlab.example.com/api/v4/projects/20/cluster_agents" \ + -H "Content-Type:application/json" \ + -X POST --data '{"name":"some-agent"}' +``` + +Example response: + +```json +{ + "id": 1, + "name": "agent-1", + "config_project": { + "id": 20, + "description": "", + "name": "test", + "name_with_namespace": "Administrator / test", + "path": "test", + "path_with_namespace": "root/test", + "created_at": "2022-03-20T20:42:40.221Z" + }, + "created_at": "2022-04-20T20:42:40.221Z", + "created_by_user_id": 42 +} +``` + +## Delete a registered agent + +Deletes an existing agent registration. + +You must have at least the Maintainer role to use this endpoint. + +```plaintext +DELETE /projects/:id/cluster_agents/:agent_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|------------|-------------------|----------|-----------------------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) maintained by the authenticated user | +| `agent_id` | integer | yes | ID of the agent | + +Example request: + +```shell +curl --request DELETE --header "Private-Token: <your_access_token>" "https://gitlab.example.com/api/v4/projects/20/cluster_agents/1 +``` diff --git a/doc/api/commits.md b/doc/api/commits.md index d7be9559527..7be5bc3d985 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -49,11 +49,11 @@ Example response: "title": "Replace sanitize with escape once", "author_name": "Example User", "author_email": "user@example.com", - "authored_date": "2012-09-20T11:50:22+03:00", + "authored_date": "2021-09-20T11:50:22.001+00:00", "committer_name": "Administrator", "committer_email": "admin@example.com", - "committed_date": "2012-09-20T11:50:22+03:00", - "created_at": "2012-09-20T11:50:22+03:00", + "committed_date": "2021-09-20T11:50:22.001+00:00", + "created_at": "2021-09-20T11:50:22.001+00:00", "message": "Replace sanitize with escape once", "parent_ids": [ "6104942438c14ec7bd21c6cd5bd995272b3faff6" @@ -68,7 +68,7 @@ Example response: "author_email": "user@example.com", "committer_name": "ExampleName", "committer_email": "user@example.com", - "created_at": "2012-09-20T09:06:12+03:00", + "created_at": "2021-09-20T09:06:12.201+00:00", "message": "Sanitize for network graph", "parent_ids": [ "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba" @@ -234,10 +234,10 @@ Example response: "author_email": "user@example.com", "committer_name": "Dmitriy", "committer_email": "user@example.com", - "created_at": "2012-09-20T09:06:12+03:00", + "created_at": "2021-09-20T09:06:12.300+03:00", "message": "Sanitize for network graph", - "committed_date": "2012-09-20T09:06:12+03:00", - "authored_date": "2012-09-20T09:06:12+03:00", + "committed_date": "2021-09-20T09:06:12.300+03:00", + "authored_date": "2021-09-20T09:06:12.420+03:00", "parent_ids": [ "ae1d9fb46aa2b07ee9836d49862ec4e2c46fbbba" ], diff --git a/doc/api/deployments.md b/doc/api/deployments.md index 4a09f9a6605..fb255bfa226 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -265,7 +265,7 @@ Example response: } ``` -Deployments created by users on GitLab Premium or higher include the `approvals` and `pending_approval_count` properties: +When the [unified approval setting](../ci/environments/deployment_approvals.md#unified-approval-setting) is configured, deployments created by users on GitLab Premium or higher include the `approvals` and `pending_approval_count` properties: ```json { @@ -282,13 +282,56 @@ Deployments created by users on GitLab Premium or higher include the `approvals` "web_url": "http://localhost:3000/project_6_bot" }, "status": "approved", - "created_at": "2022-02-24T20:22:30.097Z" + "created_at": "2022-02-24T20:22:30.097Z", + "comment": "Looks good to me" } ], ... } ``` +When the [multiple approval rules](../ci/environments/deployment_approvals.md#multiple-approval-rules) is configured, deployments created by users on GitLab Premium or higher include the `approval_summary` property: + +```json +{ + "approval_summary": { + "rules": [ + { + "user_id": null, + "group_id": 134, + "access_level": null, + "access_level_description": "qa-group", + "required_approvals": 1, + "deployment_approvals": [] + }, + { + "user_id": null, + "group_id": 135, + "access_level": null, + "access_level_description": "security-group", + "required_approvals": 2, + "deployment_approvals": [ + { + "user": { + "id": 100, + "username": "security-user-1", + "name": "security user-1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e130fcd3a1681f41a3de69d10841afa9?s=80&d=identicon", + "web_url": "http://localhost:3000/security-user-1" + }, + "status": "approved", + "created_at": "2022-04-11T03:37:03.058Z", + "comment": null + } + ] + } + ] + } + ... +} +``` + ## Create a deployment ```plaintext @@ -342,20 +385,7 @@ Deployments created by users on GitLab Premium or higher include the `approvals` { "status": "created", "pending_approval_count": 0, - "approvals": [ - { - "user": { - "id": 49, - "username": "project_6_bot", - "name": "****", - "state": "active", - "avatar_url": "https://www.gravatar.com/avatar/e83ac685f68ea07553ad3054c738c709?s=80&d=identicon", - "web_url": "http://localhost:3000/project_6_bot" - }, - "status": "approved", - "created_at": "2022-02-24T20:22:30.097Z" - } - ], + "approvals": [], ... } ``` @@ -420,7 +450,8 @@ Deployments created by users on GitLab Premium or higher include the `approvals` "web_url": "http://localhost:3000/project_6_bot" }, "status": "approved", - "created_at": "2022-02-24T20:22:30.097Z" + "created_at": "2022-02-24T20:22:30.097Z", + "comment": "Looks good to me" } ], ... @@ -466,9 +497,10 @@ POST /projects/:id/deployments/:deployment_id/approval | `deployment_id` | integer | yes | The ID of the deployment. | | `status` | string | yes | The status of the approval (either `approved` or `rejected`). | | `comment` | string | no | A comment to go with the approval | +| `represented_as`| string | no | The name of the User/Group/Role to use for the approval, when the user belongs to [multiple approval rules](../ci/environments/deployment_approvals.md#multiple-approval-rules). | ```shell -curl --data "status=approved&comment=Looks good to me" \ +curl --data "status=approved&comment=Looks good to me&represented_as=security" \ --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/1/approval" ``` @@ -477,12 +509,12 @@ Example response: ```json { "user": { - "name": "Administrator", - "username": "root", - "id": 1, + "id": 100, + "username": "security-user-1", + "name": "security user-1", "state": "active", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "web_url": "http://localhost:3000/root" + "avatar_url": "https://www.gravatar.com/avatar/e130fcd3a1681f41a3de69d10841afa9?s=80&d=identicon", + "web_url": "http://localhost:3000/security-user-1" }, "status": "approved", "created_at": "2022-02-24T20:22:30.097Z", diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 5f750df4a48..c30c00cef67 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -11,13 +11,14 @@ Discussions are a set of related notes on: - Snippets - Issues -- Epics **(ULTIMATE)** +- [Epics](../user/group/epics/index.md) - Merge requests - Commits -This includes system notes, which are notes about changes to the object (for example, -when a milestone changes, a corresponding system note is added). Label notes are -not part of this API, but recorded as separate events in [resource label events](resource_label_events.md). +This includes [comments and threads](../user/discussions/index.md) and system notes. +System notes are notes about changes to the object (for example, when a milestone changes). +Label notes are not part of this API, but recorded as separate events in +[resource label events](resource_label_events.md). ## Discussions pagination @@ -993,7 +994,11 @@ curl --request POST --header "PRIVATE-TOKEN: [ACCESS_TOKEN]"\ ### Resolve a merge request thread -Resolve/unresolve whole thread of a merge request. +Resolve or unresolve a thread of discussion in a merge request. + +Prerequisite: + +- You must have at least the Developer role, or be the author of the change being reviewed. ```plaintext PUT /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index afc29f03598..f5373d02156 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -24,7 +24,7 @@ GET /projects/:id/dora/metrics | Attribute | Type | Required | Description | |-------------- |-------- |----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | -| `metric` | string | yes | The [metric name](../../user/analytics/ci_cd_analytics.md#supported-metrics-in-gitlab). One of `deployment_frequency`, `lead_time_for_changes` or `time_to_restore_service`.| +| `metric` | string | yes | The metric name: `deployment_frequency`, `lead_time_for_changes` or `time_to_restore_service`.| | `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | | `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | | `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | @@ -61,14 +61,14 @@ Get group-level DORA metrics. GET /groups/:id/dora/metrics ``` -| Attribute | Type | Required | Description | -|-------------- |-------- |----------|----------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | -| `metric` | string | yes | The [metric name](../../user/analytics/ci_cd_analytics.md#supported-metrics-in-gitlab). One of `deployment_frequency`, `lead_time_for_changes` or `time_to_restore_service`. | -| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | -| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | -| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | -| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | +| Attribute | Type | Required | Description | +|-------------- |-------- |----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | +| `metric` | string | yes | One of `deployment_frequency`, `lead_time_for_changes`, `time_to_restore_service` or `change_failure_rate`. | +| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | +| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | +| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | +| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | Example request: @@ -101,4 +101,5 @@ parameter: | ------------------------ |--------------------------------------------------------------------------------------------------------------------------------------------------------------| | `deployment_frequency` | The number of successful deployments during the time period. | | `lead_time_for_changes` | The median number of seconds between the merge of the merge request (MR) and the deployment of the MR's commits for all MRs deployed during the time period. | -| `time_to_restore_service` | The median number of seconds an incident was open during the time period. Available only for production environment | +| `time_to_restore_service` | The median number of seconds an incident was open during the time period. Available only for production environment. | +| `change_failure_rate` | The number of incidents divided by the number of deployments during the time period. Available only for production environment. | diff --git a/doc/api/events.md b/doc/api/events.md index 265fc0e5fd2..3f032c72870 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ### Actions -See [User contribution events](../user/index.md#user-contribution-events) for available types for the `action` parameter. +See [User contribution events](../user/profile/index.md#user-contribution-events) for available types for the `action` parameter. These options are in lowercase. ### Target Types diff --git a/doc/api/feature_flags_legacy.md b/doc/api/feature_flags_legacy.md index 262e1c537a4..1cf05144a1b 100644 --- a/doc/api/feature_flags_legacy.md +++ b/doc/api/feature_flags_legacy.md @@ -1,13 +1,11 @@ --- -stage: Release -group: Release -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'feature_flags.md' +remove_date: '2022-06-22' --- -# Legacy Feature Flags API **(FREE)** +This document was moved to [another location](feature_flags). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in GitLab Premium 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to GitLab Free in 13.5. - -This API was removed in [GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). -Please use [the new API](feature_flags.md) instead. +<!-- This redirect file can be deleted after <2022-06-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/api/features.md b/doc/api/features.md index 593e4adedd7..0ad76829651 100644 --- a/doc/api/features.md +++ b/doc/api/features.md @@ -89,22 +89,24 @@ Example response: ```json [ { - "name": "api_kaminari_count_with_limit", - "introduced_by_url": "https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23931", - "rollout_issue_url": null, - "milestone": "11.8", - "type": "ops", - "group": "group::ecosystem", + "name": "geo_pages_deployment_replication", + "introduced_by_url": "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68662", + "rollout_issue_url": "https://gitlab.com/gitlab-org/gitlab/-/issues/337676", + "milestone": "14.3", + "log_state_changes": null, + "type": "development", + "group": "group::geo", "default_enabled": true }, { - "name": "marginalia", - "introduced_by_url": null, - "rollout_issue_url": null, - "milestone": null, - "type": "ops", - "group": null, - "default_enabled": false + "name": "analytics_devops_adoption_codeowners", + "introduced_by_url": "https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59874", + "rollout_issue_url": "https://gitlab.com/gitlab-org/gitlab/-/issues/328542", + "milestone": "13.12", + "log_state_changes": null, + "type": "development", + "group": "group::optimize", + "default_enabled": true } ] ``` diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index d2cca1a5856..93478bcf95f 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -75,7 +75,7 @@ Example response: WARNING: The `web_geo_projects_url` attribute is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80106) -for use in GitLab 14.9. +in GitLab 14.9. ## Retrieve configuration about all Geo nodes @@ -147,7 +147,7 @@ Example response: WARNING: The `web_geo_projects_url` attribute is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80106) -for use in GitLab 14.9. +in GitLab 14.9. ## Retrieve configuration about a specific Geo node @@ -249,7 +249,7 @@ Example response: WARNING: The `web_geo_projects_url` attribute is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80106) -for use in GitLab 14.9. +in GitLab 14.9. ## Delete a Geo node diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index f86ad985b5e..7ef9897a170 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -628,6 +628,8 @@ Input type: `AdminSidekiqQueuesDeleteJobsInput` | <a id="mutationadminsidekiqqueuesdeletejobsclientid"></a>`clientId` | [`String`](#string) | Delete jobs matching client_id in the context metadata. | | <a id="mutationadminsidekiqqueuesdeletejobsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationadminsidekiqqueuesdeletejobsfeaturecategory"></a>`featureCategory` | [`String`](#string) | Delete jobs matching feature_category in the context metadata. | +| <a id="mutationadminsidekiqqueuesdeletejobsjobid"></a>`jobId` | [`String`](#string) | Delete jobs matching job_id in the context metadata. | +| <a id="mutationadminsidekiqqueuesdeletejobspipelineid"></a>`pipelineId` | [`String`](#string) | Delete jobs matching pipeline_id in the context metadata. | | <a id="mutationadminsidekiqqueuesdeletejobsproject"></a>`project` | [`String`](#string) | Delete jobs matching project in the context metadata. | | <a id="mutationadminsidekiqqueuesdeletejobsqueuename"></a>`queueName` | [`String!`](#string) | Name of the queue to delete jobs from. | | <a id="mutationadminsidekiqqueuesdeletejobsrelatedclass"></a>`relatedClass` | [`String`](#string) | Delete jobs matching related_class in the context metadata. | @@ -1456,7 +1458,7 @@ Input type: `CreateIssueInput` WARNING: **Deprecated** in 14.0. -Use iterationCreate. +Manual iteration management is deprecated. Only automatic iteration cadences will be supported in the future. Input type: `CreateIterationInput` @@ -1468,7 +1470,7 @@ Input type: `CreateIterationInput` | <a id="mutationcreateiterationdescription"></a>`description` | [`String`](#string) | Description of the iteration. | | <a id="mutationcreateiterationduedate"></a>`dueDate` | [`String`](#string) | End date of the iteration. | | <a id="mutationcreateiterationgrouppath"></a>`groupPath` | [`ID`](#id) | Full path of the group with which the resource is associated. | -| <a id="mutationcreateiterationiterationscadenceid"></a>`iterationsCadenceId` | [`IterationsCadenceID`](#iterationscadenceid) | Global ID of the iterations cadence to be assigned to newly created iteration. | +| <a id="mutationcreateiterationiterationscadenceid"></a>`iterationsCadenceId` **{warning-solid}** | [`IterationsCadenceID`](#iterationscadenceid) | **Deprecated:** `iterationCadenceId` is deprecated and will be removed in the future. This argument is ignored, because you can't create an iteration in a specific cadence. In the future only automatic iteration cadences will be allowed. Deprecated in 14.10. | | <a id="mutationcreateiterationprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project with which the resource is associated. | | <a id="mutationcreateiterationstartdate"></a>`startDate` | [`String`](#string) | Start date of the iteration. | | <a id="mutationcreateiterationtitle"></a>`title` | [`String`](#string) | Title of the iteration. | @@ -1806,8 +1808,9 @@ Input type: `DastScannerProfileCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastscannerprofilecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdastscannerprofilecreatedastscannerprofile"></a>`dastScannerProfile` | [`DastScannerProfile`](#dastscannerprofile) | Created scanner profile. | | <a id="mutationdastscannerprofilecreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdastscannerprofilecreateid"></a>`id` | [`DastScannerProfileID`](#dastscannerprofileid) | ID of the scanner profile. | +| <a id="mutationdastscannerprofilecreateid"></a>`id` **{warning-solid}** | [`DastScannerProfileID`](#dastscannerprofileid) | **Deprecated:** use `dastScannerProfile` field. Deprecated in 14.10. | ### `Mutation.dastScannerProfileDelete` @@ -1851,8 +1854,9 @@ Input type: `DastScannerProfileUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastscannerprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdastscannerprofileupdatedastscannerprofile"></a>`dastScannerProfile` | [`DastScannerProfile`](#dastscannerprofile) | Updated scanner profile. | | <a id="mutationdastscannerprofileupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdastscannerprofileupdateid"></a>`id` | [`DastScannerProfileID`](#dastscannerprofileid) | ID of the scanner profile. | +| <a id="mutationdastscannerprofileupdateid"></a>`id` **{warning-solid}** | [`DastScannerProfileID`](#dastscannerprofileid) | **Deprecated:** use `dastScannerProfile` field. Deprecated in 14.10. | ### `Mutation.dastSiteProfileCreate` @@ -1877,8 +1881,9 @@ Input type: `DastSiteProfileCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastsiteprofilecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdastsiteprofilecreatedastsiteprofile"></a>`dastSiteProfile` | [`DastSiteProfile`](#dastsiteprofile) | Site Profile object. | | <a id="mutationdastsiteprofilecreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdastsiteprofilecreateid"></a>`id` | [`DastSiteProfileID`](#dastsiteprofileid) | ID of the site profile. | +| <a id="mutationdastsiteprofilecreateid"></a>`id` **{warning-solid}** | [`DastSiteProfileID`](#dastsiteprofileid) | **Deprecated:** use `dastSiteProfile.id` field. Deprecated in 14.10. | ### `Mutation.dastSiteProfileDelete` @@ -1923,8 +1928,9 @@ Input type: `DastSiteProfileUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastsiteprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdastsiteprofileupdatedastsiteprofile"></a>`dastSiteProfile` | [`DastSiteProfile`](#dastsiteprofile) | Site profile object. | | <a id="mutationdastsiteprofileupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdastsiteprofileupdateid"></a>`id` | [`DastSiteProfileID`](#dastsiteprofileid) | ID of the site profile. | +| <a id="mutationdastsiteprofileupdateid"></a>`id` **{warning-solid}** | [`DastSiteProfileID`](#dastsiteprofileid) | **Deprecated:** use `dastSiteProfile.id` field. Deprecated in 14.10. | ### `Mutation.dastSiteTokenCreate` @@ -2372,6 +2378,8 @@ Input type: `EnableDevopsAdoptionNamespaceInput` ### `Mutation.environmentsCanaryIngressUpdate` +**Deprecated** This endpoint is planned to be removed along with certificate-based clusters. [See this epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) for more information. + Input type: `EnvironmentsCanaryIngressUpdateInput` #### Arguments @@ -3209,6 +3217,10 @@ Input type: `IterationCadenceUpdateInput` ### `Mutation.iterationCreate` +WARNING: +**Deprecated** in 14.10. +Manual iteration management is deprecated. Only automatic iteration cadences will be supported in the future. + Input type: `iterationCreateInput` #### Arguments @@ -3219,7 +3231,7 @@ Input type: `iterationCreateInput` | <a id="mutationiterationcreatedescription"></a>`description` | [`String`](#string) | Description of the iteration. | | <a id="mutationiterationcreateduedate"></a>`dueDate` | [`String`](#string) | End date of the iteration. | | <a id="mutationiterationcreategrouppath"></a>`groupPath` | [`ID`](#id) | Full path of the group with which the resource is associated. | -| <a id="mutationiterationcreateiterationscadenceid"></a>`iterationsCadenceId` | [`IterationsCadenceID`](#iterationscadenceid) | Global ID of the iterations cadence to be assigned to newly created iteration. | +| <a id="mutationiterationcreateiterationscadenceid"></a>`iterationsCadenceId` **{warning-solid}** | [`IterationsCadenceID`](#iterationscadenceid) | **Deprecated:** `iterationCadenceId` is deprecated and will be removed in the future. This argument is ignored, because you can't create an iteration in a specific cadence. In the future only automatic iteration cadences will be allowed. Deprecated in 14.10. | | <a id="mutationiterationcreateprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project with which the resource is associated. | | <a id="mutationiterationcreatestartdate"></a>`startDate` | [`String`](#string) | Start date of the iteration. | | <a id="mutationiterationcreatetitle"></a>`title` | [`String`](#string) | Title of the iteration. | @@ -3234,6 +3246,10 @@ Input type: `iterationCreateInput` ### `Mutation.iterationDelete` +WARNING: +**Deprecated** in 14.10. +Manual iteration management is deprecated. Only automatic iteration cadences will be supported in the future. + Input type: `IterationDeleteInput` #### Arguments @@ -4261,7 +4277,26 @@ Input type: `SavedReplyCreateInput` | ---- | ---- | ----------- | | <a id="mutationsavedreplycreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationsavedreplycreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationsavedreplycreatesavedreply"></a>`savedReply` | [`SavedReply`](#savedreply) | Updated saved reply. | +| <a id="mutationsavedreplycreatesavedreply"></a>`savedReply` | [`SavedReply`](#savedreply) | Saved reply after mutation. | + +### `Mutation.savedReplyDestroy` + +Input type: `SavedReplyDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsavedreplydestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsavedreplydestroyid"></a>`id` | [`UsersSavedReplyID!`](#userssavedreplyid) | Global ID of the saved reply. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsavedreplydestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsavedreplydestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationsavedreplydestroysavedreply"></a>`savedReply` | [`SavedReply`](#savedreply) | Saved reply after mutation. | ### `Mutation.savedReplyUpdate` @@ -4282,11 +4317,11 @@ Input type: `SavedReplyUpdateInput` | ---- | ---- | ----------- | | <a id="mutationsavedreplyupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationsavedreplyupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationsavedreplyupdatesavedreply"></a>`savedReply` | [`SavedReply`](#savedreply) | Updated saved reply. | +| <a id="mutationsavedreplyupdatesavedreply"></a>`savedReply` | [`SavedReply`](#savedreply) | Saved reply after mutation. | ### `Mutation.scanExecutionPolicyCommit` -Commits the `policy_yaml` content to the assigned security policy project for the given project(`project_path`). +Commits the `policy_yaml` content to the assigned security policy project for the given project (`full_path`). Input type: `ScanExecutionPolicyCommitInput` @@ -4295,10 +4330,11 @@ Input type: `ScanExecutionPolicyCommitInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationscanexecutionpolicycommitclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationscanexecutionpolicycommitfullpath"></a>`fullPath` | [`String`](#string) | Full path of the project. | | <a id="mutationscanexecutionpolicycommitname"></a>`name` | [`String`](#string) | Name of the policy. If the name is null, the `name` field from `policy_yaml` is used. | | <a id="mutationscanexecutionpolicycommitoperationmode"></a>`operationMode` | [`MutationOperationMode!`](#mutationoperationmode) | Changes the operation mode. | | <a id="mutationscanexecutionpolicycommitpolicyyaml"></a>`policyYaml` | [`String!`](#string) | YAML snippet of the policy. | -| <a id="mutationscanexecutionpolicycommitprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | +| <a id="mutationscanexecutionpolicycommitprojectpath"></a>`projectPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Use `fullPath`. Deprecated in 14.10. | #### Fields @@ -4310,7 +4346,7 @@ Input type: `ScanExecutionPolicyCommitInput` ### `Mutation.securityPolicyProjectAssign` -Assigns the specified project(`security_policy_project_id`) as security policy project for the given project(`project_path`). If the project already has a security policy project, this reassigns the project's security policy project with the given `security_policy_project_id`. +Assigns the specified project(`security_policy_project_id`) as security policy project for the given project(`full_path`). If the project already has a security policy project, this reassigns the project's security policy project with the given `security_policy_project_id`. Input type: `SecurityPolicyProjectAssignInput` @@ -4319,7 +4355,8 @@ Input type: `SecurityPolicyProjectAssignInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationsecuritypolicyprojectassignclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationsecuritypolicyprojectassignprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectassignfullpath"></a>`fullPath` | [`String`](#string) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectassignprojectpath"></a>`projectPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Use `fullPath`. Deprecated in 14.10. | | <a id="mutationsecuritypolicyprojectassignsecuritypolicyprojectid"></a>`securityPolicyProjectId` | [`ProjectID!`](#projectid) | ID of the security policy project. | #### Fields @@ -4331,7 +4368,7 @@ Input type: `SecurityPolicyProjectAssignInput` ### `Mutation.securityPolicyProjectCreate` -Creates and assigns a security policy project for the given project(`project_path`). +Creates and assigns a security policy project for the given project (`full_path`). Input type: `SecurityPolicyProjectCreateInput` @@ -4340,7 +4377,8 @@ Input type: `SecurityPolicyProjectCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationsecuritypolicyprojectcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationsecuritypolicyprojectcreateprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectcreatefullpath"></a>`fullPath` | [`String`](#string) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectcreateprojectpath"></a>`projectPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Use `fullPath`. Deprecated in 14.10. | #### Fields @@ -4352,7 +4390,7 @@ Input type: `SecurityPolicyProjectCreateInput` ### `Mutation.securityPolicyProjectUnassign` -Unassigns the security policy project for the given project(`project_path`). +Unassigns the security policy project for the given project (`full_path`). Input type: `SecurityPolicyProjectUnassignInput` @@ -4361,7 +4399,8 @@ Input type: `SecurityPolicyProjectUnassignInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationsecuritypolicyprojectunassignclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationsecuritypolicyprojectunassignprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectunassignfullpath"></a>`fullPath` | [`String`](#string) | Full path of the project. | +| <a id="mutationsecuritypolicyprojectunassignprojectpath"></a>`projectPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Use `fullPath`. Deprecated in 14.10. | #### Fields @@ -4611,6 +4650,7 @@ Input type: `TodosMarkAllDoneInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationtodosmarkalldoneclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationtodosmarkalldonetargetid"></a>`targetId` | [`TodoableID`](#todoableid) | Global ID of the to-do item's parent. Issues, merge requests, designs, and epics are supported. If argument is omitted, all pending to-do items of the current user are marked as done. | #### Fields @@ -4922,11 +4962,11 @@ Input type: `UpdateIterationInput` | ---- | ---- | ----------- | | <a id="mutationupdateiterationclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationupdateiterationdescription"></a>`description` | [`String`](#string) | Description of the iteration. | -| <a id="mutationupdateiterationduedate"></a>`dueDate` | [`String`](#string) | End date of the iteration. | +| <a id="mutationupdateiterationduedate"></a>`dueDate` **{warning-solid}** | [`String`](#string) | **Deprecated:** Manual iteration updates are deprecated, only `description` updates will be allowed in the future. Deprecated in 14.10. | | <a id="mutationupdateiterationgrouppath"></a>`groupPath` | [`ID!`](#id) | Group of the iteration. | | <a id="mutationupdateiterationid"></a>`id` | [`ID!`](#id) | Global ID of the iteration. | -| <a id="mutationupdateiterationstartdate"></a>`startDate` | [`String`](#string) | Start date of the iteration. | -| <a id="mutationupdateiterationtitle"></a>`title` | [`String`](#string) | Title of the iteration. | +| <a id="mutationupdateiterationstartdate"></a>`startDate` **{warning-solid}** | [`String`](#string) | **Deprecated:** Manual iteration updates are deprecated, only `description` updates will be allowed in the future. Deprecated in 14.10. | +| <a id="mutationupdateiterationtitle"></a>`title` **{warning-solid}** | [`String`](#string) | **Deprecated:** Manual iteration updates are deprecated, only `description` updates will be allowed in the future. Deprecated in 14.10. | #### Fields @@ -4974,7 +5014,7 @@ Input type: `UpdateNoteInput` | ---- | ---- | ----------- | | <a id="mutationupdatenotebody"></a>`body` | [`String`](#string) | Content of the note. | | <a id="mutationupdatenoteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationupdatenoteconfidential"></a>`confidential` | [`Boolean`](#boolean) | Confidentiality flag of a note. Default is false. | +| <a id="mutationupdatenoteconfidential"></a>`confidential` **{warning-solid}** | [`Boolean`](#boolean) | **Deprecated:** No longer allowed to update confidentiality of notes. Deprecated in 14.10. | | <a id="mutationupdatenoteid"></a>`id` | [`NoteID!`](#noteid) | Global ID of the note to update. | #### Fields @@ -9317,6 +9357,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cijobduration"></a>`duration` | [`Int`](#int) | Duration of the job in seconds. | | <a id="cijobfinishedat"></a>`finishedAt` | [`Time`](#time) | When a job has finished running. | | <a id="cijobid"></a>`id` | [`JobID`](#jobid) | ID of the job. | +| <a id="cijobkind"></a>`kind` | [`CiJobKind!`](#cijobkind) | Indicates the type of job. | | <a id="cijobmanualjob"></a>`manualJob` | [`Boolean`](#boolean) | Whether the job has a manual action. | | <a id="cijobname"></a>`name` | [`String`](#string) | Name of the job. | | <a id="cijobneeds"></a>`needs` | [`CiBuildNeedConnection`](#cibuildneedconnection) | References to builds that must complete before the jobs run. (see [Connections](#connections)) | @@ -9409,6 +9450,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cirunnershortsha"></a>`shortSha` | [`String`](#string) | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID. | | <a id="cirunnertaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | | <a id="cirunnertokenexpiresat"></a>`tokenExpiresAt` | [`Time`](#time) | Runner token expiration time. | +| <a id="cirunnerupgradestatus"></a>`upgradeStatus` **{warning-solid}** | [`CiRunnerUpgradeStatusType`](#cirunnerupgradestatustype) | **Deprecated** in 14.10. This feature is in Alpha, and can be removed or changed at any point. | | <a id="cirunneruserpermissions"></a>`userPermissions` | [`RunnerPermissions!`](#runnerpermissions) | Permissions for the current user on the resource. | | <a id="cirunnerversion"></a>`version` | [`String`](#string) | Version of the runner. | @@ -9731,6 +9773,7 @@ A container repository. | <a id="containerrepositoryexpirationpolicystartedat"></a>`expirationPolicyStartedAt` | [`Time`](#time) | Timestamp when the cleanup done by the expiration policy was started on the container repository. | | <a id="containerrepositoryid"></a>`id` | [`ID!`](#id) | ID of the container repository. | | <a id="containerrepositorylocation"></a>`location` | [`String!`](#string) | URL of the container repository. | +| <a id="containerrepositorymigrationstate"></a>`migrationState` | [`String!`](#string) | Migration state of the container repository. | | <a id="containerrepositoryname"></a>`name` | [`String!`](#string) | Name of the container repository. | | <a id="containerrepositorypath"></a>`path` | [`String!`](#string) | Path of the container repository. | | <a id="containerrepositoryproject"></a>`project` | [`Project!`](#project) | Project of the container registry. | @@ -9752,6 +9795,7 @@ Details of a container repository. | <a id="containerrepositorydetailsexpirationpolicystartedat"></a>`expirationPolicyStartedAt` | [`Time`](#time) | Timestamp when the cleanup done by the expiration policy was started on the container repository. | | <a id="containerrepositorydetailsid"></a>`id` | [`ID!`](#id) | ID of the container repository. | | <a id="containerrepositorydetailslocation"></a>`location` | [`String!`](#string) | URL of the container repository. | +| <a id="containerrepositorydetailsmigrationstate"></a>`migrationState` | [`String!`](#string) | Migration state of the container repository. | | <a id="containerrepositorydetailsname"></a>`name` | [`String!`](#string) | Name of the container repository. | | <a id="containerrepositorydetailspath"></a>`path` | [`String!`](#string) | Path of the container repository. | | <a id="containerrepositorydetailsproject"></a>`project` | [`Project!`](#project) | Project of the container registry. | @@ -9981,6 +10025,7 @@ Input type for DastSiteProfile authentication. | <a id="dastsiteprofileauthenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates whether authentication is enabled. | | <a id="dastsiteprofileauthpassword"></a>`password` | [`String`](#string) | Redacted password to authenticate with on the target website. | | <a id="dastsiteprofileauthpasswordfield"></a>`passwordField` | [`String`](#string) | Name of password field at the sign-in HTML form. | +| <a id="dastsiteprofileauthsubmitfield"></a>`submitField` | [`String`](#string) | Name or ID of sign-in submit button at the sign-in HTML form. | | <a id="dastsiteprofileauthurl"></a>`url` | [`String`](#string) | The URL of the page containing the sign-in HTML form on the target website. | | <a id="dastsiteprofileauthusername"></a>`username` | [`String`](#string) | Username to authenticate with on the target website. | | <a id="dastsiteprofileauthusernamefield"></a>`usernameField` | [`String`](#string) | Name of username field at the sign-in HTML form. | @@ -10059,6 +10104,7 @@ Dependency proxy manifest. | <a id="dependencyproxymanifestid"></a>`id` | [`DependencyProxyManifestID!`](#dependencyproxymanifestid) | ID of the manifest. | | <a id="dependencyproxymanifestimagename"></a>`imageName` | [`String!`](#string) | Name of the image. | | <a id="dependencyproxymanifestsize"></a>`size` | [`String!`](#string) | Size of the manifest file. | +| <a id="dependencyproxymanifeststatus"></a>`status` | [`DependencyProxyManifestStatus!`](#dependencyproxymanifeststatus) | Status of the manifest (default, pending_destruction, processing, error). | | <a id="dependencyproxymanifestupdatedat"></a>`updatedAt` | [`Time!`](#time) | Date of most recent update. | ### `DependencyProxySetting` @@ -10381,7 +10427,6 @@ Snapshot. | <a id="devopsadoptionsnapshotrecordedat"></a>`recordedAt` | [`Time!`](#time) | Time the snapshot was recorded. | | <a id="devopsadoptionsnapshotrunnerconfigured"></a>`runnerConfigured` | [`Boolean!`](#boolean) | At least one runner was used. | | <a id="devopsadoptionsnapshotsastenabledcount"></a>`sastEnabledCount` | [`Int`](#int) | Total number of projects with enabled SAST. | -| <a id="devopsadoptionsnapshotsecurityscansucceeded"></a>`securityScanSucceeded` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 14.1. Substituted with specific security metrics. Always false. | | <a id="devopsadoptionsnapshotstarttime"></a>`startTime` | [`Time!`](#time) | Start time for the snapshot where the data points were collected. | | <a id="devopsadoptionsnapshottotalprojectscount"></a>`totalProjectsCount` | [`Int`](#int) | Total number of projects. | | <a id="devopsadoptionsnapshotvulnerabilitymanagementusedcount"></a>`vulnerabilityManagementUsedCount` | [`Int`](#int) | Total number of projects with vulnerability management used at least once. | @@ -11024,7 +11069,7 @@ four standard [pagination arguments](#connection-pagination-arguments): ##### `GeoNode.jobArtifactRegistries` -Find Job Artifact registries on this Geo node Available only when feature flag `geo_job_artifact_replication` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. +Find Job Artifact registries on this Geo node. Returns [`JobArtifactRegistryConnection`](#jobartifactregistryconnection). @@ -12590,7 +12635,7 @@ A user assigned to a merge request. | <a id="mergerequestassigneeprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="mergerequestassigneeprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestassigneepublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="mergerequestassigneesavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="mergerequestassigneesavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestassigneestate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestassigneestatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="mergerequestassigneeuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -12820,7 +12865,7 @@ The author of the merge request. | <a id="mergerequestauthorprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="mergerequestauthorprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestauthorpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="mergerequestauthorsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="mergerequestauthorsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestauthorstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestauthorstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="mergerequestauthoruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -13067,7 +13112,7 @@ A user participating in a merge request. | <a id="mergerequestparticipantprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="mergerequestparticipantprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestparticipantpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="mergerequestparticipantsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="mergerequestparticipantsavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestparticipantstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestparticipantstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="mergerequestparticipantuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -13315,7 +13360,7 @@ A user assigned to a merge request as a reviewer. | <a id="mergerequestreviewerprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="mergerequestreviewerprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="mergerequestreviewerpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="mergerequestreviewersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="mergerequestreviewersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="mergerequestreviewerstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="mergerequestreviewerstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="mergerequestrevieweruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -16020,6 +16065,7 @@ Represents a URL related to a security training. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="securitytrainingurlidentifier"></a>`identifier` | [`String`](#string) | Name of the vulnerability identifier. | | <a id="securitytrainingurlname"></a>`name` | [`String`](#string) | Name of the training provider. | | <a id="securitytrainingurlstatus"></a>`status` | [`TrainingUrlRequestStatus`](#trainingurlrequeststatus) | Status of the request to training provider. | | <a id="securitytrainingurlurl"></a>`url` | [`String`](#string) | URL of the link for security training content. | @@ -16694,7 +16740,7 @@ Core represention of a GitLab user. | <a id="usercoreprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="usercoreprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="usercorepublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="usercoresavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="usercoresavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="usercorestate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="usercorestatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="usercoreuserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -17736,6 +17782,13 @@ Values for YAML processor result. | <a id="ciconfigstatusinvalid"></a>`INVALID` | Configuration file is not valid. | | <a id="ciconfigstatusvalid"></a>`VALID` | Configuration file is valid. | +### `CiJobKind` + +| Value | Description | +| ----- | ----------- | +| <a id="cijobkindbridge"></a>`BRIDGE` | Bridge CI job connecting a parent and child pipeline. | +| <a id="cijobkindbuild"></a>`BUILD` | Standard CI job. | + ### `CiJobStatus` | Value | Description | @@ -17792,6 +17845,14 @@ Values for sorting runners. | <a id="cirunnertypeinstance_type"></a>`INSTANCE_TYPE` | A runner that is instance type. | | <a id="cirunnertypeproject_type"></a>`PROJECT_TYPE` | A runner that is project type. | +### `CiRunnerUpgradeStatusType` + +| Value | Description | +| ----- | ----------- | +| <a id="cirunnerupgradestatustypeavailable"></a>`AVAILABLE` | An update is available for the runner. | +| <a id="cirunnerupgradestatustypenot_available"></a>`NOT_AVAILABLE` | An update is not available for the runner. | +| <a id="cirunnerupgradestatustyperecommended"></a>`RECOMMENDED` | An update is available and recommended for the runner. | + ### `CodeQualityDegradationSeverity` | Value | Description | @@ -18038,6 +18099,15 @@ Weight of the data visualization palette. | <a id="datavisualizationweightenumweight_900"></a>`WEIGHT_900` | 900 weight. | | <a id="datavisualizationweightenumweight_950"></a>`WEIGHT_950` | 950 weight. | +### `DependencyProxyManifestStatus` + +| Value | Description | +| ----- | ----------- | +| <a id="dependencyproxymanifeststatusdefault"></a>`DEFAULT` | Dependency proxy manifest has a status of default. | +| <a id="dependencyproxymanifeststatuserror"></a>`ERROR` | Dependency proxy manifest has a status of error. | +| <a id="dependencyproxymanifeststatuspending_destruction"></a>`PENDING_DESTRUCTION` | Dependency proxy manifest has a status of pending_destruction. | +| <a id="dependencyproxymanifeststatusprocessing"></a>`PROCESSING` | Dependency proxy manifest has a status of processing. | + ### `DeploymentTier` All environment deployment tiers. @@ -18096,6 +18166,7 @@ All supported DORA metric types. | Value | Description | | ----- | ----------- | +| <a id="dorametrictypechange_failure_rate"></a>`CHANGE_FAILURE_RATE` | Change failure rate. | | <a id="dorametrictypedeployment_frequency"></a>`DEPLOYMENT_FREQUENCY` | Deployment frequency. | | <a id="dorametrictypelead_time_for_changes"></a>`LEAD_TIME_FOR_CHANGES` | Lead time for changes. | | <a id="dorametrictypetime_to_restore_service"></a>`TIME_TO_RESTORE_SERVICE` | Time to restore service. | @@ -18279,6 +18350,8 @@ Values for sorting issues. | <a id="issuesortcreated_desc"></a>`CREATED_DESC` | Created at descending order. | | <a id="issuesortdue_date_asc"></a>`DUE_DATE_ASC` | Due date by ascending order. | | <a id="issuesortdue_date_desc"></a>`DUE_DATE_DESC` | Due date by descending order. | +| <a id="issuesortescalation_status_asc"></a>`ESCALATION_STATUS_ASC` | Status from triggered to resolved. Defaults to `CREATED_DESC` if `incident_escalations` feature flag is disabled. | +| <a id="issuesortescalation_status_desc"></a>`ESCALATION_STATUS_DESC` | Status from resolved to triggered. Defaults to `CREATED_DESC` if `incident_escalations` feature flag is disabled. | | <a id="issuesortlabel_priority_asc"></a>`LABEL_PRIORITY_ASC` | Label priority by ascending order. | | <a id="issuesortlabel_priority_desc"></a>`LABEL_PRIORITY_DESC` | Label priority by descending order. | | <a id="issuesortmilestone_due_asc"></a>`MILESTONE_DUE_ASC` | Milestone due date by ascending order. | @@ -19044,6 +19117,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumgeo_migrate_hashed_storage"></a>`GEO_MIGRATE_HASHED_STORAGE` | Callout feature name for geo_migrate_hashed_storage. | | <a id="usercalloutfeaturenameenumgke_cluster_integration"></a>`GKE_CLUSTER_INTEGRATION` | Callout feature name for gke_cluster_integration. | | <a id="usercalloutfeaturenameenumgold_trial_billings"></a>`GOLD_TRIAL_BILLINGS` | Callout feature name for gold_trial_billings. | +| <a id="usercalloutfeaturenameenumminute_limit_banner"></a>`MINUTE_LIMIT_BANNER` | Callout feature name for minute_limit_banner. | | <a id="usercalloutfeaturenameenumnew_user_signups_cap_reached"></a>`NEW_USER_SIGNUPS_CAP_REACHED` | Callout feature name for new_user_signups_cap_reached. | | <a id="usercalloutfeaturenameenumpersonal_access_token_expiry"></a>`PERSONAL_ACCESS_TOKEN_EXPIRY` | Callout feature name for personal_access_token_expiry. | | <a id="usercalloutfeaturenameenumpipeline_needs_banner"></a>`PIPELINE_NEEDS_BANNER` | Callout feature name for pipeline_needs_banner. | @@ -20213,7 +20287,7 @@ Implementations: | <a id="userprofileenablegitpodpath"></a>`profileEnableGitpodPath` | [`String`](#string) | Web path to enable Gitpod for the user. | | <a id="userprojectmemberships"></a>`projectMemberships` | [`ProjectMemberConnection`](#projectmemberconnection) | Project memberships of the user. (see [Connections](#connections)) | | <a id="userpublicemail"></a>`publicEmail` | [`String`](#string) | User's public email. | -| <a id="usersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. (see [Connections](#connections)) | +| <a id="usersavedreplies"></a>`savedReplies` | [`SavedReplyConnection`](#savedreplyconnection) | Saved replies authored by the user. Will not return saved replies if `saved_replies` feature flag is disabled. (see [Connections](#connections)) | | <a id="userstate"></a>`state` | [`UserState!`](#userstate) | State of the user. | | <a id="userstatus"></a>`status` | [`UserStatus`](#userstatus) | User status. | | <a id="useruserpermissions"></a>`userPermissions` | [`UserPermissions!`](#userpermissions) | Permissions for the current user on the resource. | @@ -20499,8 +20573,8 @@ Field that are available while modifying the custom mapping attributes for an HT | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="complianceviolationinputmergedafter"></a>`mergedAfter` | [`Date`](#date) | Merged date of merge requests merged after a compliance violation was created. | -| <a id="complianceviolationinputmergedbefore"></a>`mergedBefore` | [`Date`](#date) | Merged date of merge requests merged before a compliance violation was created. | +| <a id="complianceviolationinputmergedafter"></a>`mergedAfter` | [`Date`](#date) | Merge requests merged after this date (inclusive). | +| <a id="complianceviolationinputmergedbefore"></a>`mergedBefore` | [`Date`](#date) | Merge requests merged before this date (inclusive). | | <a id="complianceviolationinputprojectids"></a>`projectIds` | [`[ProjectID!]`](#projectid) | Filter compliance violations by project. | ### `DastProfileCadenceInput` @@ -20538,6 +20612,7 @@ Input type for DastSiteProfile authentication. | <a id="dastsiteprofileauthinputenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates whether authentication is enabled. | | <a id="dastsiteprofileauthinputpassword"></a>`password` | [`String`](#string) | Password to authenticate with on the target website. | | <a id="dastsiteprofileauthinputpasswordfield"></a>`passwordField` | [`String`](#string) | Name of password field at the sign-in HTML form. | +| <a id="dastsiteprofileauthinputsubmitfield"></a>`submitField` | [`String`](#string) | Name or ID of sign-in submit button at the sign-in HTML form. | | <a id="dastsiteprofileauthinputurl"></a>`url` | [`String`](#string) | The URL of the page containing the sign-in HTML form on the target website. | | <a id="dastsiteprofileauthinputusername"></a>`username` | [`String`](#string) | Username to authenticate with on the target website. | | <a id="dastsiteprofileauthinputusernamefield"></a>`usernameField` | [`String`](#string) | Name of username field at the sign-in HTML form. | diff --git a/doc/api/group_access_tokens.md b/doc/api/group_access_tokens.md index 45366885c5c..0d1878ebf39 100644 --- a/doc/api/group_access_tokens.md +++ b/doc/api/group_access_tokens.md @@ -20,7 +20,7 @@ GET groups/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/<group_id>/access_tokens" @@ -44,6 +44,41 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ] ``` +## Get a group access token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82714) in GitLab 14.10. + +Get a [group access token](../user/group/settings/group_access_tokens.md) by ID. + +```plaintext +GET groups/:id/access_tokens/:token_id +``` + +| Attribute | Type | required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `token_id` | integer or string | yes | ID of the group access token | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/<group_id>/access_tokens/<token_id>" +``` + +```json +{ + "user_id" : 141, + "scopes" : [ + "api" + ], + "name" : "token", + "expires_at" : "2021-01-31", + "id" : 42, + "active" : true, + "created_at" : "2021-01-20T22:11:48.151Z", + "revoked" : false, + "access_level": 40 +} +``` + ## Create a group access token > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77236) in GitLab 14.7. @@ -56,11 +91,11 @@ POST groups/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | -| `name` | String | yes | The name of the group access token | +| `id` | integer or string | yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `name` | String | yes | Name of the group access token | | `scopes` | `Array[String]` | yes | [List of scopes](../user/group/settings/group_access_tokens.md#scopes-for-a-group-access-token) | | `access_level` | Integer | no | A valid access level. Default value is 40 (Maintainer). Other allowed values are 10 (Guest), 20 (Reporter), and 30 (Developer). | -| `expires_at` | Date | no | The token expires at midnight UTC on that date | +| `expires_at` | Date | no | Token expires at midnight UTC on that date | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -99,8 +134,8 @@ DELETE groups/:id/access_tokens/:token_id | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | -| `token_id` | integer or string | yes | The ID of the group access token | +| `id` | integer or string | yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `token_id` | integer or string | yes | ID of the group access token | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/<group_id>/access_tokens/<token_id>" diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index 360790daf8c..b1166ba5b54 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication and Authorization +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/group_protected_environments.md b/doc/api/group_protected_environments.md index 6ce4e1791b0..f8f9b853354 100644 --- a/doc/api/group_protected_environments.md +++ b/doc/api/group_protected_environments.md @@ -107,6 +107,7 @@ POST /groups/:id/protected_environments | `name` | string | yes | The deployment tier of the protected environment. One of `production`, `staging`, `testing`, `development`, or `other`. Read more about [deployment tiers](../ci/environments/index.md#deployment-tier-of-environments).| | `deploy_access_levels` | array | yes | Array of access levels allowed to deploy, with each described by a hash. One of `user_id`, `group_id` or `access_level`. They take the form of `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}` respectively. | | `required_approval_count` | integer | no | The number of approvals required to deploy to this environment. This is part of Deployment Approvals, which isn't yet available for use. For details, see [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/343864). | +| `approval_rules` | array | no | Array of access levels allowed to approve, with each described by a hash. One of `user_id`, `group_id` or `access_level`. They take the form of `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}` respectively. You can also specify the number of required approvals from the specified entity with `required_approvals` field. See [Multiple approval rules](../ci/environments/deployment_approvals.md#multiple-approval-rules) for more information. | The assignable `user_id` are the users who belong to the given group with the Maintainer role (or above). The assignable `group_id` are the sub-groups under the given group. diff --git a/doc/api/group_releases.md b/doc/api/group_releases.md new file mode 100644 index 00000000000..06ce55b7d48 --- /dev/null +++ b/doc/api/group_releases.md @@ -0,0 +1,77 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Group releases API **(FREE)** + +Review your groups' [releases](../user/project/releases/index.md) with the REST API. + +NOTE: +For information about the project releases API, visit the [Releases API](releases/index.md) page. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `group_releases_finder_inoperator`. + +## List group releases + +Returns a list of group releases. + +```plaintext +GET /groups/:id/releases +GET /groups/:id/releases?simple=true +``` + +Parameters: + +| Attribute | Type | Required | Description | +|---------------------|----------------|----------|---------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `sort` | string | no | The direction of the order. Either `desc` (default) for descending order or `asc` for ascending order. | +| `simple` | boolean | no | Return only limited fields for each release. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/releases" +``` + +Example response: + +```json +[ + { + "name": "standard release", + "tag_name": "releasetag", + "description": "", + "created_at": "2022-01-10T15:23:15.529Z", + "released_at": "2022-01-10T15:23:15.529Z", + "author": { + "id": 1, + "username": "root", + "name": "Administrator", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "https://gitlab.com/root" + }, + "commit": { + "id": "e8cbb845ae5a53a2fef2938cf63cf82efc10d993", + "short_id": "e8cbb845", + "created_at": "2022-01-10T15:20:29.000+00:00", + "parent_ids": [], + "title": "Update test", + "message": "Update test", + "author_name": "Administrator", + "author_email": "admin@example.com", + "authored_date": "2022-01-10T15:20:29.000+00:00", + "committer_name": "Administrator", + "committer_email": "admin@example.com", + "committed_date": "2022-01-10T15:20:29.000+00:00", + "trailers": {}, + "web_url": "https://gitlab.com/groups/gitlab-org/-/commit/e8cbb845ae5a53a2fef2938cf63cf82efc10d993" + }, + "upcoming_release": false, + "commit_path": "/testgroup/test/-/commit/e8cbb845ae5a53a2fef2938cf63cf82efc10d993", + "tag_path": "/testgroup/test/-/tags/testtag" + } +] +``` diff --git a/doc/api/groups.md b/doc/api/groups.md index 120090c18a2..e04e5207c95 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication and Authorization +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -483,6 +483,8 @@ Example response: ## Details of a group +> The `membership_lock` field was [added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82271) in GitLab 14.10. + Get all details of a group. This endpoint can be accessed without authentication if the group is publicly accessible. In case the user that requests is an administrator if the group is publicly accessible. With authentication, it returns the `runners_token` @@ -715,6 +717,18 @@ the `marked_for_deletion_on` attribute: } ``` +Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see +the `membership_lock` attribute: + +```json +{ + "id": 4, + "description": "Aliquid qui quis dignissimos distinctio ut commodi voluptas est.", + "membership_lock": false, + ... +} +``` + When adding the parameter `with_projects=false`, projects aren't returned. ```shell @@ -795,24 +809,24 @@ Parameters: | ------------------------------------------------------- | ------- | -------- | ----------- | | `name` | string | yes | The name of the group. | | `path` | string | yes | The path of the group. | -| `description` | string | no | The group's description. | -| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | -| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | -| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | -| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | -| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | | `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | -| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | -| `emails_disabled` | boolean | no | Disable email notifications | | `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | -| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | +| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | +| `description` | string | no | The group's description. | +| `emails_disabled` | boolean | no | Disable email notifications. | | `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | -| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned. | | `parent_id` | integer | no | The parent group ID for creating nested group. | -| `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). Default to the global level default branch protection setting. | -| `shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | +| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | +| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | +| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | +| `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | +| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | | `extra_shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | +| `shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | ### Options for `default_branch_protection` @@ -898,27 +912,27 @@ PUT /groups/:id | `id` | integer | yes | The ID of the group. | | `name` | string | no | The name of the group. | | `path` | string | no | The path of the group. | -| `description` | string | no | The description of the group. | -| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | -| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | -| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | -| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | -| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | -| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | | `auto_devops_enabled` | boolean | no | Default to Auto DevOps pipeline for all projects within this group. | -| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | -| `emails_disabled` | boolean | no | Disable email notifications | | `avatar` | mixed | no | Image file for avatar of the group. [Introduced in GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36681) | -| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned | -| `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | -| `request_access_enabled` | boolean | no | Allow users to request member access. | | `default_branch_protection` | integer | no | See [Options for `default_branch_protection`](#options-for-default_branch_protection). | +| `description` | string | no | The description of the group. | +| `emails_disabled` | boolean | no | Disable email notifications. | +| `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group. | +| `mentions_disabled` | boolean | no | Disable the capability of a group from getting mentioned. | +| `prevent_sharing_groups_outside_hierarchy` | boolean | no | See [Prevent group sharing outside the group hierarchy](../user/group/index.md#prevent-group-sharing-outside-the-group-hierarchy). This attribute is only available on top-level groups. [Introduced in GitLab 14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/333721) | +| `project_creation_level` | string | no | Determine if developers can create projects in the group. Can be `noone` (No one), `maintainer` (users with the Maintainer role), or `developer` (users with the Developer or Maintainer role). | +| `request_access_enabled` | boolean | no | Allow users to request member access. | +| `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | +| `shared_runners_setting` | string | no | See [Options for `shared_runners_setting`](#options-for-shared_runners_setting). Enable or disable shared runners for a group's subgroups and projects. | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | +| `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | +| `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | +| `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | +| `extra_shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | | `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | +| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | +| `prevent_forking_outside_group` **(PREMIUM)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces. | | `shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | -| `extra_shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | -| `prevent_forking_outside_group` **(PREMIUM)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces -| `shared_runners_setting` | string | no | See [Options for `shared_runners_setting`](#options-for-shared_runners_setting). Enable or disable shared runners for a group's subgroups and projects. | -| `prevent_sharing_groups_outside_hierarchy` | boolean | no | See [Prevent group sharing outside the group hierarchy](../user/group/index.md#prevent-group-sharing-outside-the-group-hierarchy). This attribute is only available on top-level groups. [Introduced in GitLab 14.1](https://gitlab.com/gitlab-org/gitlab/-/issues/333721) | NOTE: The `projects` and `shared_projects` attributes in the response are deprecated and [scheduled for removal in API v5](https://gitlab.com/gitlab-org/gitlab/-/issues/213797). diff --git a/doc/api/index.md b/doc/api/index.md index 178c2f05a6d..f78a501fb11 100644 --- a/doc/api/index.md +++ b/doc/api/index.md @@ -150,6 +150,10 @@ message with a status code of `401`: } ``` +NOTE: +Deploy tokens can't be used with the GitLab public API. For details, see +[Deploy Tokens](../user/project/deploy_tokens/index.md). + ### OAuth2 tokens You can use an [OAuth2 token](oauth2.md) to authenticate with the API by passing diff --git a/doc/api/integrations.md b/doc/api/integrations.md index 90bb26ffd3d..c1564826944 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -315,14 +315,15 @@ PUT /projects/:id/integrations/datadog Parameters: -| Parameter | Type | Required | Description | -| ---------------------- | ------- | -------- | ----------- | -| `api_key` | string | true | API key used for authentication with Datadog | -| `api_url` | string | false | (Advanced) The full URL for your Datadog site | -| `datadog_env` | string | false | For self-managed deployments, set the env% tag for all the data sent to Datadog. | -| `datadog_service` | string | false | Tag all data from this GitLab instance in Datadog. Useful when managing several self-managed deployments | -| `datadog_site` | string | false | The Datadog site to send data to. To send data to the EU site, use `datadoghq.eu` | -| `datadog_tags` | string | false | Custom tags in Datadog. Specify one tag per line in the format: `key:value\nkey2:value2` ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79665) in GitLab 14.8.) | +| Parameter | Type | Required | Description | +|:-----------------:|:------:|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api_key` | string | true | API key used for authentication with Datadog. | +| `api_url` | string | false | (Advanced) The full URL for your Datadog site | +| `datadog_env` | string | false | For self-managed deployments, set the env% tag for all the data sent to Datadog. | +| `datadog_service` | string | false | Tag all data from this GitLab instance in Datadog. Useful when managing several self-managed deployments | +| `datadog_site` | string | false | The Datadog site to send data to. To send data to the EU site, use `datadoghq.eu` | +| `datadog_tags` | string | false | Custom tags in Datadog. Specify one tag per line in the format: `key:value\nkey2:value2` ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79665) in GitLab 14.8.) | + <!-- | `archive_trace_events` | boolean | false | When enabled, job logs are collected by Datadog and displayed along with pipeline execution traces ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346339) in GitLab 14.7) | --> <!-- TODO: uncomment the archive_trace_events field once :datadog_integration_logs_collection is rolled out. Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/issues/346339 --> diff --git a/doc/api/invitations.md b/doc/api/invitations.md index 0bf9d106404..eb7351c2e09 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Invitations API **(FREE)** -Use the Invitations API to send email to users you want to join a group or project, and to list pending +Use the Invitations API to invite or add users to a group or project, and to list pending invitations. ## Valid access levels @@ -26,9 +26,9 @@ WARNING: Due to [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/219299), projects in personal namespaces don't show owner (`50`) permission. -## Invite by email to group or project +## Add a member to a group or project -Invites a new user by email to join a group or project. +Adds a new member. You can specify a user ID or invite a user by email. ```plaintext POST /groups/:id/invitations @@ -38,7 +38,8 @@ POST /projects/:id/invitations | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project or group](index.md#namespaced-path-encoding) owned by the authenticated user | -| `email` | string | yes | The email of the new member or multiple emails separated by commas | +| `email` | string | yes (if `user_id` isn't provided) | The email of the new member or multiple emails separated by commas. | +| `user_id` | integer/string | yes (if `email` isn't provided) | The ID of the new member or multiple IDs separated by commas. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350999) in GitLab 14.10. | | `access_level` | integer | yes | A valid access level | | `expires_at` | string | no | A date string in the format YEAR-MONTH-DAY | | `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | @@ -47,9 +48,9 @@ POST /projects/:id/invitations ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/invitations" + --data "email=test@example.com&user_id=1&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/invitations" curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/invitations" + --data "email=test@example.com&user_id=1&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/invitations" ``` Example responses: @@ -67,7 +68,8 @@ When there was any error sending the email: "status": "error", "message": { "test@example.com": "Invite email has already been taken", - "test2@example.com": "User already exists in source" + "test2@example.com": "User already exists in source", + "test_username": "Access level is not included in the list" } } ``` @@ -77,7 +79,7 @@ When there was any error sending the email: Gets a list of invited group or project members viewable by the authenticated user. Returns invitations to direct members only, and not through inherited ancestors' groups. -This function takes pagination parameters `page` and `per_page` to restrict the list of users. +This function takes pagination parameters `page` and `per_page` to restrict the list of members. ```plaintext GET /groups/:id/invitations diff --git a/doc/api/issues.md b/doc/api/issues.md index ef0727e1c13..e82aa8da8ed 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -165,6 +165,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links":{ "self":"http://gitlab.example.com/api/v4/projects/1/issues/76", "notes":"http://gitlab.example.com/api/v4/projects/1/issues/76/notes", @@ -390,6 +391,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links":{ "self":"http://gitlab.example.com/api/v4/projects/4/issues/41", "notes":"http://gitlab.example.com/api/v4/projects/4/issues/41/notes", @@ -598,6 +600,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links":{ "self":"http://gitlab.example.com/api/v4/projects/4/issues/41", "notes":"http://gitlab.example.com/api/v4/projects/4/issues/41/notes", @@ -755,6 +758,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "task_completion_status": { "count": 0, "completed_count": 0 @@ -917,6 +921,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links": { "self": "http://gitlab.example.com/api/v4/projects/1/issues/2", "notes": "http://gitlab.example.com/api/v4/projects/1/issues/2/notes", @@ -1064,6 +1069,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links": { "self": "http://gitlab.example.com/api/v4/projects/1/issues/2", "notes": "http://gitlab.example.com/api/v4/projects/1/issues/2/notes", @@ -1238,6 +1244,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links": { "self": "http://gitlab.example.com/api/v4/projects/1/issues/2", "notes": "http://gitlab.example.com/api/v4/projects/1/issues/2/notes", @@ -1421,6 +1428,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links": { "self": "http://gitlab.example.com/api/v4/projects/1/issues/2", "notes": "http://gitlab.example.com/api/v4/projects/1/issues/2/notes", @@ -1560,6 +1568,7 @@ Example response: "confidential":false, "discussion_locked":null, "issue_type":"issue", + "severity": "UNKNOWN", "web_url":"https://gitlab.example.com/namespace1/project2/-/issues/1", "time_stats":{ "time_estimate":0, @@ -1669,6 +1678,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "_links": { "self": "http://gitlab.example.com/api/v4/projects/1/issues/2", "notes": "http://gitlab.example.com/api/v4/projects/1/issues/2/notes", @@ -1797,6 +1807,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "task_completion_status":{ "count":0, "completed_count":0 @@ -1906,6 +1917,7 @@ Example response: "confidential": false, "discussion_locked": false, "issue_type": "issue", + "severity": "UNKNOWN", "task_completion_status":{ "count":0, "completed_count":0 diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index d272f259ddf..517ffde0046 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -287,11 +287,8 @@ If the artifacts were deleted successfully, a response with status `204 No Conte ## Delete project artifacts -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223793) in GitLab 14.7 [with a flag](../administration/feature_flags.md) named `bulk_expire_project_artifacts`. Enabled by default on GitLab self-managed. Enabled on GitLab.com. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to -[disable the `bulk_expire_project_artifacts` flag](../administration/feature_flags.md). On GitLab.com, this feature is available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223793) in GitLab 14.7 [with a flag](../administration/feature_flags.md) named `bulk_expire_project_artifacts`. Enabled by default on GitLab self-managed. Enabled on GitLab.com. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/350609) in GitLab 14.10. Delete artifacts of a project that can be deleted. diff --git a/doc/api/linked_epics.md b/doc/api/linked_epics.md index 89168c344f3..df302be0555 100644 --- a/doc/api/linked_epics.md +++ b/doc/api/linked_epics.md @@ -88,3 +88,239 @@ Example response: } ] ``` + +## Create a related epic link + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352840) in GitLab 14.10. + +Create a two-way relation between two epics. The user must be allowed to +update both epics to succeed. + +```plaintext +POST /groups/:id/epics/:epic_iid/related_epics +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|---------------------|----------------|-----------------------------|---------------------------------------| +| `epic_iid` | integer | **{check-circle}** Yes | Internal ID of a group's epic. | +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `target_epic_iid` | integer/string | **{check-circle}** Yes | Internal ID of a target group's epic. | +| `target_group_id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the target group](index.md#namespaced-path-encoding). | +| `link_type` | string | **{dotted-circle}** No | Type of the relation (`relates_to`, `blocks`, `is_blocked_by`), defaults to `relates_to`. | + +Example request: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/26/epics/1/related_epics?target_group_id=26&target_epic_iid=5" +``` + +Example response: + +```json +{ + "source_epic": { + "id": 21, + "iid": 1, + "color": "#1068bf", + "text_color": "#FFFFFF", + "group_id": 26, + "parent_id": null, + "parent_iid": null, + "title": "Aspernatur recusandae distinctio omnis et qui est iste.", + "description": "some description", + "confidential": false, + "author": { + "id": 15, + "username": "trina", + "name": "Theresia Robel", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/085e28df717e16484cbf6ceca75e9a93?s=80&d=identicon", + "web_url": "http://gitlab.example.com/trina" + }, + "start_date": null, + "end_date": null, + "due_date": null, + "state": "opened", + "web_url": "http://gitlab.example.com/groups/flightjs/-/epics/1", + "references": { + "short": "&1", + "relative": "&1", + "full": "flightjs&1" + }, + "created_at": "2022-01-31T15:10:44.988Z", + "updated_at": "2022-03-16T09:32:35.712Z", + "closed_at": null, + "labels": [], + "upvotes": 0, + "downvotes": 0, + "_links": { + "self": "http://gitlab.example.com/api/v4/groups/26/epics/1", + "epic_issues": "http://gitlab.example.com/api/v4/groups/26/epics/1/issues", + "group": "http://gitlab.example.com/api/v4/groups/26", + "parent": null + } + }, + "target_epic": { + "id": 25, + "iid": 5, + "color": "#1068bf", + "text_color": "#FFFFFF", + "group_id": 26, + "parent_id": null, + "parent_iid": null, + "title": "Aut assumenda id nihil distinctio fugiat vel numquam est.", + "description": "some description", + "confidential": false, + "author": { + "id": 3, + "username": "valerie", + "name": "Erika Wolf", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/9ef7666abb101418a4716a8ed4dded80?s=80&d=identicon", + "web_url": "http://gitlab.example.com/valerie" + }, + "start_date": null, + "end_date": null, + "due_date": null, + "state": "opened", + "web_url": "http://gitlab.example.com/groups/flightjs/-/epics/5", + "references": { + "short": "&5", + "relative": "&5", + "full": "flightjs&5" + }, + "created_at": "2022-01-31T15:10:45.080Z", + "updated_at": "2022-03-16T09:32:35.842Z", + "closed_at": null, + "labels": [], + "upvotes": 0, + "downvotes": 0, + "_links": { + "self": "http://gitlab.example.com/api/v4/groups/26/epics/5", + "epic_issues": "http://gitlab.example.com/api/v4/groups/26/epics/5/issues", + "group": "http://gitlab.example.com/api/v4/groups/26", + "parent": null + } + }, + "link_type": "relates_to" +} +``` + +## Delete a related epic link + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352840) in GitLab 14.10. + +Delete a two-way relation between two epics. The user must be allowed to +update both epics to succeed. + +```plaintext +DELETE /groups/:id/epics/:epic_iid/related_epics/:related_epic_link_id +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +|--------------------------|----------------|-----------------------------|---------------------------------------| +| `epic_iid` | integer | **{check-circle}** Yes | Internal ID of a group's epic. | +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `related_epic_link_id` | integer/string | **{check-circle}** Yes | Internal ID of a related epic link. | + +Example request: + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/26/epics/1/related_epics/1" +``` + +Example response: + +```json +{ + "source_epic": { + "id": 21, + "iid": 1, + "color": "#1068bf", + "text_color": "#FFFFFF", + "group_id": 26, + "parent_id": null, + "parent_iid": null, + "title": "Aspernatur recusandae distinctio omnis et qui est iste.", + "description": "some description", + "confidential": false, + "author": { + "id": 15, + "username": "trina", + "name": "Theresia Robel", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/085e28df717e16484cbf6ceca75e9a93?s=80&d=identicon", + "web_url": "http://gitlab.example.com/trina" + }, + "start_date": null, + "end_date": null, + "due_date": null, + "state": "opened", + "web_url": "http://gitlab.example.com/groups/flightjs/-/epics/1", + "references": { + "short": "&1", + "relative": "&1", + "full": "flightjs&1" + }, + "created_at": "2022-01-31T15:10:44.988Z", + "updated_at": "2022-03-16T09:32:35.712Z", + "closed_at": null, + "labels": [], + "upvotes": 0, + "downvotes": 0, + "_links": { + "self": "http://gitlab.example.com/api/v4/groups/26/epics/1", + "epic_issues": "http://gitlab.example.com/api/v4/groups/26/epics/1/issues", + "group": "http://gitlab.example.com/api/v4/groups/26", + "parent": null + } + }, + "target_epic": { + "id": 25, + "iid": 5, + "color": "#1068bf", + "text_color": "#FFFFFF", + "group_id": 26, + "parent_id": null, + "parent_iid": null, + "title": "Aut assumenda id nihil distinctio fugiat vel numquam est.", + "description": "some description", + "confidential": false, + "author": { + "id": 3, + "username": "valerie", + "name": "Erika Wolf", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/9ef7666abb101418a4716a8ed4dded80?s=80&d=identicon", + "web_url": "http://gitlab.example.com/valerie" + }, + "start_date": null, + "end_date": null, + "due_date": null, + "state": "opened", + "web_url": "http://gitlab.example.com/groups/flightjs/-/epics/5", + "references": { + "short": "&5", + "relative": "&5", + "full": "flightjs&5" + }, + "created_at": "2022-01-31T15:10:45.080Z", + "updated_at": "2022-03-16T09:32:35.842Z", + "closed_at": null, + "labels": [], + "upvotes": 0, + "downvotes": 0, + "_links": { + "self": "http://gitlab.example.com/api/v4/groups/26/epics/5", + "epic_issues": "http://gitlab.example.com/api/v4/groups/26/epics/5/issues", + "group": "http://gitlab.example.com/api/v4/groups/26", + "parent": null + } + }, + "link_type": "relates_to" +} +``` diff --git a/doc/api/members.md b/doc/api/members.md index 10072baf2ff..77a91436e6c 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -18,10 +18,9 @@ The access levels are defined in the `Gitlab::Access` module. Currently, these l - Maintainer (`40`) - Owner (`50`) - Only valid to set for groups -WARNING: -Due to [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/219299), -projects in personal namespaces don't show owner (`50`) permission -for owner. +NOTE: +In [GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/issues/351211) and later, projects in personal namespaces have an `access_level` of `50`(Owner). +In GitLab 14.8 and earlier, projects in personal namespaces have an `access_level` of `40` (Maintainer) due to [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/219299) ## Limitations @@ -63,6 +62,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-09-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, "group_saml_identity": null, @@ -75,6 +83,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-09-22T14:13:35Z", + "created_by": { + "id": 1, + "username": "raymond_smith", + "name": "Raymond Smith", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, "email": "john@example.com", @@ -132,6 +149,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-09-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, "group_saml_identity": null, @@ -144,6 +170,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-09-22T14:13:35Z", + "created_by": { + "id": 1, + "username": "raymond_smith", + "name": "Raymond Smith", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, "email": "john@example.com", @@ -161,6 +196,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-11-22T14:13:35Z", "access_level": 30, "group_saml_identity": null, @@ -201,6 +245,14 @@ Example response: "access_level": 30, "email": "john@example.com", "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": null, "group_saml_identity": null, "membership_state": "active" @@ -239,6 +291,15 @@ Example response: "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", "access_level": 30, + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "email": "john@example.com", "expires_at": null, "group_saml_identity": null, @@ -250,7 +311,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217384) in GitLab 13.5. -Gets a list of group members that count as billable. The list includes members in the subgroup or subproject. +Gets a list of group members that count as billable. The list includes members in subgroups and projects. This API endpoint works on top-level groups only. It does not work on subgroups. @@ -267,11 +328,12 @@ respectively. GET /groups/:id/billable_members ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- |--------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | -| `search` | string | no | A query string to search for group members by name, username, or public email. | -| `sort` | string | no | A query string containing parameters that specify the sort attribute and order. See supported values below. | +| Attribute | Type | Required | Description | +| ----------------------------- | --------------- | --------- |-------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `search` | string | no | A query string to search for group members by name, username, or public email. | +| `sort` | string | no | A query string containing parameters that specify the sort attribute and order. See supported values below. | +| `include_awaiting_members` | boolean | no | Determines if awaiting members are included. | The supported values for the `sort` attribute are: @@ -454,6 +516,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 30, "email": "john@example.com", @@ -492,6 +563,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 40, "email": "john@example.com", @@ -529,6 +609,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22T14:13:35Z", "access_level": 40, "email": "john@example.com", @@ -566,6 +655,15 @@ Example response: "state": "active", "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", "web_url": "http://192.168.1.8:3000/root", + "created_at": "2012-10-22T14:13:35Z", + "created_by": { + "id": 2, + "username": "john_doe", + "name": "John Doe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/c2525a7f58ae3776070e44c106c48e15?s=80&d=identicon", + "web_url": "http://192.168.1.8:3000/root" + }, "expires_at": "2012-10-22", "access_level": 40, "email": "john@example.com", diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index e569abd323e..721e4db3314 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -281,7 +281,7 @@ GET /projects/:id/approval_rules/:approval_rule_id WARNING: The Vulnerability-Check feature, including the Vulnerability-Check attributes listed here, is in its end-of-life process. It is [deprecated](../update/deprecations.md#vulnerability-check) -for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new [Security Approval Policies](../user/application_security/policies/#scan-result-policy-editor). You can create project approval rules using the following endpoint: @@ -413,7 +413,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ WARNING: The Vulnerability-Check feature, including the Vulnerability-Check attributes listed here, is in its end-of-life process. It is [deprecated](../update/deprecations.md#vulnerability-check) -for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new [Security Approval Policies](../user/application_security/policies/#scan-result-policy-editor). You can update project approval rules using the following endpoint: @@ -776,6 +776,82 @@ GET /projects/:id/merge_requests/:merge_request_iid/approval_rules ] ``` +### Get a single merge request level rule + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82767) in GitLab 14.10. + +You can request information about a single merge request approval rule using the following endpoint: + +```plaintext +GET /projects/:id/merge_requests/:merge_request_iid/approval_rules/:approval_rule_id +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding). | +| `merge_request_iid` | integer | yes | The IID of a merge request. | +| `approval_rule_id` | integer | yes | The ID of an approval rule. | + +```json +{ + "id": 1, + "name": "security", + "rule_type": "regular", + "eligible_approvers": [ + { + "id": 5, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + }, + { + "id": 50, + "name": "Group Member 1", + "username": "group_member_1", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/group_member_1" + } + ], + "approvals_required": 3, + "source_rule": null, + "users": [ + { + "id": 5, + "name": "John Doe", + "username": "jdoe", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/0?s=80&d=identicon", + "web_url": "http://localhost/jdoe" + } + ], + "groups": [ + { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + ], + "contains_hidden_groups": false, + "overridden": false +} +``` + ### Create merge request level rule > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11877) in GitLab 12.3. @@ -881,13 +957,13 @@ These are system generated rules. | Attribute | Type | Required | Description | |----------------------|---------|----------|------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding) | -| `merge_request_iid` | integer | yes | The ID of MR | -| `approval_rule_id` | integer | yes | The ID of a approval rule | -| `name` | string | yes | The name of the approval rule | -| `approvals_required` | integer | yes | The number of required approvals for this rule | -| `user_ids` | Array | no | The ids of users as approvers | -| `group_ids` | Array | no | The ids of groups as approvers | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding). | +| `merge_request_iid` | integer | yes | The IID of a merge request. | +| `approval_rule_id` | integer | yes | The ID of an approval rule. | +| `name` | string | yes | The name of the approval rule. | +| `approvals_required` | integer | yes | The number of required approvals for this rule. | +| `user_ids` | Array | no | The IDs of users as approvers. | +| `group_ids` | Array | no | The IDs of groups as approvers. | ```json { diff --git a/doc/api/notes.md b/doc/api/notes.md index 83631c70f8a..fbcf5e28f79 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -142,8 +142,8 @@ Parameters: | Attribute | Type | Required | Description | |----------------|----------------|----------|------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | -| `issue_iid` | integer | yes | The IID of an issue. | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `issue_iid` | integer | yes | The IID of an issue. | | `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | | `confidential` | boolean | no | The confidential flag of a note. Default is false. | | `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | @@ -162,13 +162,13 @@ PUT /projects/:id/issues/:issue_iid/notes/:note_id Parameters: -| Attribute | Type | Required | Description | -|----------------|----------------|----------|----------------------------------------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | -| `issue_iid` | integer | yes | The IID of an issue. | -| `note_id` | integer | yes | The ID of a note. | -| `body` | string | no | The content of a note. Limited to 1,000,000 characters. | -| `confidential` | boolean | no | The confidential flag of a note. | +| Attribute | Type | Required | Description | +|----------------|----------------|-------------|----------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | +| `issue_iid` | integer | yes | The IID of an issue. | +| `note_id` | integer | yes | The ID of a note. | +| `body` | string | no | The content of a note. Limited to 1,000,000 characters. | +| `confidential` | boolean | no | **Deprecated:** will be removed in GitLab 16.0. The confidential flag of a note. Default is false. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note" @@ -403,7 +403,7 @@ Parameters: | `merge_request_iid` | integer | yes | The IID of a project merge request | | `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | | `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | -| `merge_request_diff_sha`| string | no | The SHA of the head commit which is used to ensure that the merge request hasn't been updated since the API request was sent. This is required for the /merge quick action | +| `merge_request_diff_sha`| string | no | Required for the `/merge` [quick action](../user/project/quick_actions.md). The SHA of the head commit, which ensures the merge request wasn't updated after the API request was sent. | ### Modify existing merge request note @@ -415,12 +415,13 @@ PUT /projects/:id/merge_requests/:merge_request_iid/notes/:note_id Parameters: -| Attribute | Type | Required | Description | -|---------------------|----------------|----------|---------------------------------------------------------------------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | -| `merge_request_iid` | integer | yes | The IID of a project merge request | -| `note_id` | integer | no | The ID of a note | -| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| Attribute | Type | Required | Description | +|---------------------|-------------------|----------|----------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of a project merge request | +| `note_id` | integer | no | The ID of a note | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `confidential` | boolean | no | **Deprecated:** will be removed in GitLab 16.0. The confidential flag of a note. Default is false. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/merge_requests/11/notes?body=note" @@ -452,6 +453,11 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git Gets a list of all notes for a single epic. Epic notes are comments users can post to an epic. +NOTE: +The epics notes API uses the epic ID instead of epic IID. If you use the epic's IID, GitLab returns either a 404 +error or notes for the wrong epic. It's different from the [issue notes API](#issues) and +[merge requests notes API](#merge-requests). + ```plaintext GET /groups/:id/epics/:epic_id/notes GET /groups/:id/epics/:epic_id/notes?sort=asc&order_by=updated_at @@ -519,11 +525,12 @@ POST /groups/:id/epics/:epic_id/notes Parameters: -| Attribute | Type | Required | Description | -| --------- | -------------- | -------- | ----------- | -| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | -| `epic_id` | integer | yes | The ID of an epic | -| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `epic_id` | integer | yes | The ID of an epic | +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `confidential` | boolean | no | The confidential flag of a note. Default is `false`. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/epics/11/notes?body=note" @@ -539,12 +546,13 @@ PUT /groups/:id/epics/:epic_id/notes/:note_id Parameters: -| Attribute | Type | Required | Description | -| --------- | -------------- | -------- | ----------- | -| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | -| `epic_id` | integer | yes | The ID of an epic | -| `note_id` | integer | yes | The ID of a note | -| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| Attribute | Type | Required | Description | +| ---------------| ----------------- | -------- | ---------------------------------------------------------------------------------------------------| +| `id` | integer or string | yes | The ID or [URL-encoded path of the group](index.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | +| `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | +| `confidential` | boolean | no | **Deprecated:** will be removed in GitLab 16.0. The confidential flag of a note. Default is false. | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/epics/11/notes?body=note" diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 7b38ac39b96..ad93d8033d0 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -24,10 +24,6 @@ GitLab supports the following authorization flows: and is recommended for both client and server apps. - **Authorization code:** Secure and common flow. Recommended option for secure server-side apps. -- **Implicit grant:** Originally designed for user-agent only apps, such as - single page web apps running on GitLab Pages. - The [Internet Engineering Task Force (IETF)](https://tools.ietf.org/html/draft-ietf-oauth-security-topics-09#section-2.1.2) - recommends against Implicit grant flow. - **Resource owner password credentials:** To be used **only** for securely hosted, first-party services. GitLab recommends against use of this flow. @@ -64,7 +60,7 @@ As OAuth 2.0 bases its security entirely on the transport layer, you should not URIs. For more information, see the [OAuth 2.0 RFC](https://tools.ietf.org/html/rfc6749#section-3.1.2.1) and the [OAuth 2.0 Threat Model RFC](https://tools.ietf.org/html/rfc6819#section-4.4.2.1). These factors are particularly important when using the -[Implicit grant flow](#implicit-grant-flow), where actual credentials are included in the `redirect_uri`. +[Implicit grant flow](#implicit-grant-flow-deprecated), where actual credentials are included in the `redirect_uri`. In the following sections you can find detailed instructions on how to obtain authorization with each flow. @@ -242,40 +238,6 @@ authorization request. You can now make requests to the API with the access token returned. -### Implicit grant flow - -WARNING: -Implicit grant flow is inherently insecure and the IETF has removed it in [OAuth 2.1](https://oauth.net/2.1/). -It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/288516) for use in GitLab 14.0, and is planned for -[removal](https://gitlab.com/gitlab-org/gitlab/-/issues/344609) in GitLab 15.0. - -We recommend that you use [Authorization code with PKCE](#authorization-code-with-proof-key-for-code-exchange-pkce) -instead. - -Unlike the authorization code flow, the client receives an `access token` -immediately as a result of the authorization request. The flow does not use the -client secret or the authorization code, as the application -code and storage is accessible on client browsers and mobile devices. - -To request the access token, you should redirect the user to the -`/oauth/authorize` endpoint using `token` response type: - -```plaintext -https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=token&state=YOUR_UNIQUE_STATE_HASH&scope=REQUESTED_SCOPES -``` - -This prompts the user to approve the applications access to their account -based on the scopes specified in `REQUESTED_SCOPES` and then redirect back to -the `REDIRECT_URI` you provided. The [scope parameter](../integration/oauth_provider.md#authorized-applications) - is a space-separated list of scopes you want to have access to (for example, `scope=read_user+profile` -would request `read_user` and `profile` scopes). The redirect -includes a fragment with `access_token` as well as token details in GET -parameters, for example: - -```plaintext -https://example.com/oauth/redirect#access_token=ABCDExyz123&state=YOUR_UNIQUE_STATE_HASH&token_type=bearer&expires_in=3600 -``` - ### Resource owner password credentials flow NOTE: @@ -357,6 +319,40 @@ access_token = client.password.get_token('user@example.com', 'secret') puts access_token.token ``` +### Implicit grant flow (DEPRECATED) + +WARNING: +Implicit grant flow is inherently insecure and the IETF has removed it in [OAuth 2.1](https://oauth.net/2.1/). +It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/288516) in GitLab 14.0, and is planned for +[removal](https://gitlab.com/gitlab-org/gitlab/-/issues/344609) in GitLab 15.0. + +We recommend that you use [Authorization code with PKCE](#authorization-code-with-proof-key-for-code-exchange-pkce) +instead. + +Unlike the authorization code flow, the client receives an `access token` +immediately as a result of the authorization request. The flow does not use the +client secret or the authorization code, as the application +code and storage is accessible on client browsers and mobile devices. + +To request the access token, you should redirect the user to the +`/oauth/authorize` endpoint using `token` response type: + +```plaintext +https://gitlab.example.com/oauth/authorize?client_id=APP_ID&redirect_uri=REDIRECT_URI&response_type=token&state=YOUR_UNIQUE_STATE_HASH&scope=REQUESTED_SCOPES +``` + +This prompts the user to approve the applications access to their account +based on the scopes specified in `REQUESTED_SCOPES` and then redirect back to +the `REDIRECT_URI` you provided. The [scope parameter](../integration/oauth_provider.md#authorized-applications) + is a space-separated list of scopes you want to have access to (for example, `scope=read_user+profile` +would request `read_user` and `profile` scopes). The redirect +includes a fragment with `access_token` as well as token details in GET +parameters, for example: + +```plaintext +https://example.com/oauth/redirect#access_token=ABCDExyz123&state=YOUR_UNIQUE_STATE_HASH&token_type=bearer&expires_in=3600 +``` + ## Access GitLab API with `access token` The `access token` allows you to make requests to the API on behalf of a user. diff --git a/doc/api/packages/composer.md b/doc/api/packages/composer.md index b3a27519729..ea2e917bbba 100644 --- a/doc/api/packages/composer.md +++ b/doc/api/packages/composer.md @@ -262,6 +262,8 @@ Example response: ## Download a package archive +> Authorization for this endpoint was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331601) in GitLab 14.10. + Download a Composer package. This URL is provided in the [v1](#v1-package-metadata) or [v2 package metadata](#v2-package-metadata) response. A `.zip` file extension must be in the request. @@ -287,3 +289,6 @@ curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v ``` This writes the downloaded file to `package.tar.gz` in the current directory. + +NOTE: +This endpoint requires authorization in GitLab 14.10 and later. In GitLab 14.9 and earlier, it was publicly accessible. diff --git a/doc/api/pages.md b/doc/api/pages.md index 7316d225dbc..57982188858 100644 --- a/doc/api/pages.md +++ b/doc/api/pages.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index c1f81ffa361..610fd97c810 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/project_access_tokens.md b/doc/api/project_access_tokens.md index f6eced4f08a..b13005ec436 100644 --- a/doc/api/project_access_tokens.md +++ b/doc/api/project_access_tokens.md @@ -20,7 +20,7 @@ GET projects/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens" @@ -44,9 +44,46 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ] ``` +## Get a project access token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82714) in GitLab 14.10. + +Get a [project access token](../user/project/settings/project_access_tokens.md) by ID. + +```plaintext +GET projects/:id/access_tokens/:token_id +``` + +| Attribute | Type | required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `token_id` | integer or string | yes | ID of the project access token | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens/<token_id>" +``` + +```json +{ + "user_id" : 141, + "scopes" : [ + "api" + ], + "name" : "token", + "expires_at" : "2021-01-31", + "id" : 42, + "active" : true, + "created_at" : "2021-01-20T22:11:48.151Z", + "revoked" : false, + "access_level": 40, + "last_used_at": "2022-03-15T11:05:42.437Z" +} +``` + ## Create a project access token -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55408) in GitLab 13.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55408) in GitLab 13.10. +> - The `token` attribute was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55408) in GitLab 13.10. Create a [project access token](../user/project/settings/project_access_tokens.md). @@ -56,11 +93,11 @@ POST projects/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | -| `name` | String | yes | The name of the project access token | +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `name` | String | yes | Name of the project access token | | `scopes` | `Array[String]` | yes | [List of scopes](../user/project/settings/project_access_tokens.md#scopes-for-a-project-access-token) | | `access_level` | Integer | no | A valid access level. Default value is 40 (Maintainer). Other allowed values are 10 (Guest), 20 (Reporter), and 30 (Developer). | -| `expires_at` | Date | no | The token expires at midnight UTC on that date | +| `expires_at` | Date | no | Token expires at midnight UTC on that date | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -99,8 +136,8 @@ DELETE projects/:id/access_tokens/:token_id | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | -| `token_id` | integer or string | yes | The ID of the project access token | +| `id` | integer or string | yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | +| `token_id` | integer or string | yes | ID of the project access token | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens/<token_id>" diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index 46fa05d6cbc..b9ebfa0fc5c 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -248,10 +248,11 @@ The `Content-Type` header must be `application/gzip`. ## Import a file from AWS S3 -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348874) in GitLab 14.9 in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta), [with a flag](../administration/feature_flags.md) named `import_project_from_remote_file_s3`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/348874) in GitLab 14.9 in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta), [with a flag](../administration/feature_flags.md) named `import_project_from_remote_file_s3`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/348874) in GitLab 14.10. FLAG: -On self-managed GitLab and GitLab.com, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `import_project_from_remote_file_s3`. This feature is not ready for production use. +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `import_project_from_remote_file_s3`. On GitLab.com, this feature is available. ```plaintext POST /projects/remote-import-s3 @@ -261,6 +262,7 @@ POST /projects/remote-import-s3 | ------------------- | -------------- | -------- | ---------------------------------------- | | `namespace` | integer/string | no | The ID or path of the namespace to import the project to. Defaults to the current user's namespace. | | `name` | string | no | The name of the project to import. If not provided, defaults to the path of the project. | +| `path` | string | yes | The full path of the new project. | | `region` | string | yes | [AWS S3 region name where the file is stored.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#Regions) | | `bucket_name` | string | yes | [AWS S3 bucket name where the file is stored.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) | | `file_key` | string | yes | [AWS S3 file key to identify the file.](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingObjects.html) | @@ -271,7 +273,7 @@ The passed override parameters take precedence over all values defined in the ex ```shell curl --request POST \ - --url "http://localhost:3000/api/v4/projects/remote-import-s3" \ + --url "https://gitlab.example.com/api/v4/projects/remote-import-s3" \ --header "PRIVATE-TOKEN: <your gitlab access key>" \ --header 'Content-Type: application/json' \ --data '{ diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 2251b0fc7fd..4205e6699fe 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -170,5 +170,5 @@ This parameter is used for filtering by attributes, such as `environment_scope`. Example usage: ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/VARIABLE_1?filter[environment_scope]=production" +curl --request DELETE --globoff --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/VARIABLE_1?filter[environment_scope]=production" ``` diff --git a/doc/api/projects.md b/doc/api/projects.md index ee649377745..c40d39ee981 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -19,7 +19,7 @@ Values for the project visibility level are: - `internal`: the project can be cloned by any signed-in user except [external users](../user/permissions.md#external-users). - `public`: the project can be accessed without any authentication. -For more, read [Project visibility](../public_access/public_access.md). +For more, read [Project visibility](../user/public_access.md). ## Project merge method @@ -49,6 +49,7 @@ GET /projects | `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | | `id_after` | integer | **{dotted-circle}** No | Limit results to projects with IDs greater than the specified ID. | | `id_before` | integer | **{dotted-circle}** No | Limit results to projects with IDs less than the specified ID. | +| `imported` | boolean | **{dotted-circle}** No | Limit results to projects which were imported from external systems by current user. | | `last_activity_after` | datetime | **{dotted-circle}** No | Limit results to projects with last_activity after specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | | `last_activity_before` | datetime | **{dotted-circle}** No | Limit results to projects with last_activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | | `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | @@ -374,7 +375,7 @@ Get a list of visible projects owned by the given user. When accessed without authentication, only public projects are returned. NOTE: -Only the projects in the user's (specified in `user_id`) namespace are returned. Projects owned by the user in any group or subgroups are not returned. +Only the projects in the user's (specified in `user_id`) namespace are returned. Projects owned by the user in any group or subgroups are not returned. An empty list is returned if a profile is set to private. This endpoint supports [keyset pagination](index.md#keyset-based-pagination) for selected `order_by` options. @@ -1228,7 +1229,7 @@ POST /projects | `build_timeout` | integer | **{dotted-circle}** No | The maximum amount of time, in seconds, that a job can run. | | `builds_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | -| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). Valid values for `cadence` are: `1d` (every day), `7d` (every week), `14d` (every two weeks), `1month` (every month), or `3month` (every quarter). | +| `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). See the [Container Registry](../user/packages/container_registry/reduce_container_registry_storage.md#use-the-cleanup-policy-api) documentation for more information on `cadence`, `keep_n` and `older_than` values. | | `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | | `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | | `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | @@ -2563,7 +2564,7 @@ POST /projects/:id/push_rule | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | | `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository that were committed with one of their own verified emails. | +| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository if the committer email is one of their own verified emails. | | `commit_message_negative_regex` | string | **{dotted-circle}** No | No commit message is allowed to match this, for example `ssh\:\/\/`. | | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | @@ -2586,7 +2587,7 @@ PUT /projects/:id/push_rule | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `author_email_regex` | string | **{dotted-circle}** No | All commit author emails must match this, for example `@my-company.com$`. | | `branch_name_regex` | string | **{dotted-circle}** No | All branch names must match this, for example `(feature|hotfix)\/*`. | -| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository that were committed with one of their own verified emails. | +| `commit_committer_check` | boolean | **{dotted-circle}** No | Users can only push commits to this repository if the committer email is one of their own verified emails. | | `commit_message_negative_regex` | string | **{dotted-circle}** No | No commit message is allowed to match this, for example `ssh\:\/\/`. | | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | diff --git a/doc/api/protected_environments.md b/doc/api/protected_environments.md index 61587136a14..fc7eb5caf6d 100644 --- a/doc/api/protected_environments.md +++ b/doc/api/protected_environments.md @@ -99,7 +99,7 @@ POST /projects/:id/protected_environments ```shell curl --header 'Content-Type: application/json' --request POST \ - --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' \ + --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}], "approval_rules": [{"group_id": 134}, {"group_id": 135, "required_approvals": 2}]}' \ --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/projects/22034114/protected_environments" ``` @@ -110,8 +110,9 @@ curl --header 'Content-Type: application/json' --request POST \ | `name` | string | yes | The name of the environment. | | `deploy_access_levels` | array | yes | Array of access levels allowed to deploy, with each described by a hash. | | `required_approval_count` | integer | no | The number of approvals required to deploy to this environment. This is part of Deployment Approvals, which isn't yet available for use. For details, see [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/343864). | +| `approval_rules` | array | no | Array of access levels allowed to approve, with each described by a hash. See [Multiple approval rules](../ci/environments/deployment_approvals.md#multiple-approval-rules) for more information. | -Elements in the `deploy_access_levels` array should be one of `user_id`, `group_id` or +Elements in the `deploy_access_levels` and `approval_rules` array should be one of `user_id`, `group_id` or `access_level`, and take the form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). @@ -129,7 +130,23 @@ Example response: "group_id": 9899826 } ], - "required_approval_count": 0 + "required_approval_count": 0, + "approval_rules": [ + { + "user_id": null, + "group_id": 134, + "access_level": null, + "access_level_description": "qa-group", + "required_approvals": 1 + }, + { + "user_id": null, + "group_id": 135, + "access_level": null, + "access_level_description": "security-group", + "required_approvals": 2 + } + ] } ``` diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index c603be9489c..6023adb79d0 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -53,7 +53,7 @@ Example response: [ { "tag_name":"v0.2", - "description":"## CHANGELOG\r\n\r\n- Escape label and milestone titles to prevent XSS in GFM autocomplete. !2740\r\n- Prevent private snippets from being embeddable.\r\n- Add subresources removal to member destroy service.", + "description":"## CHANGELOG\r\n\r\n- Escape label and milestone titles to prevent XSS in GLFM autocomplete. !2740\r\n- Prevent private snippets from being embeddable.\r\n- Add subresources removal to member destroy service.", "name":"Awesome app v0.2 beta", "created_at":"2019-01-03T01:56:19.539Z", "released_at":"2019-01-03T01:56:19.539Z", @@ -241,10 +241,10 @@ Get a Release for the given tag. GET /projects/:id/releases/:tag_name ``` -| Attribute | Type | Required | Description | -| ------------- | -------------- | -------- | ----------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding). | -| `tag_name` | string | yes | The Git tag the release is associated with. | +| Attribute | Type | Required | Description | +|----------------------------| -------------- | -------- | ----------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding). | +| `tag_name` | string | yes | The Git tag the release is associated with. | | `include_html_description` | boolean | no | If `true`, a response includes HTML rendered Markdown of the release description. | Example request: diff --git a/doc/api/releases/links.md b/doc/api/releases/links.md index 282ef0adc78..66808900ab6 100644 --- a/doc/api/releases/links.md +++ b/doc/api/releases/links.md @@ -11,6 +11,16 @@ links. For manipulating other Release assets, see [Release API](index.md). GitLab supports links to `http`, `https`, and `ftp` assets. +## Authentication + +For authentication, the Release Links API accepts: + +- A [Personal Access Token](../../user/profile/personal_access_tokens.md) using the + `PRIVATE-TOKEN` header. +- A [Project Access Token](../../user/project/settings/project_access_tokens.md) using the `PRIVATE-TOKEN` header. + +The [GitLab CI/CD job token](../../ci/jobs/ci_job_token.md) `$CI_JOB_TOKEN` is not supported. See [GitLab issue #50819](https://gitlab.com/gitlab-org/gitlab/-/issues/250819) for more details. + ## Get links Get assets as links from a Release. diff --git a/doc/api/remote_mirrors.md b/doc/api/remote_mirrors.md index 8b584285033..351706e8514 100644 --- a/doc/api/remote_mirrors.md +++ b/doc/api/remote_mirrors.md @@ -51,6 +51,43 @@ NOTE: For security reasons, the `url` attribute is always scrubbed of username and password information. +## Get a single project's remote mirror + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82770) in GitLab 14.10. + +Returns a remote mirror and its statuses: + +```plaintext +GET /projects/:id/remote_mirrors/:mirror_id +``` + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/remote_mirrors/101486" +``` + +Example response: + +```json +{ + "enabled": true, + "id": 101486, + "last_error": null, + "last_successful_update_at": "2020-01-06T17:32:02.823Z", + "last_update_at": "2020-01-06T17:32:02.823Z", + "last_update_started_at": "2020-01-06T17:31:55.864Z", + "only_protected_branches": true, + "keep_divergent_refs": true, + "update_status": "finished", + "url": "https://*****:*****@gitlab.com/gitlab-org/security/gitlab.git" +} +``` + +NOTE: +For security reasons, the `url` attribute is always scrubbed of username +and password information. + ## Create a pull mirror Learn how to [configure a pull mirror](projects.md#configure-pull-mirroring-for-a-project) using the Projects API. @@ -136,3 +173,23 @@ Example response: "url": "https://*****:*****@gitlab.com/gitlab-org/security/gitlab.git" } ``` + +## Delete a remote mirror + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82778) in GitLab 14.10. + +Delete a remote mirror. + +```plaintext +DELETE /projects/:id/remote_mirrors/:mirror_id +``` + +| Attribute | Type | Required | Description | +| :---------- | :----- | :--------- |:------------------| +| `mirror_id` | Integer | yes | Remote mirror ID. | + +Example request: + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/remote_mirrors/101486" +``` diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 23acebf5aab..1fdc48a6d92 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -23,6 +23,8 @@ in the following table. ## Get file from repository +> The `execute_filemode` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83499) in GitLab 14.10. + Allows you to receive information about file in repository like name, size, content. File content is Base64 encoded. This endpoint can be accessed without authentication if the repository is publicly accessible. @@ -54,7 +56,8 @@ Example response: "ref": "master", "blob_id": "79f7bbd25901e8334750839545a9bd021f0e4c83", "commit_id": "d5a3ff139356ce33e37e73add446f16869741b50", - "last_commit_id": "570e7b2abdd848b95f2f578043fc23bd6f6fd24d" + "last_commit_id": "570e7b2abdd848b95f2f578043fc23bd6f6fd24d", + "execute_filemode": false } ``` @@ -85,6 +88,7 @@ X-Gitlab-File-Path: app/models/key.rb X-Gitlab-Last-Commit-Id: 570e7b2abdd848b95f2f578043fc23bd6f6fd24d X-Gitlab-Ref: master X-Gitlab-Size: 1476 +X-Gitlab-Execute-Filemode: false ... ``` @@ -155,6 +159,7 @@ X-Gitlab-File-Path: path/to/file.rb X-Gitlab-Last-Commit-Id: 570e7b2abdd848b95f2f578043fc23bd6f6fd24d X-Gitlab-Ref: master X-Gitlab-Size: 1476 +X-Gitlab-Execute-Filemode: false ... ``` @@ -179,6 +184,8 @@ Like [Get file from repository](repository_files.md#get-file-from-repository) yo ## Create new file in repository +> The `execute_filemode` parameter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83499) in GitLab 14.10. + This allows you to create a single file. For creating multiple files with a single request see the [commits API](commits.md#create-a-commit-with-multiple-files-and-actions). ```plaintext @@ -196,6 +203,7 @@ POST /projects/:id/repository/files/:file_path | `author_name` | string | no | The commit author's name. | | `content` | string | yes | The file's content. | | `commit_message` | string | yes | The commit message. | +| `execute_filemode` | boolean | no | Enables or disables the `execute` flag on the file. Can be `true` or `false`. | ```shell curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ @@ -216,6 +224,8 @@ Example response: ## Update existing file in repository +> The `execute_filemode` parameter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83499) in GitLab 14.10. + This allows you to update a single file. For updating multiple files with a single request see the [commits API](commits.md#create-a-commit-with-multiple-files-and-actions). ```plaintext @@ -234,6 +244,7 @@ PUT /projects/:id/repository/files/:file_path | `content` | string | yes | The file's content. | | `commit_message` | string | yes | The commit message. | | `last_commit_id` | string | no | Last known file commit ID. | +| `execute_filemode` | boolean | no | Enables or disables the `execute` flag on the file. Can be `true` or `false`. | ```shell curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ diff --git a/doc/api/runners.md b/doc/api/runners.md index 7437860239e..304f2494f70 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -48,7 +48,7 @@ GET /runners?tag_list=tag1,tag2 |------------|--------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online` and `offline`; showing all runners if none provided | | `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `online` and `offline`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 16.0 | +| `status` | string | no | The status of runners to show, one of: `online`, `offline` and `not_connected`. `active` and `paused` are also possible values which were deprecated in GitLab 14.8 and will be removed in GitLab 16.0 | | `paused` | boolean | no | Whether to include only runners that are accepting or ignoring new jobs | | `tag_list` | string array | no | List of the runner's tags | @@ -332,7 +332,7 @@ PUT --form "paused=true" /runners/:runner_id # --or-- -# Deprecated: removal planned in 15.0 +# Deprecated: removal planned in 16.0 PUT --form "active=false" /runners/:runner_id ``` @@ -346,7 +346,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ # --or-- -# Deprecated: removal planned in 15.0 +# Deprecated: removal planned in 16.0 curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ --form "active=false" "https://gitlab.example.com/api/v4/runners/6" ``` @@ -448,7 +448,7 @@ Example response: ## List project's runners -List all runners available in the project, including from ancestor groups and [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners). +List all runners available in the project, including from ancestor groups and [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners-for-a-project). ```plaintext GET /projects/:id/runners @@ -566,7 +566,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git ## List group's runners -List all runners available in the group as well as its ancestor groups, including [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners). +List all runners available in the group as well as its ancestor groups, including [any allowed shared runners](../ci/runners/runners_scope.md#enable-shared-runners-for-a-group). ```plaintext GET /groups/:id/runners diff --git a/doc/api/secure_files.md b/doc/api/secure_files.md index 96190920a33..7d30cc0c4c7 100644 --- a/doc/api/secure_files.md +++ b/doc/api/secure_files.md @@ -11,8 +11,7 @@ type: reference, api FLAG: On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `ci_secure_files`. -The feature is not ready for production use. +ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `ci_secure_files`. Limited to 100 secure files per project. Files must be smaller than 5 MB. The feature is not ready for production use. ## List project secure files @@ -101,12 +100,12 @@ POST /projects/:project_id/secure_files Supported attributes: -| Attribute | Type | Required | Description | -|---------------|----------------|------------------------|-------------| -| `project_id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | -| `name` | string | **{check-circle}** Yes | The `name` of the file being uploaded. | -| `file` | file | **{check-circle}** Yes | The `file` being uploaded. | -| `permissions` | string | **{dotted-circle}** No | The file is created with the specified permissions when created in the CI/CD job. Available types are: `read_only` (default), `read_write`, and `execute`. | +| Attribute | Type | Required | Description | +|-----------------|----------------|------------------------|-------------| +| `project_id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `name` | string | **{check-circle}** Yes | The `name` of the file being uploaded. The file name must be unique within the project. | +| `file` | file | **{check-circle}** Yes | The `file` being uploaded (5 MB limit). | +| `permissions` | string | **{dotted-circle}** No | The file is created with the specified permissions when created in the CI/CD job. Available types are: `read_only` (default), `read_write`, and `execute`. | Example request: diff --git a/doc/api/settings.md b/doc/api/settings.md index 16e281af916..cc9df7917e2 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -274,6 +274,7 @@ listed in the descriptions of the relevant settings. | `default_projects_limit` | integer | no | Project limit per user. Default is `100000`. | | `default_snippet_visibility` | string | no | What visibility level new snippets receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `delayed_project_deletion` **(PREMIUM SELF)** | boolean | no | Enable delayed project deletion by default in new groups. Default is `false`. | +| `delete_inactive_projects` | boolean | no | Enable inactive project deletion feature. Default is `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84519) in GitLab 14.10. | | `deletion_adjourned_period` **(PREMIUM SELF)** | integer | no | The number of days to wait before deleting a project or group that is marked for deletion. Value must be between 0 and 90. | `diff_max_patch_bytes` | integer | no | Maximum [diff patch size](../user/admin_area/diff_limits.md), in bytes. | | `diff_max_files` | integer | no | Maximum [files in a diff](../user/admin_area/diff_limits.md). | @@ -342,7 +343,7 @@ listed in the descriptions of the relevant settings. | `help_text` **(PREMIUM)** | string | no | GitLab server administrator information. | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties in GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | -| `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | +| `housekeeping_bitmaps_enabled` | boolean | no | Git pack file bitmap creation is always enabled and cannot be changed via API and UI. This API field is deprecated and always returns `true`. | | `housekeeping_enabled` | boolean | no | (**If enabled, requires:** `housekeeping_bitmaps_enabled`, `housekeeping_full_repack_period`, `housekeeping_gc_period`, and `housekeeping_incremental_repack_period`) Enable or disable Git housekeeping. | | `housekeeping_full_repack_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which an incremental `git repack` is run. | | `housekeeping_gc_period` | integer | required by: `housekeeping_enabled` | Number of Git pushes after which `git gc` is run. | @@ -350,6 +351,9 @@ listed in the descriptions of the relevant settings. | `html_emails_enabled` | boolean | no | Enable HTML emails. | | `import_sources` | array of strings | no | Sources to allow project import from, possible values: `github`, `bitbucket`, `bitbucket_server`, `gitlab`, `fogbugz`, `git`, `gitlab_project`, `gitea`, `manifest`, and `phabricator`. | | `in_product_marketing_emails_enabled` | boolean | no | Enable [in-product marketing emails](../user/profile/notifications.md#global-notification-settings). Enabled by default. | +| `inactive_projects_delete_after_months` | integer | no | If `delete_inactive_projects` is `true`, the time (in months) to wait before deleting inactive projects. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84519) in GitLab 14.10. | +| `inactive_projects_min_size_mb` | integer | no | If `delete_inactive_projects` is `true`, the minimum repository size for projects to be checked for inactivity. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84519) in GitLab 14.10. | +| `inactive_projects_send_warning_email_after_months` | integer | no | If `delete_inactive_projects` is `true`, sets the time (in months) to wait before emailing maintainers that the project will be deleted because it is inactive. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/84519) in GitLab 14.10. | | `invisible_captcha_enabled` | boolean | no | Enable Invisible CAPTCHA spam detection during sign-up. Disabled by default. | | `issues_create_limit` | integer | no | Max number of issue creation requests per minute per user. Disabled by default.| | `keep_latest_artifact` | boolean | no | Prevent the deletion of the artifacts from the most recent successful jobs, regardless of the expiry time. Enabled by default. | @@ -437,6 +441,9 @@ listed in the descriptions of the relevant settings. | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | | `throttle_authenticated_api_period_in_seconds` | integer | required by:<br>`throttle_authenticated_api_enabled` | Rate limit period (in seconds). | | `throttle_authenticated_api_requests_per_period` | integer | required by:<br>`throttle_authenticated_api_enabled` | Maximum requests per period per user. | +| `throttle_authenticated_packages_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_packages_api_period_in_seconds` and `throttle_authenticated_packages_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | +| `throttle_authenticated_packages_api_period_in_seconds` | integer | required by:<br>`throttle_authenticated_packages_api_enabled` | Rate limit period (in seconds). View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | +| `throttle_authenticated_packages_api_requests_per_period` | integer | required by:<br>`throttle_authenticated_packages_api_enabled` | Maximum requests per period per user. View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | | `throttle_authenticated_web_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_web_period_in_seconds` and `throttle_authenticated_web_requests_per_period`) Enable authenticated web request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | | `throttle_authenticated_web_period_in_seconds` | integer | required by:<br>`throttle_authenticated_web_enabled` | Rate limit period (in seconds). | | `throttle_authenticated_web_requests_per_period` | integer | required by:<br>`throttle_authenticated_web_enabled` | Maximum requests per period per user. | @@ -446,6 +453,9 @@ listed in the descriptions of the relevant settings. | `throttle_unauthenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_api_period_in_seconds` and `throttle_unauthenticated_api_requests_per_period`) Enable unauthenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | | `throttle_unauthenticated_api_period_in_seconds` | integer | required by:<br>`throttle_unauthenticated_api_enabled` | Rate limit period in seconds. | | `throttle_unauthenticated_api_requests_per_period` | integer | required by:<br>`throttle_unauthenticated_api_enabled` | Max requests per period per IP. | +| `throttle_unauthenticated_packages_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_packages_api_period_in_seconds` and `throttle_unauthenticated_packages_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | +| `throttle_unauthenticated_packages_api_period_in_seconds` | integer | required by:<br>`throttle_unauthenticated_packages_api_enabled` | Rate limit period (in seconds). View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | +| `throttle_unauthenticated_packages_api_requests_per_period` | integer | required by:<br>`throttle_unauthenticated_packages_api_enabled` | Maximum requests per period per user. View [Package Registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) for more details. | | `throttle_unauthenticated_web_enabled` | boolean | no | (**If enabled, requires:** `throttle_unauthenticated_web_period_in_seconds` and `throttle_unauthenticated_web_requests_per_period`) Enable unauthenticated web request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | | `throttle_unauthenticated_web_period_in_seconds` | integer | required by:<br>`throttle_unauthenticated_web_enabled` | Rate limit period in seconds. | | `throttle_unauthenticated_web_requests_per_period` | integer | required by:<br>`throttle_unauthenticated_web_enabled` | Max requests per period per IP. | diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index f14f1322184..a883c3c1613 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -13,7 +13,7 @@ You can configure the URL endpoint of the system hooks from the GitLab user inte 1. On the top bar, select **Menu > Admin**. 1. Select **System Hooks** (`/admin/hooks`). -Read more about [system hooks](../system_hooks/system_hooks.md). +Read more about [system hooks](../administration/system_hooks.md). ## List system hooks diff --git a/doc/api/templates/gitlab_ci_ymls.md b/doc/api/templates/gitlab_ci_ymls.md index abed008218c..8a55ee84402 100644 --- a/doc/api/templates/gitlab_ci_ymls.md +++ b/doc/api/templates/gitlab_ci_ymls.md @@ -135,7 +135,7 @@ Example response: ```json { "name": "Ruby", - "content": "# This file is a template, and might need editing before it works on your project.\n# Official language image. Look for the different tagged releases at:\n# https://hub.docker.com/r/library/ruby/tags/\nimage: \"ruby:2.5\"\n\n# Pick zero or more services to be used on all builds.\n# Only needed when using a docker container to run your tests in.\n# Check out: http://docs.gitlab.com/ee/ci/services/index.html\n - mysql:latest\n - redis:latest\n - postgres:latest\n\nvariables:\n POSTGRES_DB: database_name\n\n# Cache gems in between builds\ncache:\n paths:\n - vendor/ruby\n\n# This is a basic example for a gem or script which doesn't use\n# services such as redis or postgres\nbefore_script:\n - ruby -v # Print out ruby version for debugging\n # Uncomment next line if your rails app needs a JS runtime:\n # - apt-get update -q && apt-get install nodejs -yqq\n - bundle install -j $(nproc) --path vendor # Install dependencies into ./vendor/ruby\n\n# Optional - Delete if not using `rubocop`\nrubocop:\n script:\n - rubocop\n\nrspec:\n script:\n - rspec spec\n\nrails:\n variables:\n DATABASE_URL: \"postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB\"\n script:\n - rails db:migrate\n - rails db:seed\n - rails test\n\n# This deploy job uses a simple deploy flow to Heroku, other providers, e.g. AWS Elastic Beanstalk\n# are supported too: https://github.com/travis-ci/dpl\ndeploy:\n type: deploy\n environment: production\n script:\n - gem install dpl\n - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_PRODUCTION_KEY\n" + "content": "# This file is a template, and might need editing before it works on your project.\n# To contribute improvements to CI/CD templates, please follow the Development guide at:\n# https://docs.gitlab.com/ee/development/cicd/templates.html\n# This specific template is located at:\n# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml\n\n# Official language image. Look for the different tagged releases at:\n# https://hub.docker.com/r/library/ruby/tags/\nimage: ruby:latest\n\n# Pick zero or more services to be used on all builds.\n# Only needed when using a docker container to run your tests in.\n# Check out: http://docs.gitlab.com/ee/ci/docker/using_docker_images.html#what-is-a-service\nservices:\n - mysql:latest\n - redis:latest\n - postgres:latest\n\nvariables:\n POSTGRES_DB: database_name\n\n# Cache gems in between builds\ncache:\n paths:\n - vendor/ruby\n\n# This is a basic example for a gem or script which doesn't use\n# services such as redis or postgres\nbefore_script:\n - ruby -v # Print out ruby version for debugging\n # Uncomment next line if your rails app needs a JS runtime:\n # - apt-get update -q && apt-get install nodejs -yqq\n - bundle config set path 'vendor' # Install dependencies into ./vendor/ruby\n - bundle install -j $(nproc)\n\n# Optional - Delete if not using `rubocop`\nrubocop:\n script:\n - rubocop\n\nrspec:\n script:\n - rspec spec\n\nrails:\n variables:\n DATABASE_URL: \"postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB\"\n script:\n - rails db:migrate\n - rails db:seed\n - rails test\n\n# This deploy job uses a simple deploy flow to Heroku, other providers, e.g. AWS Elastic Beanstalk\n# are supported too: https://github.com/travis-ci/dpl\ndeploy:\n stage: deploy\n environment: production\n script:\n - gem install dpl\n - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_PRODUCTION_KEY\n" } ``` diff --git a/doc/api/users.md b/doc/api/users.md index de9af59de93..7b4962735e6 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -97,8 +97,18 @@ In addition, to exclude external users from the users' list, you can use the par GET /users?exclude_external=true ``` +To exclude [bot users for projects](../user/project/settings/project_access_tokens.md#bot-users-for-projects) +and [bot users for groups](../user/group/settings/group_access_tokens.md#bot-users-for-groups), you can use the +parameter `without_project_bots=true`. + +```plaintext +GET /users?without_project_bots=true +``` + ### For admins +> The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. + ```plaintext GET /users ``` @@ -151,7 +161,8 @@ GET /users "external": false, "private_profile": false, "current_sign_in_ip": "196.165.1.102", - "last_sign_in_ip": "172.127.2.22" + "last_sign_in_ip": "172.127.2.22", + "namespace_id": 1 }, { "id": 2, @@ -185,7 +196,8 @@ GET /users "external": false, "private_profile": false, "current_sign_in_ip": "10.165.1.102", - "last_sign_in_ip": "172.127.2.22" + "last_sign_in_ip": "172.127.2.22", + "namespace_id": 2 } ] ``` @@ -293,13 +305,18 @@ Parameters: "website_url": "", "organization": "", "job_title": "Operations Specialist", + "pronouns": "he/him", + "work_information": null, "followers": 1, - "following": 1 + "following": 1, + "local_time": "3:38 PM" } ``` ### For admin +> The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. + ```plaintext GET /users/:id ``` @@ -332,6 +349,11 @@ Example Responses: "website_url": "", "organization": "", "job_title": "Operations Specialist", + "pronouns": "he/him", + "work_information": null, + "followers": 1, + "following": 1, + "local_time": "3:38 PM", "last_sign_in_at": "2012-06-01T11:41:01Z", "confirmed_at": "2012-05-23T09:05:22Z", "theme_id": 1, @@ -355,7 +377,8 @@ Example Responses: "last_sign_in_ip": "172.127.2.22", "plan": "gold", "trial": true, - "sign_in_count": 1337 + "sign_in_count": 1337, + "namespace_id": 1 } ``` @@ -404,6 +427,8 @@ GET /users/:id?with_custom_attributes=true ## User creation +> The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. + Creates a new user. Note only administrators can create new users. Either `password`, `reset_password`, or `force_random_password` must be specified. If `reset_password` and `force_random_password` are @@ -459,6 +484,8 @@ Parameters: ## User modification +> The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. + Modifies an existing user. Only administrators can change attributes of a user. ```plaintext @@ -561,6 +588,13 @@ GET /user "twitter": "", "website_url": "", "organization": "", + "job_title": "", + "pronouns": "he/him", + "bot": false, + "work_information": null, + "followers": 0, + "following": 0, + "local_time": "3:38 PM", "last_sign_in_at": "2012-06-01T11:41:01Z", "confirmed_at": "2012-05-23T09:05:22Z", "theme_id": 1, @@ -577,12 +611,17 @@ GET /user "can_create_project": true, "two_factor_enabled": true, "external": false, - "private_profile": false + "private_profile": false, + "commit_email": "admin@example.com", } ``` +Users on [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `shared_runners_minutes_limit`, `extra_shared_runners_minutes_limit` parameters. + ## List current user (for admins) +> The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. + ```plaintext GET /user ``` @@ -632,7 +671,9 @@ Parameters: "private_profile": false, "commit_email": "john-codes@example.com", "current_sign_in_ip": "196.165.1.102", - "last_sign_in_ip": "172.127.2.22" + "last_sign_in_ip": "172.127.2.22", + "namespace_id": 1, + "note": null } ``` @@ -1566,7 +1607,7 @@ Returns: - `404 User Not Found` if the user cannot be found. - `403 Forbidden` when trying to unban a user that is not banned. -### Get user contribution events +## Get user contribution events Please refer to the [Events API documentation](events.md#get-user-contribution-events) @@ -1828,7 +1869,7 @@ POST /users/:user_id/personal_access_tokens | `user_id` | integer | yes | The ID of the user | | `name` | string | yes | The name of the personal access token | | `expires_at` | date | no | The expiration date of the personal access token in ISO format (`YYYY-MM-DD`) | -| `scopes` | array | yes | The array of scopes of the personal access token (`api`, `read_user`, `read_api`, `read_repository`, `write_repository`) | +| `scopes` | array | yes | The array of scopes of the personal access token. See [personal access token scopes](../user/profile/personal_access_tokens.md#personal-access-token-scopes) for possible values. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" \ diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index ba0bee04f35..963977be549 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -150,7 +150,7 @@ Following the GitLab [Go standards and style guidelines](../../../development/go The design and development of the registry database adhere to the GitLab [database guidelines](../../../development/database/). Being a Go application, the required tooling to support the database will have to be developed, such as for running database migrations. -Running *online* and [*post deployment*](../../../development/post_deployment_migrations.md) migrations is already supported by the registry CLI, as described in the [documentation](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/database-migrations.md). +Running *online* and [*post deployment*](../../../development/database/post_deployment_migrations.md) migrations is already supported by the registry CLI, as described in the [documentation](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/database-migrations.md). #### Partitioning diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 098ec9bdd02..c2b18dc6467 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -74,6 +74,13 @@ build: - if: $CI_COMMIT_TAG ``` +If you authenticate against the [Dependency Proxy](../../user/packages/dependency_proxy/index.md#authenticate-within-cicd), +you must add the corresponding CI/CD variables for authentication to the `config.json` file: + +```yaml +- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"},\"$CI_DEPENDENCY_PROXY_SERVER\":{\"auth\":\"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json +``` + ### Building an image with kaniko behind a proxy If you use a custom GitLab Runner behind an http(s) proxy, kaniko needs to be set diff --git a/doc/ci/environments/deployment_approvals.md b/doc/ci/environments/deployment_approvals.md index 45af78aa036..2a5927a96c2 100644 --- a/doc/ci/environments/deployment_approvals.md +++ b/doc/ci/environments/deployment_approvals.md @@ -15,7 +15,7 @@ This feature is in a [Beta](../../policy/alpha-beta-support.md#beta-features) st It may be useful to require additional approvals before deploying to certain protected environments (for example, production). This pre-deployment approval requirement is useful to accommodate testing, security, or compliance processes that must happen before each deployment. -When a protected environment requires one or more approvals, all deployments to that environment become blocked and wait for the required approvals before running. +When a protected environment requires one or more approvals, all deployments to that environment become blocked and wait for the required approvals from the `Allowed to Deploy` list before running. NOTE: See the [epic](https://gitlab.com/groups/gitlab-org/-/epics/6832) for planned features. @@ -52,6 +52,19 @@ Example: ### Require approvals for a protected environment +There are two ways to configure the approval requirements: + +- [Unified approval setting](#unified-approval-setting) ... You can define who can execute **and** approve deployments. + This is useful when there is no separation of duties between executors and approvers in your organization. +- [Multiple approval rules](#multiple-approval-rules) ... You can define who can execute **or** approve deployments. + This is useful when there is a separation of duties between executors and approvers in your organization. + +NOTE: +Multiple approval rules is a more flexible option than the unified approval setting, thus both configurations shouldn't +co-exist and multiple approval rules takes the precedence over the unified approval setting if it happens. + +#### Unified approval setting + NOTE: At this time, it is not possible to require approvals for an existing protected environment. The workaround is to unprotect the environment and configure approvals when re-protecting the environment. @@ -77,6 +90,35 @@ NOTE: To protect, update, or unprotect an environment, you must have at least the Maintainer role. +#### Multiple approval rules + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345678) in GitLab 14.10 with a flag named `deployment_approval_rules`. Disabled by default. + +1. Using the [REST API](../../api/group_protected_environments.md#protect-an-environment). + 1. `deploy_access_levels` represents which entity can execute the deployment job. + 1. `approval_rules` represents which entity can approve the deployment job. + +After this is configured, all jobs deploying to this environment automatically go into a blocked state and wait for approvals before running. Ensure that the number of required approvals is less than the number of users allowed to deploy. + +Example: + +```shell +curl --header 'Content-Type: application/json' --request POST \ + --data '{"name": "production", "deploy_access_levels": [{"group_id": 138}], "approval_rules": [{"group_id": 134}, {"group_id": 135, "required_approvals": 2}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/128/protected_environments" +``` + +With this setup: + +- The operator group (`group_id: 138`) has permission to execute the deployment jobs to the `production` environment in the organization (`group_id: 128`). +- The QA tester group (`group_id: 134`) and security group (`group_id: 135`) have permission to approve the deployment jobs to the `production` environment in the organization (`group_id: 128`). +- Unless two approvals from security group and one approval from QA tester group have been collected, the operator group can't execute the deployment jobs. + +NOTE: +To protect, update, or unprotect an environment, you must have at least the +Maintainer role. + ## Approve or reject a deployment > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342180/) in GitLab 14.9 @@ -99,6 +141,10 @@ To approve or reject a deployment to a protected environment using the UI: 1. In the deployment's row, select **Approval options** (**{thumb-up}**). 1. Select **Approve** or **Reject**. +NOTE: +This feature might not work as expected when [Multiple approval rules](#multiple-approval-rules) is configured. +See the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/355708) for planned improvement. + ### Approve or reject a deployment using the API Prerequisites: @@ -127,11 +173,14 @@ curl --data "status=approved&comment=Looks good to me" \ ### Using the API -Use the [Deployments API](../../api/deployments.md) to see deployments. +Use the [Deployments API](../../api/deployments.md#get-a-specific-deployment) to see deployments. - The `status` field indicates if a deployment is blocked. -- The `pending_approval_count` field indicates how many approvals are remaining to run a deployment. -- The `approvals` field contains the deployment's approvals. +- When the [unified approval setting](#unified-approval-setting) is configured: + - The `pending_approval_count` field indicates how many approvals are remaining to run a deployment. + - The `approvals` field contains the deployment's approvals. +- When the [multiple approval rules](#multiple-approval-rules) is configured: + - The `approval_summary` field contains the current approval status per rule. ## Related features diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index c2612b6ff16..3822b32181f 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -558,6 +558,55 @@ Because `stop_review_app` is set to `auto_stop_in: 1 week`, if a merge request is inactive for more than a week, GitLab automatically triggers the `stop_review_app` job to stop the environment. +#### Multiple stop actions for an environment + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22456) in GitLab 14.10 [with a flag](../../administration/feature_flags.md) named `environment_multiple_stop_actions`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `environment_multiple_stop_actions`. +On GitLab.com, this feature is not available. We are enabling in phases and the status can be tracked in [issue 358911](https://gitlab.com/gitlab-org/gitlab/-/issues/358911). + +This feature is useful when you need to perform multiple **parallel** stop actions on an environment. + +To configure multiple stop actions on an environment, specify the [`on_stop`](../yaml/index.md#environmenton_stop) +keyword across multiple [deployment jobs](../jobs/index.md#deployment-jobs) for the same `environment`, as defined in the `.gitlab-ci.yml` file. + +When an environment is stopped, the matching `on_stop` actions from *successful deployment jobs* alone are run in parallel in no particular order. + +In the following example, for the `test` environment there are two deployment jobs `deploy-to-cloud-a` +and `deploy-to-cloud-b`. + +```yaml +deploy-to-cloud-a: + script: echo "Deploy to cloud a" + environment: + name: test + on_stop: teardown-cloud-a + +deploy-to-cloud-b: + script: echo "Deploy to cloud b" + environment: + name: test + on_stop: teardown-cloud-b + +teardown-cloud-a: + script: echo "Delete the resources in cloud a" + environment: + name: test + action: stop + when: manual + +teardown-cloud-b: + script: echo "Delete the resources in cloud b" + environment: + name: test + action: stop + when: manual +``` + +When the environment is stopped, the system runs `on_stop` actions +`teardown-cloud-a` and `teardown-cloud-b` in parallel. + #### View a deployment's scheduled stop time You can view a deployment's expiration date in the GitLab UI. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 5fca3513ff7..389429f3f0f 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -277,3 +277,19 @@ read_secrets: ```  + +### Limit token access to Vault secrets + +You can control `CI_JOB_JWT` access to Vault secrets by using Vault protections +and GitLab features. For example, restrict the token by: + +- Using Vault [bound_claims](https://www.vaultproject.io/docs/auth/jwt#bound-claims) + for specific groups using `group_claim`. +- Hard coding values for Vault bound claims based on the `user_login` and `user_email` + of specific users. +- Setting Vault time limits for TTL of the token as specified in [`token_explicit_max_ttl`](https://www.vaultproject.io/api/auth/jwt#token_explicit_max_ttl), + where the token expires after authentication. +- Scoping the JWT to [GitLab projected branches](../../../user/project/protected_branches.md) + that are restricted to a subset of project users. +- Scoping the JWT to [GitLab projected tags](../../../user/project/protected_tags.md), + that are restricted to a subset of project users. diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 6dbec0dfc8b..0735029f0be 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -104,7 +104,7 @@ sudo apt install acl ### Add SSH key -Let's suppose we want to deploy our app to the production server from a private repository on GitLab. First, we need to [generate a new SSH key pair **with no passphrase**](../../../ssh/index.md) for the deployer user. +Let's suppose we want to deploy our app to the production server from a private repository on GitLab. First, we need to [generate a new SSH key pair **with no passphrase**](../../../user/ssh.md) for the deployer user. After that, we need to copy the private key, which will be used to connect to our server as the deployer user with SSH, to be able to automate our deployment process: diff --git a/doc/ci/index.md b/doc/ci/index.md index d0c2e1986b2..05e97613384 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -38,79 +38,76 @@ read the [Introduction to CI/CD with GitLab](introduction/index.md). <iframe src="https://www.youtube.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen="true"> </iframe> </figure> -## GitLab CI/CD concepts +## Concepts GitLab CI/CD uses a number of concepts to describe and run your build and deploy. -| Concept | Description | -|:--------------------------------------------------------|:-------------------------------------------------------------------------------| -| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | -| [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | +| Concept | Description | +|:--------------------------------------------------------|:--------------------------------------------------------------------------------------| +| [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | +| [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | | [Environments](environments/index.md) | Deploy your application to different environments (for example, staging, production). | -| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | -| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | -| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | -| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | -| [Test cases](test_cases/index.md) | Create testing scenarios. | +| [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | +| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | +| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | +| [Pipeline efficiency](pipelines/pipeline_efficiency.md) | Configure your pipelines to run quickly and efficiently. | +| [Test cases](test_cases/index.md) | Create testing scenarios. | -## GitLab CI/CD configuration +## Configuration GitLab CI/CD supports numerous configuration options: -| Configuration | Description | -|:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| -| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#specify-a-custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | -| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | -| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | -| [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | -| [Merge request pipelines](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | -| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | -| [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | +| Configuration | Description | +|:---------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| +| [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | +| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#specify-a-custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | +| [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | +| [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | +| [Merge request pipelines](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | +| [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | +| [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | +| [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | Certain operations can only be performed according to the [user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. -## GitLab CI/CD features +## Features GitLab CI/CD features, grouped by DevOps stage, include: -| Feature | Description | -|:------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------| -| **Configure** | | -| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | -| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Verify** | | -| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | -| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | -| [CI services](services/index.md) | Link Docker containers with your base image. | -| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | -| [Interactive Web Terminals](interactive_web_terminal/index.md) **(FREE SELF)** | Open an interactive web terminal to debug the running jobs. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | -| [Unit test reports](unit_test_reports.md) | Identify test failures directly on merge requests. | -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Release** | | -| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | -| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | -| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | -| [Deploy boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature Flags](../operations/feature_flags.md) | Deploy your features behind Feature Flags. | -| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | -| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | -|-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| -| **Secure** | | -| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | -| [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities. | -| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | -| [License Compliance](../user/compliance/license_compliance/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | -| [Security Test reports](../user/application_security/index.md) **(ULTIMATE)** | Check for app vulnerabilities. | - -## GitLab CI/CD examples +| Feature | Description | +|:---------------------------------------------------------------------------------------------|:------------| +| **Configure** | | +| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | +| [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | +| [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. | +| **Verify** | | +| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | +| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [CI services](services/index.md) | Link Docker containers with your base image. | +| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +| [Unit test reports](unit_test_reports.md) | Identify test failures directly on merge requests. | +| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | +| **Release** | | +| [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | +| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | +| [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | +| [Deploy boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [Feature Flags](../operations/feature_flags.md) | Deploy your features behind Feature Flags. | +| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | +| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | +| [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | +| **Secure** | | +| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | +| [Container Scanning](../user/application_security/container_scanning/index.md) | Check your Docker containers for known vulnerabilities. | +| [Dependency Scanning](../user/application_security/dependency_scanning/index.md) | Analyze your dependencies for known vulnerabilities. | +| [License Compliance](../user/compliance/license_compliance/index.md) | Search your project dependencies for their licenses. | +| [Security Test reports](../user/application_security/index.md) | Check for app vulnerabilities. | + +## Examples See the [CI/CD examples](examples/index.md) page for example project code and tutorials for using GitLab CI/CD with various: @@ -119,7 +116,7 @@ using GitLab CI/CD with various: - Languages - Platforms -## GitLab CI/CD Administration +## Administration You can change the default behavior of GitLab CI/CD for: @@ -136,7 +133,7 @@ Learn more about GitLab CI/CD: - [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/). - [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/). -- [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) +- [Five teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/). - If you use VS Code to edit your GitLab CI/CD configuration, the [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) helps you [validate your configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration) diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index b0002f559ba..8e5c8c8ab45 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -83,8 +83,7 @@ to be accessed by authenticating with the current project's job token. By defaul the token scope only allows access to the same project where the token comes from. Other projects can be added and removed by maintainers with access to both projects. -This setting is enabled by default for all new projects, and disabled by default in projects -created before GitLab 14.1. It is strongly recommended that project maintainers enable this +This setting is disabled by default for all new projects. It is recommended that project maintainers enable this setting at all times, and configure the allowlist for cross-project access if needed. For example, when the setting is enabled, jobs in a pipeline in project `A` have diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 4fa251eb3cf..b8129e1cf18 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -43,6 +43,18 @@ Clicking an individual job shows you its job log, and allows you to: - Retry the job. - Erase the job log. +### View all jobs in a project + +> - An improved view was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293862) in GitLab 14.10, [with a flag](../../administration/feature_flags.md) named `jobs_table_vue`. Disabled by default. +> - The job status filter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82539) in GitLab 14.10, [with a flag](../../administration/feature_flags.md) named `jobs_table_vue_search`. Disabled by default. + +To view the full list of jobs that ran in a project: + +1. On the top bar, select **Menu > Projects** and find the project. +1. On the left sidebar, select **CI/CD > Jobs**. + +You can filter the list by [job status](#the-order-of-jobs-in-a-pipeline). + ## See why a job failed When a pipeline fails or is allowed to fail, there are several places where you @@ -63,9 +75,9 @@ You can also see the reason it failed on the Job detail page. The order of jobs in a pipeline depends on the type of pipeline graph. - For [full pipeline graphs](../pipelines/index.md#view-full-pipeline-graph), jobs are sorted by name. -- For [pipeline mini graphs](../pipelines/index.md#pipeline-mini-graphs), jobs are sorted by severity and then by name. +- For [pipeline mini graphs](../pipelines/index.md#pipeline-mini-graphs), jobs are sorted by status, and then by name. -The order of severity is: +The job status order is: - failed - warning @@ -84,6 +96,9 @@ For example: ## Job name limitations +> - [Enabled 255-character job length on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/342800) in GitLab 14.5. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/344665) in GitLab 14.10. [Feature flag `ci_validate_job_length`](https://gitlab.com/gitlab-org/gitlab/-/issues/344665) removed. + You can't use these keywords as job names: - `image` @@ -99,9 +114,7 @@ You can't use these keywords as job names: - `false` - `nil` -Job names must be 255 characters or less. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342800) -in GitLab 14.5, [with a feature flag](../../administration/feature_flags.md) named `ci_validate_job_length`. -Enabled by default. To disable it, ask an administrator to [disable the feature flag](../../administration/feature_flags.md). +Job names must be 255 characters or fewer. Use unique names for your jobs. If multiple jobs have the same name, only one is added to the pipeline, and it's difficult to predict which one is chosen. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 256e669a66e..0811cc87e91 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -25,8 +25,7 @@ configuration added with the [`includes` keyword](yaml/index.md#include). To check CI/CD configuration with the CI lint tool: 1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select: - - **CI/CD > Pipelines** +1. On the left sidebar, select **CI/CD > Pipelines**. 1. In the top right, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Validate**. @@ -47,8 +46,7 @@ Prerequisites: To simulate a pipeline: 1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select: - - **CI/CD > Pipelines** +1. On the left sidebar, select **CI/CD > Pipelines**. 1. In the top right, select **CI lint**. 1. Paste a copy of the CI/CD configuration you want to check into the text box. 1. Select **Simulate pipeline creation for the default branch**. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index bfa2eb54806..19b1a6cad1f 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -199,7 +199,7 @@ GitLab takes advantage of our connected ecosystem to automatically pull these ki your merge requests, pipeline details pages, and other locations. You may find that you actually don't need to configure anything to have these appear. -If they aren't working as expected, or if you'd like to see what's available, our [CI feature index](../index.md#gitlab-cicd-features) has the full list +If they aren't working as expected, or if you'd like to see what's available, our [CI/CD feature index](../index.md#features) has the full list of bundled features and links to the documentation for each. ### Templates diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index d80b745e6bc..75408de2721 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -20,19 +20,20 @@ Branch pipelines: - Run when you push a new commit to a branch. - Are the default type of pipeline. - Have access to [some predefined variables](../variables/predefined_variables.md). -- Have access to [protected variables](../variables/index.md#protect-a-cicd-variable). +- Have access to [protected variables](../variables/index.md#protect-a-cicd-variable) and [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). Merge request pipelines: - Run when you: - - Create a new merge request. + - Create a new merge request from a source branch with one or more commits. - Push a new commit to the source branch for a merge request. - Select **Run pipeline** from the **Pipelines** tab in a merge request. This option - is only available when merge request pipelines are configured for the pipeline. + is only available when merge request pipelines are configured for the pipeline + and the source branch has at least one commit. - Do not run by default. The jobs in the CI/CD configuration file [must be configured](#prerequisites) to run in merge request pipelines. - Have access to [more predefined variables](#available-predefined-variables). -- Do not have access to [protected variables](../variables/index.md#protect-a-cicd-variable). +- Do not have access to [protected variables](../variables/index.md#protect-a-cicd-variable) or [protected runners](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). Both of these types of pipelines can appear on the **Pipelines** tab of a merge request. @@ -147,7 +148,7 @@ a warning that you must accept before you can trigger the pipeline. Prerequisites: - You must be a member of the parent project and have at least the [Developer role](../../user/permissions.md). -- The fork project must be [visible](../../public_access/public_access.md) to the +- The fork project must be [visible](../../user/public_access.md) to the user running the pipeline. Otherwise, the **Pipelines** tab does not display in the merge request. diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 2163527e3ca..e6af9292fe1 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -282,7 +282,7 @@ tag in a different project. Prerequisites: -- The upstream project must be [public](../../public_access/public_access.md). +- The upstream project must be [public](../../user/public_access.md). - The user must have the Developer role in the upstream project. diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 7960d0afa85..b6ea16ae224 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -76,7 +76,14 @@ To avoid this scenario: 1. Select the **Skip outdated deployment jobs** checkbox. 1. Select **Save changes**. -Older deployment jobs are skipped when a new deployment starts. +When a new deployment starts, older deployment jobs are skipped. Skipped jobs are labeled: + +- `forward deployment failure` in the pipeline view. +- `The deployment job is older than the previously succeeded deployment job, and therefore cannot be run` + when viewing the completed job. + +Job age is determined by the job start time, not the commit time, so a newer commit +can be skipped in some circumstances. For more information, see [Deployment safety](../environments/deployment_safety.md). @@ -223,7 +230,7 @@ Setting the regular expression this way takes precedence over project settings. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) -for use in GitLab 14.9, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. +in GitLab 14.9, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/17633) in GitLab 15.0. You can add test coverage results to merge requests using the Project's CI/CD settings: diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index ed29b33bc3a..56e8b96d11f 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -192,9 +192,12 @@ After you have the route mapping set up, it takes effect in the following locati > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10761) in GitLab 12.0. > - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. -> - It's [deployed behind a feature flag](../../user/feature_flags.md), enabled by default. +> - It's [deployed behind a feature flag](../../user/feature_flags.md), `anonymous_visual_review_feedback`, disabled by default. > - It's enabled on GitLab.com. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-visual-reviews). **(PREMIUM SELF)** + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `anonymous_visual_review_feedback`. With Visual Reviews, members of any team (Product, Design, Quality, and so on) can provide feedback comments through a form in your review apps. The comments are added to the merge request that triggered the review app. @@ -284,30 +287,12 @@ can supply the ID by either: - Dynamically adding the `data-merge-request-id` value during the build of the app. - Supplying it manually through the visual review form in the app. -### Enable or disable Visual Reviews **(PREMIUM SELF)** - -Visual Reviews is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To disable it: - -```ruby -Feature.disable(:anonymous_visual_review_feedback) -``` - -To enable it: - -```ruby -Feature.enable(:anonymous_visual_review_feedback) -``` - ### Authentication for Visual Reviews > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/42750#note_317271120) in GitLab 12.10. To enable visual reviews for private and internal projects, set the -[`data-require-auth` variable](#enable-or-disable-visual-reviews) to `true`. When enabled, +[`data-require-auth` variable](#configure-review-apps-for-visual-reviews) to `true`. When enabled, the user must enter a [personal access token](../../user/profile/personal_access_tokens.md) with `api` scope before submitting feedback. diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md index aa7e268e800..6082a17d001 100644 --- a/doc/ci/runners/runners_scope.md +++ b/doc/ci/runners/runners_scope.md @@ -37,22 +37,31 @@ If you are using GitLab.com: - The shared runners consume the [CI/CD minutes](../pipelines/cicd_minutes.md) included with your account. -### Enable shared runners +### Enable shared runners for a project On GitLab.com, [shared runners](index.md) are enabled in all projects by default. -On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) -and [register](https://docs.gitlab.com/runner/register/index.html) them. +On self-managed instances of GitLab, an administrator can +[enable them for all new projects](../../user/admin_area/settings/continuous_integration.md#enable-shared-runners-for-new-projects). -You can also enable shared runners for individual projects. +For existing projects, an administrator must +[install](https://docs.gitlab.com/runner/install/index.html) and +[register](https://docs.gitlab.com/runner/register/index.html) them. -To enable shared runners: +To enable shared runners for a project: 1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. 1. Select **Enable shared runners for this project**. -### Disable shared runners +### Enable shared runners for a group + +To enable shared runners for a group: + +1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. +1. Select **Enable shared runners for this group**. + +### Disable shared runners for a project You can disable shared runners for individual projects or for groups. You must have the Owner role for the project @@ -68,6 +77,8 @@ Shared runners are automatically disabled for a project: - If the shared runners setting for the parent group is disabled, and - If overriding this setting is not permitted at the project level. +### Disable shared runners for a group + To disable shared runners for a group: 1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. @@ -78,7 +89,7 @@ To disable shared runners for a group: NOTE: To re-enable the shared runners for a group, turn on the **Enable shared runners for this group** toggle. -Then, an owner or maintainer must explicitly change this setting +Then, a user with the Owner or Maintainer role must explicitly change this setting for each project subgroup or project. ### How shared runners pick jobs @@ -143,6 +154,8 @@ Group runners process jobs by using a first in, first out ([FIFO](https://en.wik ### Create a group runner +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19819) in GitLab 14.10, path changed from **Settings > CI/CD > Runners**. + You can create a group runner for your self-managed GitLab instance or for GitLab.com. You must have the Owner role for the group. @@ -150,7 +163,7 @@ To create a group runner: 1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). 1. Go to the group you want to make the runner work for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. On the left sidebar, select **CI/CD > Runners**. 1. Note the URL and token. 1. [Register the runner](https://docs.gitlab.com/runner/register/). @@ -163,7 +176,7 @@ You can do this for your self-managed GitLab instance or for GitLab.com. You must have the Owner role for the group. 1. Go to the group where you want to view the runners. -1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. On the left sidebar, select **CI/CD > Runners**. 1. The following fields are displayed. | Attribute | Description | @@ -186,7 +199,7 @@ You can pause or remove a group runner for your self-managed GitLab instance or You must have the Owner role for the group. 1. Go to the group you want to remove or pause the runner for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. On the left sidebar, select **CI/CD > Runners**. 1. Click **Pause** or **Remove runner**. - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. - From the group view, you cannot remove a runner that is assigned to more than one project. diff --git a/doc/ci/runners/saas/macos/environment.md b/doc/ci/runners/saas/macos/environment.md index d1943a487a7..4d209fe4cd6 100644 --- a/doc/ci/runners/saas/macos/environment.md +++ b/doc/ci/runners/saas/macos/environment.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# VM instances and images for SaaS runners on macOS **(FREE SAAS)** +# VM instances and images for SaaS runners on macOS **(PREMIUM SAAS)** When you use SaaS runners on macOS: @@ -35,6 +35,7 @@ Each image is running a specific version of macOS and Xcode. | macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | | macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | | macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | +| macos-11-xcode-13 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/monterey.yml> ### Image update policy diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 885a76a7b46..bad9da960b2 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# SaaS runners on macOS (beta) **(FREE SAAS)** +# SaaS runners on macOS (beta) **(PREMIUM SAAS)** SaaS runners on macOS are in [Beta](../../../policy/alpha-beta-support.md#beta-features) and shouldn't be relied upon for mission-critical production jobs. diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 5595559b5f0..ba395108966 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -92,8 +92,8 @@ To configure your Vault server: specified when the authentication method was configured. - `VAULT_AUTH_PATH` - Optional. The path where the authentication method is mounted, default is `jwt`. - `VAULT_NAMESPACE` - Optional. The [Vault Enterprise namespace](https://www.vaultproject.io/docs/enterprise/namespaces) to use for reading secrets and authentication. - If no namespace is specified, Vault uses the `root` ("`/`") namespace. - The setting is ignored by Vault Open Source. + If no namespace is specified, Vault uses the `root` ("`/`") namespace. + The setting is ignored by Vault Open Source. NOTE: Support for providing these values in the user interface [is tracked in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/218677). diff --git a/doc/ci/secure_files/index.md b/doc/ci/secure_files/index.md new file mode 100644 index 00000000000..3cc38a8b74c --- /dev/null +++ b/doc/ci/secure_files/index.md @@ -0,0 +1,98 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Project-level Secure Files **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8. [Deployed behind the `ci_secure_files` flag](../../administration/feature_flags.md), disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) +named `ci_secure_files`. Limited to 100 secure files per project. Files must be smaller +than 5 MB. The feature is not ready for production use. + +You can securely store files for use in CI/CD pipelines as "secure files". These files +are stored securely outside of your project's repository, and are not version controlled. +It is safe to store sensitive information in these files. Secure files support both +plain text and binary file types. + +Secure files can be [downloaded and used by CI/CD jobs](#use-secure-files-in-cicd-jobs) +by using the [load-secure-files](https://gitlab.com/gitlab-org/incubation-engineering/devops-for-mobile-apps/load-secure-files) +tool. + +NOTE: +This feature is in active development and is likely to change, potentially in a breaking way. +Additional features and capabilities are planned. + +## Add a secure file to a project + +To add a secure file to a project, use the [Secure Files API](../../api/secure_files.md#create-secure-file). + +Send a POST request to the secure files endpoint for your project: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/:project_id/secure_files" --form "name=myfile.jks" --form "file=@/path/to/file/myfile.jks" +``` + +The response returns all of the metadata for the file you just uploaded. For example: + +```json +{ + "id": 1, + "name": "myfile.jks", + "checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac", + "checksum_algorithm": "sha256", + "permissions": "read_only", + "created_at": "2022-02-22T22:22:22.222Z" +} +``` + +## List all secure files in a project + +To list all secure files in a project, use the [Secure Files API](../../api/secure_files.md#list-project-secure-files). + +Send a GET request to the secure files endpoint for your project: + +```shell +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/:project_id/secure_files" +``` + +The response returns an array of all of the secure files in the project. For example: + +```json +[{ + "id": 1, + "name": "myfile.jks", + "checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac", + "checksum_algorithm": "sha256", + "permissions": "read_only", + "created_at": "2022-02-22T22:22:22.222Z" +}] +``` + +## Use secure files in CI/CD jobs + +To use your secure files in a CI/CD job, you must use the [`load-secure-files`](https://gitlab.com/gitlab-org/incubation-engineering/devops-for-mobile-apps/load-secure-files) +tool to download the files in the job. After they are downloaded, you can use them +with your other script commands. + +Add a command in the `script` section of your job to download the `load-secure-files` tool +and execute it. The files download into a `.secure_files` directory in the root of the project. +To change the download location for the secure files, set the path in the `SECURE_FILES_DOWNLOAD_PATH` +[CI/CD variable](../variables/index.md). + +For example: + +```yaml +test: + variables: + SECURE_FILES_DOWNLOAD_PATH: './where/files/should/go/' + script: + - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/devops-for-mobile-apps/load-secure-files/-/raw/main/installer" | bash +``` diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index 817263374f1..30b688c0b96 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -48,7 +48,7 @@ contained) and you want to deploy your code in a private server, you need a way to access it. In this case, you can use an SSH key pair. 1. You first must create an SSH key pair. For more information, follow - the instructions to [generate an SSH key](../../ssh/index.md#generate-an-ssh-key-pair). + the instructions to [generate an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. @@ -124,7 +124,7 @@ on, and use that key for all projects that are run on this machine. ``` 1. Generate the SSH key pair as described in the instructions to - [generate an SSH key](../../ssh/index.md#generate-an-ssh-key-pair). + [generate an SSH key](../../user/ssh.md#generate-an-ssh-key-pair). **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 14350ac884f..029bd20c553 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -77,7 +77,7 @@ If a test failed in the project's default branch in the last 14 days, a message To enable the Unit test reports in merge requests, you must add [`artifacts:reports:junit`](yaml/artifacts_reports.md#artifactsreportsjunit) -in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. +in `.gitlab-ci.yml`, and specify the paths of the generated test reports. The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). In the following examples, the job in the `test` stage runs and GitLab diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index e4e562bb005..30d53a340a9 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -346,6 +346,7 @@ To mask a variable: 1. Select the **Mask variable** checkbox. 1. Select **Update variable**. +The method used to mask variables [limits what can be included in a masked variable](](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784#note_106756757)). The value of the variable must: - Be a single line. diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 0f3461b3674..3b504b02fae 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -58,6 +58,7 @@ There are also a number of [variables you can use to configure runner behavior]( | `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Available if [`environment:url`](../yaml/index.md#environmenturl) is set. | | `CI_ENVIRONMENT_ACTION` | 13.11 | all | The action annotation specified for this job's environment. Available if [`environment:action`](../yaml/index.md#environmentaction) is set. Can be `start`, `prepare`, or `stop`. | | `CI_ENVIRONMENT_TIER` | 14.0 | all | The [deployment tier of the environment](../environments/index.md#deployment-tier-of-environments) for this job. | +| `CI_GITLAB_FIPS_MODE` | 14.10 | all | The configuration setting for whether FIPS mode is enabled in the GitLab instance. | | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | @@ -105,7 +106,7 @@ There are also a number of [variables you can use to configure runner behavior]( | `CI_RUNNER_EXECUTABLE_ARCH` | all | 10.6 | The OS/architecture of the GitLab Runner executable. Might not be the same as the environment of the executor. | | `CI_RUNNER_ID` | 8.10 | 0.5 | The unique ID of the runner being used. | | `CI_RUNNER_REVISION` | all | 10.6 | The revision of the runner running the job. | -| `CI_RUNNER_SHORT_TOKEN` | all | 12.3 | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID. | +| `CI_RUNNER_SHORT_TOKEN` | all | 12.3 | The runner's unique ID, used to authenticate new job requests. In [GitLab 14.9](https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/2251) and later, the token contains a prefix, and the first 17 characters are used. Prior to 14.9, the first eight characters are used. | | `CI_RUNNER_TAGS` | 8.10 | 0.5 | A comma-separated list of the runner tags. | | `CI_RUNNER_VERSION` | all | 10.6 | The version of the GitLab Runner running the job. | | `CI_SERVER_HOST` | 12.1 | all | The host of the GitLab instance URL, without protocol or port. For example `gitlab.example.com`. | @@ -113,6 +114,7 @@ There are also a number of [variables you can use to configure runner behavior]( | `CI_SERVER_PORT` | 12.8 | all | The port of the GitLab instance URL, without host or protocol. For example `8080`. | | `CI_SERVER_PROTOCOL` | 12.8 | all | The protocol of the GitLab instance URL, without host or port. For example `https`. | | `CI_SERVER_REVISION` | all | all | GitLab revision that schedules jobs. | +| `CI_SERVER_TLS_CA_FILE` | all | all | File containing the CA certificate to verify the GitLab server. | | `CI_SERVER_URL` | 12.7 | all | The base URL of the GitLab instance, including protocol and port. For example `https://gitlab.example.com:8080`. | | `CI_SERVER_VERSION_MAJOR` | 11.4 | all | The major version of the GitLab instance. For example, if the GitLab version is `13.6.1`, the `CI_SERVER_VERSION_MAJOR` is `13`. | | `CI_SERVER_VERSION_MINOR` | 11.4 | all | The minor version of the GitLab instance. For example, if the GitLab version is `13.6.1`, the `CI_SERVER_VERSION_MINOR` is `6`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 4f304a10256..64d61ec9f11 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -137,8 +137,6 @@ The following variables are known as "persisted": - `CI_JOB_ID` - `CI_JOB_TOKEN` - `CI_JOB_STARTED_AT` -- `CI_BUILD_ID` -- `CI_BUILD_TOKEN` - `CI_REGISTRY_USER` - `CI_REGISTRY_PASSWORD` - `CI_REPOSITORY_URL` diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index e010dd21b9e..b30b13e1ec0 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -80,9 +80,14 @@ GitLab can display the results of one or more reports in: - The [security dashboard](../../user/application_security/security_dashboard/index.md). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -## `artifacts:reports:cobertura` +## `artifacts:reports:cobertura` (DEPRECATED) -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78132) in GitLab 14.9. + +WARNING: +This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78132) in GitLab +14.9 and replaced with `artifacts:reports:coverage_report` in 14.10. The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). The collected Cobertura coverage reports upload to GitLab as an artifact. @@ -93,6 +98,28 @@ GitLab can display the results of one or more reports in the merge request Cobertura was originally developed for Java, but there are many third-party ports for other languages such as JavaScript, Python, and Ruby. +## `artifacts:reports:coverage_report` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344533) in GitLab 14.9. + +Use `coverage_report` to collect coverage report in Cobertura format, similar to `artifacts:reports:cobertura`. + +NOTE: +`artifacts:reports:coverage_report` cannot be used at the same time with `artifacts:reports:cobertura`. + +```yaml +artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml +``` + +The collected coverage report is uploaded to GitLab as an artifact. + +GitLab can display the results of coverage report in the merge request +[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). + ## `artifacts:reports:codequality` > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index a3b91da8f08..b335e8e8545 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -395,13 +395,13 @@ When no rules evaluate to true, the pipeline does not run. ```yaml workflow: rules: - - if: $CI_COMMIT_MESSAGE =~ /-draft$/ + - if: $CI_COMMIT_TITLE =~ /-draft$/ when: never - if: $CI_PIPELINE_SOURCE == "merge_request_event" - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -In this example, pipelines run if the commit message does not have `-drafts` in it +In this example, pipelines run if the commit title (first line of the commit message) does not end with `-draft` and the pipeline is for either: - A merge request @@ -2484,8 +2484,10 @@ Use `changes` in pipelines with the following refs: - Paths to files. - Wildcard paths for single directories, for example `path/to/directory/*`, or a directory and all its subdirectories, for example `path/to/directory/**/*`. -- Wildcard ([glob](https://en.wikipedia.org/wiki/Glob_(programming))) paths for all +- Wildcard [glob](https://en.wikipedia.org/wiki/Glob_(programming)) paths for all files with the same extension or multiple extensions, for example `*.md` or `path/to/directory/*.{rb,py,sh}`. + See the [Ruby `fnmatch` documentation](https://docs.ruby-lang.org/en/master/File.html#method-c-fnmatch) + for the supported syntax list. - Wildcard paths to files in the root directory, or all directories, wrapped in double quotes. For example `"*.json"` or `"**/*.json"`. @@ -2620,9 +2622,11 @@ Multiple runners must exist, or a single runner must be configured to run multip **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: +**Possible inputs**: An array of hashes of variables: -- A numeric value from `2` to `50`. +- The variable names can use only numbers, letters, and underscores (`_`). +- The values must be either a string, or an array of strings. +- The number of permutations cannot exceed 50. **Example of `parallel:matrix`**: @@ -2692,18 +2696,19 @@ you can use this image from the GitLab Container Registry: `registry.gitlab.com/ **Example of `release` keyword**: - ```yaml - release_job: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG # Run this job when a tag is created manually - script: - - echo "Running the release job." - release: - name: 'Release $CI_COMMIT_TAG' - description: 'Release created using the release-cli.' - ``` +```yaml +release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG # Run this job when a tag is created manually + script: + - echo "Running the release job." + release: + tag_name: $CI_COMMIT_TAG + name: 'Release $CI_COMMIT_TAG' + description: 'Release created using the release-cli.' +``` This example creates a release: @@ -3100,7 +3105,7 @@ to specific files. WARNING: You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. You can use `rules: changes` with other pipeline types, but `rules: changes` always -evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, manual pipelines, and so on do **not** have a Git `push` event associated with them. A `rules: changes` job is **always** added to those pipelines if there is no `if` that limits the job to branch or merge request pipelines. @@ -3709,7 +3714,7 @@ and [multi-project pipelines](../pipelines/multi_project_pipelines.md). - `yaml_variables`: `true` (default), or `false`. When `true`, variables defined in the trigger job are passed to downstream pipelines. -- `pipeline_variables`: `true` or `false` (default). When `true`, [manual pipeline variables](../variables/index.md#override-a-defined-cicd-variable) +- `pipeline_variables`: `true` or `false` (default). When `true`, [manual pipeline variables](../variables/index.md#override-a-defined-cicd-variable) and [scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule) are passed to downstream pipelines. **Example of `trigger:forward`**: diff --git a/doc/cloud_seed/index.md b/doc/cloud_seed/index.md new file mode 100644 index 00000000000..d4b8d040758 --- /dev/null +++ b/doc/cloud_seed/index.md @@ -0,0 +1,120 @@ +--- +stage: Release +group: Incubation +info: Cloud Seed (formerly 5mp) is a GitLab Incubation Engineering program. No technical writer assigned to this group. +--- + +# Cloud Seed + +Cloud Seed is an open-source program led +by [GitLab Incubation Engineering](https://about.gitlab.com/handbook/engineering/incubation/) in collaboration with +[Google Cloud](https://cloud.google.com/). + +Cloud Seed is in `private-testing` mode and is available to a select group of users. If you are interested in joining +this group, please fill in +the [Cloud Seed Trusted Testers invitation form](https://docs.google.com/forms/d/e/1FAIpQLSeJPtFE8Vpqs_YTAKkFK42p5mO9zIYA2jr_PiP2h32cs8R39Q/viewform) +and we will reach out to you. + +## Purpose + +We believe that it should be **trivial** to deploy web applications (and other workloads) from GitLab to major cloud +providers. + +To support this effort, Cloud Seed makes it simple and intuitive to consume appropriate Google Cloud services +within GitLab. + +## Why Google Cloud + +*or Why not AWS or Azure?* + +Cloud Seed is an open-source program that can be extended by anyone, and we'd love to work with every major cloud +provider. We chose to work with Google Cloud because their team is accessible, supportive, and collaborative in +this effort. + +As an open-source project, [everyone can contribute](#contribute-to-cloud-seed) and shape our direction. + +## Deploy to Google Cloud Run + +After you have your web application in a GitLab project, follow these steps +to deploy your application from GitLab to Google Cloud with Cloud Seed: + +1. [Set up deployment credentials](#set-up-deployment-credentials) +1. (Optional) [Configure your preferred GCP region](#configure-your-preferred-gcp-region) +1. [Configure the Cloud Run deployment pipeline](#configure-the-cloud-run-deployment-pipeline) + +### Set up deployment credentials + +Cloud Seed provides an interface to create Google Cloud Platform (GCP) service accounts from your GitLab project. The associated GCP project +must be selected during the service account creation workflow. This process generates a service account, keys, and deployment permissions. + +To create a service account: + +1. Go to the `Project :: Infrastructure :: Google Cloud` page. +1. Select **Create Service Account**. +1. Follow the Google OAuth 2 workflow and authorize GitLab. +1. Select your GCP project. +1. Associate a Git reference (such as a branch or tag) for the selected GCP project. +1. Submit the form to create the service account. + +The generated service account, service account key, and associated GCP project ID are stored in GitLab as project CI +variables. You can review and manage these in the `Project :: Settings :: CI` page. + +The generated service account has the following roles: + +- `roles/iam.serviceAccountUser` +- `roles/artifactregistry.admin` +- `roles/cloudbuild.builds.builder` +- `roles/run.admin` +- `roles/storage.admin` +- `roles/cloudsql.admin` +- `roles/browser` + +You can enhance security by storing CI variables in secret managers. Learn more about [secret management with GitLab](../ci/secrets/index.md). + +### Configure your preferred GCP region + +When you configure GCP regions for your deployments, the list of regions offered is a subset of +all GCP regions available. + +To configure a region: + +1. Go to the `Project :: Infrastructure :: Google Cloud` page. +1. Select **Configure GCP Region**. +1. Select your preferred GCP region. +1. Associate a Git reference (such as a branch or tag) for the selected GCP region. +1. Submit the form to configure the GCP region. + +The configured GCP region is stored in GitLab as a project CI variable. You can review and manage these in +the `Project :: Settings :: CI` page. + +### Configure the Cloud Run deployment pipeline + +You can configure the Google Cloud Run deployment job in your pipeline. A typical use case for such +a pipeline is continuous deployment of your web application. + +The project pipeline itself could have a broader purpose spanning across several stages, such as build, test, and secure. +Therefore, the Cloud Run deployment offering comes packaged as one job that fits into a much larger pipeline. + +To configure the Cloud Run deployment pipeline: + +1. Go to the `Project :: Infrastructure :: Google Cloud` page. +1. Go to the `Deployments` tab. +1. For `Cloud Run`, select **Configure via Merge Request**. +1. Review the changes and submit to create a merge request. + +This creates a new branch with the Cloud Run deployment pipeline (or injected into an existing pipeline) +and creates an associated merge request where the changes and deployment pipeline execution can be reviewed and merged +into the main branch. + +## Contribute to Cloud Seed + +There are several ways you can contribute to Cloud Seed: + +- [Become a Cloud Seed user](https://docs.google.com/forms/d/e/1FAIpQLSeJPtFE8Vpqs_YTAKkFK42p5mO9zIYA2jr_PiP2h32cs8R39Q/viewform) + in GitLab + and [share feedback](https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/feedback/-/issues/new?template=general_feedback). +- If you are familiar with Ruby on Rails or Vue.js, + consider [contributing to GitLab](../development/contributing/index.md) as a developer. + - Much of Cloud Seed is an internal module within the GitLab code base. +- If you are familiar with GitLab pipelines, consider contributing to + the [Cloud Seed Library](https://gitlab.com/gitlab-org/incubation-engineering/five-minute-production/library) project. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 417ccba26a0..4f27e811b11 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -827,35 +827,54 @@ A description of a field or argument is given using the `description:` keyword. For example: ```ruby -field :id, GraphQL::Types::ID, description: 'ID of the resource.' +field :id, GraphQL::Types::ID, description: 'ID of the issue.' +field :confidential, GraphQL::Types::Boolean, description: 'Indicates the issue is confidential.' +field :closed_at, Types::TimeType, description: 'Timestamp of when the issue was closed.' ``` -Descriptions of fields and arguments are viewable to users through: +You can view descriptions of fields and arguments in: - The [GraphiQL explorer](#graphiql). - The [static GraphQL API reference](../api/graphql/reference/index.md). ### Description style guide -To ensure consistency, the following should be followed whenever adding or updating -descriptions: +#### Language and punctuation -- Mention the name of the resource in the description. Example: - `'Labels of the issue'` (issue being the resource). -- Use `"{x} of the {y}"` where possible. Example: `'Title of the issue'`. - Do not start descriptions with `The` or `A`, for consistency and conciseness. -- Descriptions of `GraphQL::Types::Boolean` fields should answer the question: "What does - this field do?". Example: `'Indicates project has a Git repository'`. -- Always include the word `"timestamp"` when describing an argument or - field of type `Types::TimeType`. This lets the reader know that the - format of the property is `Time`, rather than just `Date`. -- Must end with a period (`.`). +Use `{x} of the {y}` where possible, where `{x}` is the item you're describing, +and `{y}` is the resource it applies to. For example: -Example: +```plaintext +ID of the issue. +``` + +Do not start descriptions with `The` or `A`, for consistency and conciseness. + +End all descriptions with a period (`.`). + +#### Booleans + +For a boolean field (`GraphQL::Types::Boolean`), start with a verb that describes +what it does. For example: + +```plaintext +Indicates the issue is confidential. +``` + +If necessary, provide the default. For example: + +```plaintext +Sets the issue to confidential. Default is false. +``` + +#### `Types::TimeType` field description + +For `Types::TimeType` GraphQL fields, include the word `timestamp`. This lets +the reader know that the format of the property is `Time`, rather than just `Date`. + +For example: ```ruby -field :id, GraphQL::Types::ID, description: 'ID of the issue.' -field :confidential, GraphQL::Types::Boolean, description: 'Indicates the issue is confidential.' field :closed_at, Types::TimeType, description: 'Timestamp of when the issue was closed.' ``` @@ -1782,8 +1801,8 @@ def ready?(**args) end ``` -In the future this may be able to be done using `InputUnions` if -[this RFC](https://github.com/graphql/graphql-spec/blob/master/rfcs/InputUnion.md) +In the future this may be able to be done using `OneOf Input Objects` if +[this RFC](https://github.com/graphql/graphql-spec/pull/825) is merged. ## GitLab custom scalars diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index 15d21883bb8..c4146b5af3e 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -19,7 +19,7 @@ and communicate those limits. There is a guide about [introducing application limits](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits). -## Development +## Implement plan limits ### Insert database plan limits @@ -161,3 +161,31 @@ GitLab.com: - `opensource`: Namespaces and projects that are member of GitLab Open Source program. The `test` environment doesn't have any plans. + +## Implement rate limits using `Rack::Attack` + +We use the [`Rack::Attack`](https://github.com/rack/rack-attack) middleware to throttle Rack requests. +This applies to Rails controllers, Grape endpoints, and any other Rack requests. + +The process for adding a new throttle is loosely: + +1. Add new columns to the `ApplicationSetting` model (`*_enabled`, `*_requests_per_period`, `*_period_in_seconds`). +1. Extend `Gitlab::RackAttack` and `Gitlab::RackAttack::Request` to configure the new rate limit, + and apply it to the desired requests. +1. Add the new settings to the Admin Area form in `app/views/admin/application_settings/_ip_limits.html.haml`. +1. Document the new settings in [User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) and [Application settings API](../api/settings.md). +1. Configure the rate limit for GitLab.com and document it in [GitLab.com-specific rate limits](../user/gitlab_com/index.md#gitlabcom-specific-rate-limits). + +Refer to these past issues for implementation details: + +- [Create a separate rate limit for the Files API](https://gitlab.com/gitlab-org/gitlab/-/issues/335075). +- [Create a separate rate limit for unauthenticated API traffic](https://gitlab.com/gitlab-org/gitlab/-/issues/335300). + +## Implement rate limits using `Gitlab::ApplicationRateLimiter` + +This module implements a custom rate limiter that can be used to throttle +certain actions. Unlike `Rack::Attack` and `Rack::Throttle`, which operate at +the middleware level, this can be used at the controller or API level. + +See the `CheckRateLimit` concern for use in controllers. In other parts of the code +the `Gitlab::ApplicationRateLimiter` module can be called directly. diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index adb656761c5..a202bc419e1 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -111,7 +111,7 @@ After that, add the following information: metrics. For example: `["email_type"]`. If the significant labels for the SLI include `feature_category`, the metrics will also feed into the - [error budgets for stage groups](../stage_group_dashboards.md#error-budget). + [error budgets for stage groups](../stage_group_observability/index.md#error-budget). - `featureCategory`: if the SLI applies to a single feature category, you can specify it statically through this field to feed the SLI into the error budgets for stage groups. diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index b31c7d8756b..373589aaefc 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 NOTE: -This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_dashboards.md#error-budget) +This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_observability/index.md#error-budget) by default. You can [opt in](#error-budget-attribution-and-ownership). The request Apdex SLI (Service Level Indicator) is [an SLI defined in the application](index.md). @@ -221,7 +221,7 @@ end This SLI is used for service level monitoring. It feeds into the [error budget for stage -groups](../stage_group_dashboards.md#error-budget). For this +groups](../stage_group_observability/index.md#error-budget). For this particular SLI, we have opted everyone out by default to give time to set the correct urgencies on endpoints before it affects a group's error budget. diff --git a/doc/development/audit_event_guide/index.md b/doc/development/audit_event_guide/index.md index ae2f9748178..34f78174e5b 100644 --- a/doc/development/audit_event_guide/index.md +++ b/doc/development/audit_event_guide/index.md @@ -18,13 +18,14 @@ actions performed across the application. To instrument an audit event, the following attributes should be provided: -| Attribute | Type | Required? | Description | -|:-------------|:---------------------|:----------|:----------------------------------------------------| -| `name` | String | false | Action name to be audited. Used for error tracking | -| `author` | User | true | User who authors the change | -| `scope` | User, Project, Group | true | Scope which the audit event belongs to | -| `target` | Object | true | Target object being audited | -| `message` | String | true | Message describing the action | +| Attribute | Type | Required? | Description | +|:-------------|:---------------------|:----------|:-----------------------------------------------------------------| +| `name` | String | false | Action name to be audited. Used for error tracking | +| `author` | User | true | User who authors the change | +| `scope` | User, Project, Group | true | Scope which the audit event belongs to | +| `target` | Object | true | Target object being audited | +| `message` | String | true | Message describing the action | +| `created_at` | DateTime | false | The time when the action occured. Defaults to `DateTime.current` | ## How to instrument new Audit Events @@ -97,13 +98,21 @@ if merge_approval_rule.save author: current_user, scope: project_alpha, target: merge_approval_rule, - message: 'Created a new approval rule' + message: 'Created a new approval rule', + created_at: DateTime.current # Useful for pre-dating an audit event when created asynchronously. } ::Gitlab::Audit::Auditor.audit(audit_context) end ``` +### Data volume considerations + +Because every audit event is persisted to the database, consider the amount of data we expect to generate, and the rate of generation, for new +audit events. For new audit events that will produce a lot of data in the database, consider adding a +[streaming-only audit event](#event-streaming) instead. If you have questions about this, feel free to ping +`@gitlab-org/manage/compliance/backend` in an issue or merge request. + ## Audit Event instrumentation flows The two ways we can instrument audit events have different flows. @@ -185,5 +194,8 @@ All events where the entity is a `Group` or `Project` are recorded in the audit - `Group`, events are streamed to the group's root ancestor's event streaming destinations. - `Project`, events are streamed to the project's root ancestor's event streaming destinations. +You can add streaming-only events that are not stored in the GitLab database. This is primarily intended to be used for actions that generate +a large amount of data. See [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76719/diffs#d56e47632f0384722d411ed3ab5b15e947bd2265_26_36) +for an example. This feature is under heavy development. Follow the [parent epic](https://gitlab.com/groups/gitlab-org/-/epics/5925) for updates on feature development. diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md index 1de96df327c..d4c225b62c5 100644 --- a/doc/development/avoiding_downtime_in_migrations.md +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -1,491 +1,11 @@ --- -stage: Enablement -group: Database -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'database/avoiding_downtime_in_migrations.md' +remove_date: '2022-07-08' --- -# Avoiding downtime in migrations +This document was moved to [another location](database/avoiding_downtime_in_migrations.md). -When working with a database certain operations may require downtime. Since we -cannot have downtime in migrations we need to use a set of steps to get the -same end result without downtime. This guide describes various operations that -may appear to need downtime, their impact, and how to perform them without -requiring downtime. - -## Dropping Columns - -Removing columns is tricky because running GitLab processes may still be using -the columns. To work around this safely, you will need three steps in three releases: - -1. Ignoring the column (release M) -1. Dropping the column (release M+1) -1. Removing the ignore rule (release M+2) - -The reason we spread this out across three releases is that dropping a column is -a destructive operation that can't be rolled back easily. - -Following this procedure helps us to make sure there are no deployments to GitLab.com -and upgrade processes for self-managed installations that lump together any of these steps. - -### Step 1: Ignoring the column (release M) - -The first step is to ignore the column in the application code. This is -necessary because Rails caches the columns and re-uses this cache in various -places. This can be done by defining the columns to ignore. For example, to ignore -`updated_at` in the User model you'd use the following: - -```ruby -class User < ApplicationRecord - include IgnorableColumns - ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' -end -``` - -Multiple columns can be ignored, too: - -```ruby -ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' -``` - -If the model exists in CE and EE, the column has to be ignored in the CE model. If the -model only exists in EE, then it has to be added there. - -We require indication of when it is safe to remove the column ignore with: - -- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the - column ignore. -- `remove_after`: set to a date after which we consider it safe to remove the column - ignore, typically after the M+1 release date, during the M+2 development cycle. - -This information allows us to reason better about column ignores and makes sure we -don't remove column ignores too early for both regular releases and deployments to GitLab.com. For -example, this avoids a situation where we deploy a bulk of changes that include both changes -to ignore the column and subsequently remove the column ignore (which would result in a downtime). - -In this example, the change to ignore the column went into release 12.5. - -### Step 2: Dropping the column (release M+1) - -Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: - -```ruby - remove_column :user, :updated_at -``` - -### Step 3: Removing the ignore rule (release M+2) - -With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. -This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. - -This should only get merged with the release indicated with `remove_with` and once -the `remove_after` date has passed. - -## Renaming Columns - -Renaming columns the normal way requires downtime as an application may continue -using the old column name during/after a database migration. To rename a column -without requiring downtime we need two migrations: a regular migration, and a -post-deployment migration. Both these migration can go in the same release. - -### Step 1: Add The Regular Migration - -First we need to create the regular migration. This migration should use -`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the -renaming. For example - -```ruby -# A regular migration in db/migrate -class RenameUsersUpdatedAtToUpdatedAtTimestamp < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - def up - rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end - - def down - undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp - end -end -``` - -This will take care of renaming the column, ensuring data stays in sync, and -copying over indexes and foreign keys. - -If a column contains one or more indexes that don't contain the name of the -original column, the previously described procedure will fail. In that case, -you'll first need to rename these indexes. - -### Step 2: Add A Post-Deployment Migration - -The renaming procedure requires some cleaning up in a post-deployment migration. -We can perform this cleanup using -`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: - -```ruby -# A post-deployment migration in db/post_migrate -class CleanupUsersUpdatedAtRename < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - def up - cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end - - def down - undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp - end -end -``` - -If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. -With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. - -## Changing Column Constraints - -Adding or removing a `NOT NULL` clause (or another constraint) can typically be -done without requiring downtime. However, this does require that any application -changes are deployed _first_. Thus, changing the constraints of a column should -happen in a post-deployment migration. - -Avoid using `change_column` as it produces an inefficient query because it re-defines -the whole column type. - -You can check the following guides for each specific use case: - -- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints) -- [Adding `NOT NULL` constraints](database/not_null_constraints.md) -- [Adding limits to text columns](database/strings_and_the_text_data_type.md) - -## Changing Column Types - -Changing the type of a column can be done using -`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This -method works similarly to `rename_column_concurrently`. For example, let's say -we want to change the type of `users.username` from `string` to `text`. - -### Step 1: Create A Regular Migration - -A regular migration is used to create a new column with a temporary name along -with setting up some triggers to keep data in sync. Such a migration would look -as follows: - -```ruby -# A regular migration in db/migrate -class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - def up - change_column_type_concurrently :users, :username, :text - end - - def down - undo_change_column_type_concurrently :users, :username - end -end -``` - -### Step 2: Create A Post Deployment Migration - -Next we need to clean up our changes using a post-deployment migration: - -```ruby -# A post-deployment migration in db/post_migrate -class ChangeUsersUsernameStringToTextCleanup < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - def up - cleanup_concurrent_column_type_change :users, :username - end - - def down - undo_cleanup_concurrent_column_type_change :users, :username, :string - end -end -``` - -And that's it, we're done! - -### Casting data to a new type - -Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. -In this case, use the `type_cast_function` option. -Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles -casting errors. - -Example migration: - -```ruby - def up - change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' - end -``` - -## Changing The Schema For Large Tables - -While `change_column_type_concurrently` and `rename_column_concurrently` can be -used for changing the schema of a table without downtime, it doesn't work very -well for large tables. Because all of the work happens in sequence the migration -can take a very long time to complete, preventing a deployment from proceeding. -They can also produce a lot of pressure on the database due to it rapidly -updating many rows in sequence. - -To reduce database pressure you should instead use a background migration -when migrating a column in a large table (for example, `issues`). This will -spread the work / load over a longer time period, without slowing down deployments. - -For more information, see [the documentation on cleaning up background -migrations](background_migrations.md#cleaning-up). - -## Adding Indexes - -Adding indexes does not require downtime when `add_concurrent_index` -is used. - -See also [Migration Style Guide](migration_style_guide.md#adding-indexes) -for more information. - -## Dropping Indexes - -Dropping an index does not require downtime. - -## Adding Tables - -This operation is safe as there's no code using the table just yet. - -## Dropping Tables - -Dropping tables can be done safely using a post-deployment migration, but only -if the application no longer uses the table. - -## Renaming Tables - -Renaming tables requires downtime as an application may continue -using the old table name during/after a database migration. - -If the table and the ActiveRecord model is not in use yet, removing the old -table and creating a new one is the preferred way to "rename" the table. - -Renaming a table is possible without downtime by following our multi-release -[rename table process](database/rename_database_tables.md#rename-table-without-downtime). - -## Adding Foreign Keys - -Adding foreign keys usually works in 3 steps: - -1. Start a transaction -1. Run `ALTER TABLE` to add the constraint(s) -1. Check all existing data - -Because `ALTER TABLE` typically acquires an exclusive lock until the end of a -transaction this means this approach would require downtime. - -GitLab allows you to work around this by using -`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method -ensures that no downtime is needed. - -## Removing Foreign Keys - -This operation does not require downtime. - -## Migrating `integer` primary keys to `bigint` - -To [prevent the overflow risk](https://gitlab.com/groups/gitlab-org/-/epics/4785) for some tables -with `integer` primary key (PK), we have to migrate their PK to `bigint`. The process to do this -without downtime and causing too much load on the database is described below. - -### Initialize the conversion and start migrating existing data (release N) - -To start the process, add a regular migration to create the new `bigint` columns. Use the provided -`initialize_conversion_of_integer_to_bigint` helper. The helper also creates a database trigger -to keep in sync both columns for any new records ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072312_initialize_conversion_of_ci_stages_to_bigint.rb)): - -```ruby -class InitializeConversionOfCiStagesToBigint < ActiveRecord::Migration[6.1] - include Gitlab::Database::MigrationHelpers - - TABLE = :ci_stages - COLUMNS = %i(id) - - def up - initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) - end - - def down - revert_initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) - end -end -``` - -Ignore the new `bigint` columns: - -```ruby -module Ci - class Stage < Ci::ApplicationRecord - include IgnorableColumns - ignore_column :id_convert_to_bigint, remove_with: '14.2', remove_after: '2021-08-22' - end -``` - -To migrate existing data, we introduced new type of _batched background migrations_. -Unlike the classic background migrations, built on top of Sidekiq, batched background migrations -don't have to enqueue and schedule all the background jobs at the beginning. -They also have other advantages, like automatic tuning of the batch size, better progress visibility, -and collecting metrics. To start the process, use the provided `backfill_conversion_of_integer_to_bigint` -helper ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072346_backfill_ci_stages_for_bigint_conversion.rb)): - -```ruby -class BackfillCiStagesForBigintConversion < ActiveRecord::Migration[6.1] - include Gitlab::Database::MigrationHelpers - - TABLE = :ci_stages - COLUMNS = %i(id) - - def up - backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) - end - - def down - revert_backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) - end -end -``` - -### Monitor the background migration - -Check how the migration is performing while it's running. Multiple ways to do this are described below. - -#### High-level status of batched background migrations - -See how to [check the status of batched background migrations](../update/index.md#checking-for-background-migrations-before-upgrading). - -#### Query the database - -We can query the related database tables directly. Requires access to read-only replica. -Example queries: - -```sql --- Get details for batched background migration for given table -SELECT * FROM batched_background_migrations WHERE table_name = 'namespaces'\gx - --- Get count of batched background migration jobs by status for given table -SELECT - batched_background_migrations.id, batched_background_migration_jobs.status, COUNT(*) -FROM - batched_background_migrations - JOIN batched_background_migration_jobs ON batched_background_migrations.id = batched_background_migration_jobs.batched_background_migration_id -WHERE - table_name = 'namespaces' -GROUP BY - batched_background_migrations.id, batched_background_migration_jobs.status; - --- Batched background migration progress for given table (based on estimated total number of tuples) -SELECT - m.table_name, - LEAST(100 * sum(j.batch_size) / pg_class.reltuples, 100) AS percentage_complete -FROM - batched_background_migrations m - JOIN batched_background_migration_jobs j ON j.batched_background_migration_id = m.id - JOIN pg_class ON pg_class.relname = m.table_name -WHERE - j.status = 3 AND m.table_name = 'namespaces' -GROUP BY m.id, pg_class.reltuples; -``` - -#### Sidekiq logs - -We can also use the Sidekiq logs to monitor the worker that executes the batched background -migrations: - -1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. -1. Change the index pattern to `pubsub-sidekiq-inf-gprd*`. -1. Add filter for `json.queue: cronjob:database_batched_background_migration`. - -#### PostgreSQL slow queries log - -Slow queries log keeps track of low queries that took above 1 second to execute. To see them -for batched background migration: - -1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. -1. Change the index pattern to `pubsub-postgres-inf-gprd*`. -1. Add filter for `json.endpoint_id.keyword: Database::BatchedBackgroundMigrationWorker`. -1. Optional. To see only updates, add a filter for `json.command_tag.keyword: UPDATE`. -1. Optional. To see only failed statements, add a filter for `json.error_severity.keyword: ERROR`. -1. Optional. Add a filter by table name. - -#### Grafana dashboards - -To monitor the health of the database, use these additional metrics: - -- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that autovacuum cannot keep up. -- [PostgreSQL Overview](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1): if you see high system usage or transactions per second (TPS) on the primary database server, it might mean that the migration is causing problems. - -### Prometheus metrics - -Number of [metrics](https://gitlab.com/gitlab-org/gitlab/-/blob/294a92484ce4611f660439aa48eee4dfec2230b5/lib/gitlab/database/background_migration/batched_migration_wrapper.rb#L90-128) -for each batched background migration are published to Prometheus. These metrics can be searched for and -visualized in Thanos ([see an example](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20(rate(batched_migration_job_updated_tuples_total%7Benv%3D%22gprd%22%7D%5B5m%5D))%20by%20(migration_id)%20&g0.tab=0&g0.stacked=0&g0.range_input=3d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-06-13%2012%3A18%3A24&g0.moment_input=2021-06-13%2012%3A18%3A24)). - -### Swap the columns (release N + 1) - -After the background is completed and the new `bigint` columns are populated for all records, we can -swap the columns. Swapping is done with post-deployment migration. The exact process depends on the -table being converted, but in general it's done in the following steps: - -1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched -migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). -If the migration has not completed, the subsequent steps fail anyway. By checking in advance we -aim to have more helpful error message. -1. Create indexes using the `bigint` columns that match the existing indexes using the `integer` -column ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L28-34)). -1. Create foreign keys (FK) using the `bigint` columns that match the existing FKs using the -`integer` column. Do this both for FK referencing other tables, and FKs that reference the table -that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). -1. Inside a transaction, swap the columns: - 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). - 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) - 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). - 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). - 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). - 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). - 1. Remove old FKs (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). - -See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). - -### Remove the trigger and old `integer` columns (release N + 2) - -Using post-deployment migration and the provided `cleanup_conversion_of_integer_to_bigint` helper, -drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69714)). - -### Remove ignore rules (release N + 3) - -In the next release after the columns were dropped, remove the ignore rules as we do not need them -anymore ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71161)). - -## Data migrations - -Data migrations can be tricky. The usual approach to migrate data is to take a 3 -step approach: - -1. Migrate the initial batch of data -1. Deploy the application code -1. Migrate any remaining data - -Usually this works, but not always. For example, if a field's format is to be -changed from JSON to something else we have a bit of a problem. If we were to -change existing data before deploying application code we'll most likely run -into errors. On the other hand, if we were to migrate after deploying the -application code we could run into the same problems. - -If you merely need to correct some invalid data, then a post-deployment -migration is usually enough. If you need to change the format of data (for example, from -JSON to something else) it's typically best to add a new column for the new data -format, and have the application use that. In such a case the procedure would -be: - -1. Add a new column in the new format -1. Copy over existing data to this new column -1. Deploy the application code -1. In a post-deployment migration, copy over any remaining data - -In general there is no one-size-fits-all solution, therefore it's best to -discuss these kind of migrations in a merge request to make sure they are -implemented in the best way possible. +<!-- This redirect file can be deleted after <2022-07-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/backend/create_source_code_be/index.md b/doc/development/backend/create_source_code_be/index.md index 6421ca3754a..8661d8b4d74 100644 --- a/doc/development/backend/create_source_code_be/index.md +++ b/doc/development/backend/create_source_code_be/index.md @@ -21,14 +21,12 @@ The team works across three codebases: Workhorse, GitLab Shell and GitLab Rails. ## Workhorse -GitLab Workhorse is a smart reverse proxy for GitLab. It handles "large" HTTP +[GitLab Workhorse](../../workhorse/index.md) is a smart reverse proxy for GitLab. It handles "large" HTTP requests such as file downloads, file uploads, `git push`, `git pull` and `git` archive downloads. Workhorse itself is not a feature, but there are several features in GitLab that would not work efficiently without Workhorse. -Workhorse documentation is available in the [Workhorse repository](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse). - ## GitLab Shell GitLab Shell handles Git SSH sessions for GitLab and modifies the list of authorized keys. diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 9fffbd25518..3c9c34bccf8 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -1,497 +1,11 @@ --- -type: reference, dev -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +redirect_to: 'database/background_migrations.md' +remove_date: '2022-07-08' --- -# Background migrations +This document was moved to [another location](database/background_migrations.md). -Background migrations should be used to perform data migrations whenever a -migration exceeds [the time limits in our guidelines](migration_style_guide.md#how-long-a-migration-should-take). For example, you can use background -migrations to migrate data that's stored in a single JSON column -to a separate table instead. - -If the database cluster is considered to be in an unhealthy state, background -migrations automatically reschedule themselves for a later point in time. - -## When To Use Background Migrations - -You should use a background migration when you migrate _data_ in tables that have -so many rows that the process would exceed [the time limits in our guidelines](migration_style_guide.md#how-long-a-migration-should-take) if performed using a regular Rails migration. - -- Background migrations should be used when migrating data in [high-traffic tables](migration_style_guide.md#high-traffic-tables). -- Background migrations may also be used when executing numerous single-row queries -for every item on a large dataset. Typically, for single-record patterns, runtime is -largely dependent on the size of the dataset, hence it should be split accordingly -and put into background migrations. -- Background migrations should not be used to perform schema migrations. - -Some examples where background migrations can be useful: - -- Migrating events from one table to multiple separate tables. -- Populating one column based on JSON stored in another column. -- Migrating data that depends on the output of external services (for example, an API). - -NOTE: -If the background migration is part of an important upgrade, make sure it's announced -in the release post. Discuss with your Project Manager if you're not sure the migration falls -into this category. - -## Isolation - -Background migrations must be isolated and can not use application code (for example, -models defined in `app/models`). Since these migrations can take a long time to -run it's possible for new versions to be deployed while they are still running. - -It's also possible for different migrations to be executed at the same time. -This means that different background migrations should not migrate data in a -way that would cause conflicts. - -## Idempotence - -Background migrations are executed in a context of a Sidekiq process. -Usual Sidekiq rules apply, especially the rule that jobs should be small -and idempotent. - -See [Sidekiq best practices guidelines](https://github.com/mperham/sidekiq/wiki/Best-Practices) -for more details. - -Make sure that in case that your migration job is going to be retried data -integrity is guaranteed. - -## Background migrations for EE-only features - -All the background migration classes for EE-only features should be present in GitLab CE. -For this purpose, an empty class can be created for GitLab CE, and it can be extended for GitLab EE -as explained in the [guidelines for implementing Enterprise Edition features](ee_features.md#code-in-libgitlabbackground_migration). - -## How It Works - -Background migrations are simple classes that define a `perform` method. A -Sidekiq worker will then execute such a class, passing any arguments to it. All -migration classes must be defined in the namespace -`Gitlab::BackgroundMigration`, the files should be placed in the directory -`lib/gitlab/background_migration/`. - -## Scheduling - -Scheduling a background migration should be done in a post-deployment -migration that includes `Gitlab::Database::MigrationHelpers` -To do so, simply use the following code while -replacing the class name and arguments with whatever values are necessary for -your migration: - -```ruby -migrate_in('BackgroundMigrationClassName', [arg1, arg2, ...]) -``` - -You can use the function `queue_background_migration_jobs_by_range_at_intervals` -to automatically split the job into batches: - -```ruby -queue_background_migration_jobs_by_range_at_intervals( - ClassName, - BackgroundMigrationClassName, - 2.minutes, - batch_size: 10_000 - ) -``` - -You'll also need to make sure that newly created data is either migrated, or -saved in both the old and new version upon creation. For complex and time -consuming migrations it's best to schedule a background job using an -`after_create` hook so this doesn't affect response timings. The same applies to -updates. Removals in turn can be handled by simply defining foreign keys with -cascading deletes. - -### Rescheduling background migrations - -If one of the background migrations contains a bug that is fixed in a patch -release, the background migration needs to be rescheduled so the migration would -be repeated on systems that already performed the initial migration. - -When you reschedule the background migration, make sure to turn the original -scheduling into a no-op by clearing up the `#up` and `#down` methods of the -migration performing the scheduling. Otherwise the background migration would be -scheduled multiple times on systems that are upgrading multiple patch releases at -once. - -When you start the second post-deployment migration, you should delete any -previously queued jobs from the initial migration with the provided -helper: - -```ruby -delete_queued_jobs('BackgroundMigrationClassName') -``` - -## Cleaning Up - -NOTE: -Cleaning up any remaining background migrations _must_ be done in either a major -or minor release, you _must not_ do this in a patch release. - -Because background migrations can take a long time you can't immediately clean -things up after scheduling them. For example, you can't drop a column that's -used in the migration process as this would cause jobs to fail. This means that -you'll need to add a separate _post deployment_ migration in a future release -that finishes any remaining jobs before cleaning things up (for example, removing a -column). - -As an example, say you want to migrate the data from column `foo` (containing a -big JSON blob) to column `bar` (containing a string). The process for this would -roughly be as follows: - -1. Release A: - 1. Create a migration class that performs the migration for a row with a given ID. - You can use [background jobs tracking](#background-jobs-tracking) to simplify cleaning up. - 1. Deploy the code for this release, this should include some code that will - schedule jobs for newly created data (for example, using an `after_create` hook). - 1. Schedule jobs for all existing rows in a post-deployment migration. It's - possible some newly created rows may be scheduled twice so your migration - should take care of this. -1. Release B: - 1. Deploy code so that the application starts using the new column and stops - scheduling jobs for newly created data. - 1. In a post-deployment migration, finalize all jobs that have not succeeded by now. - If you used [background jobs tracking](#background-jobs-tracking) in release A, - you can use `finalize_background_migration` from `BackgroundMigrationHelpers` to ensure no jobs remain. - This helper will: - 1. Use `Gitlab::BackgroundMigration.steal` to process any remaining - jobs in Sidekiq. - 1. Reschedule the migration to be run directly (that is, not through Sidekiq) - on any rows that weren't migrated by Sidekiq. This can happen if, for - instance, Sidekiq received a SIGKILL, or if a particular batch failed - enough times to be marked as dead. - 1. Remove `Gitlab::Database::BackgroundMigrationJob` rows where - `status = succeeded`. To retain diagnostic information that may - help with future bug tracking you can skip this step by specifying - the `delete_tracking_jobs: false` parameter. - 1. Remove the old column. - -This may also require a bump to the [import/export version](../user/project/settings/import_export.md), if -importing a project from a prior version of GitLab requires the data to be in -the new format. - -## Example - -To explain all this, let's use the following example: the table `integrations` has a -field called `properties` which is stored in JSON. For all rows you want to -extract the `url` key from this JSON object and store it in the `integrations.url` -column. There are millions of integrations and parsing JSON is slow, thus you can't -do this in a regular migration. - -To do this using a background migration we'll start with defining our migration -class: - -```ruby -class Gitlab::BackgroundMigration::ExtractIntegrationsUrl - class Integration < ActiveRecord::Base - self.table_name = 'integrations' - end - - def perform(start_id, end_id) - Integration.where(id: start_id..end_id).each do |integration| - json = JSON.load(integration.properties) - - integration.update(url: json['url']) if json['url'] - rescue JSON::ParserError - # If the JSON is invalid we don't want to keep the job around forever, - # instead we'll just leave the "url" field to whatever the default value - # is. - next - end - end -end -``` - -Next we'll need to adjust our code so we schedule the above migration for newly -created and updated integrations. We can do this using something along the lines of -the following: - -```ruby -class Integration < ActiveRecord::Base - after_commit :schedule_integration_migration, on: :update - after_commit :schedule_integration_migration, on: :create - - def schedule_integration_migration - BackgroundMigrationWorker.perform_async('ExtractIntegrationsUrl', [id, id]) - end -end -``` - -We're using `after_commit` here to ensure the Sidekiq job is not scheduled -before the transaction completes as doing so can lead to race conditions where -the changes are not yet visible to the worker. - -Next we'll need a post-deployment migration that schedules the migration for -existing data. - -```ruby -class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - MIGRATION = 'ExtractIntegrationsUrl' - DELAY_INTERVAL = 2.minutes - - def up - queue_background_migration_jobs_by_range_at_intervals( - define_batchable_model('integrations'), - MIGRATION, - DELAY_INTERVAL) - end - - def down - end -end -``` - -Once deployed our application will continue using the data as before but at the -same time will ensure that both existing and new data is migrated. - -In the next release we can remove the `after_commit` hooks and related code. We -will also need to add a post-deployment migration that consumes any remaining -jobs and manually run on any un-migrated rows. Such a migration would look like -this: - -```ruby -class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[1.0] - disable_ddl_transaction! - - def up - # This must be included - Gitlab::BackgroundMigration.steal('ExtractIntegrationsUrl') - - # This should be included, but can be skipped - see below - define_batchable_model('integrations').where(url: nil).each_batch(of: 50) do |batch| - range = batch.pluck('MIN(id)', 'MAX(id)').first - - Gitlab::BackgroundMigration::ExtractIntegrationsUrl.new.perform(*range) - end - end - - def down - end -end -``` - -The final step runs for any un-migrated rows after all of the jobs have been -processed. This is in case a Sidekiq process running the background migrations -received SIGKILL, leading to the jobs being lost. (See -[more reliable Sidekiq queue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36791) for more information.) - -If the application does not depend on the data being 100% migrated (for -instance, the data is advisory, and not mission-critical), then this final step -can be skipped. - -This migration will then process any jobs for the ExtractIntegrationsUrl migration -and continue once all jobs have been processed. Once done you can safely remove -the `integrations.properties` column. - -## Testing - -It is required to write tests for: - -- The background migrations' scheduling migration. -- The background migration itself. -- A cleanup migration. - -The `:migration` and `schema: :latest` RSpec tags are automatically set for -background migration specs. -See the -[Testing Rails migrations](testing_guide/testing_migrations_guide.md#testing-a-non-activerecordmigration-class) -style guide. - -Keep in mind that `before` and `after` RSpec hooks are going -to migrate you database down and up, which can result in other background -migrations being called. That means that using `spy` test doubles with -`have_received` is encouraged, instead of using regular test doubles, because -your expectations defined in a `it` block can conflict with what is being -called in RSpec hooks. See [issue #35351](https://gitlab.com/gitlab-org/gitlab/-/issues/18839) -for more details. - -## Best practices - -1. Make sure to know how much data you're dealing with. -1. Make sure that background migration jobs are idempotent. -1. Make sure that tests you write are not false positives. -1. Make sure that if the data being migrated is critical and cannot be lost, the - clean-up migration also checks the final state of the data before completing. -1. When migrating many columns, make sure it won't generate too many - dead tuples in the process (you may need to directly query the number of dead tuples - and adjust the scheduling according to this piece of data). -1. Make sure to discuss the numbers with a database specialist, the migration may add - more pressure on DB than you expect (measure on staging, - or ask someone to measure on production). -1. Make sure to know how much time it'll take to run all scheduled migrations. -1. Provide an estimation section in the description, estimating both the total migration - run time and the query times for each background migration job. Explain plans for each query - should also be provided. - - For example, assuming a migration that deletes data, include information similar to - the following section: - - ```plaintext - Background Migration Details: - - 47600 items to delete - batch size = 1000 - 47600 / 1000 = 48 batches - - Estimated times per batch: - - 820ms for select statement with 1000 items (see linked explain plan) - - 900ms for delete statement with 1000 items (see linked explain plan) - Total: ~2 sec per batch - - 2 mins delay per batch (safe for the given total time per batch) - - 48 batches * 2 min per batch = 96 mins to run all the scheduled jobs - ``` - - The execution time per batch (2 sec in this example) is not included in the calculation - for total migration time. The jobs are scheduled 2 minutes apart without knowledge of - the execution time. - -## Additional tips and strategies - -### Nested batching - -A strategy to make the migration run faster is to schedule larger batches, and then use `EachBatch` -within the background migration to perform multiple statements. - -The background migration helpers that queue multiple jobs such as -`queue_background_migration_jobs_by_range_at_intervals` use [`EachBatch`](iterating_tables_in_batches.md). -The example above has batches of 1000, where each queued job takes two seconds. If the query has been optimized -to make the time for the delete statement within the [query performance guidelines](query_performance.md), -1000 may be the largest number of records that can be deleted in a reasonable amount of time. - -The minimum and most common interval for delaying jobs is two minutes. This results in two seconds -of work for each two minute job. There's nothing that prevents you from executing multiple delete -statements in each background migration job. - -Looking at the example above, you could alternatively do: - -```plaintext -Background Migration Details: - -47600 items to delete -batch size = 10_000 -47600 / 10_000 = 5 batches - -Estimated times per batch: -- Records are updated in sub-batches of 1000 => 10_000 / 1000 = 10 total updates -- 820ms for select statement with 1000 items (see linked explain plan) -- 900ms for delete statement with 1000 items (see linked explain plan) -Sub-batch total: ~2 sec per sub-batch, -Total batch time: 2 * 10 = 20 sec per batch - -2 mins delay per batch - -5 batches * 2 min per batch = 10 mins to run all the scheduled jobs -``` - -The batch time of 20 seconds still fits comfortably within the two minute delay, yet the total run -time is cut by a tenth from around 100 minutes to 10 minutes! When dealing with large background -migrations, this can cut the total migration time by days. - -When batching in this way, it is important to look at query times on the higher end -of the table or relation being updated. `EachBatch` may generate some queries that become much -slower when dealing with higher ID ranges. - -### Delay time - -When looking at the batch execution time versus the delay time, the execution time -should fit comfortably within the delay time for a few reasons: - -- To allow for a variance in query times. -- To allow autovacuum to catch up after periods of high churn. - -Never try to optimize by fully filling the delay window even if you are confident -the queries themselves have no timing variance. - -### Background jobs tracking - -`queue_background_migration_jobs_by_range_at_intervals` can create records for each job that is scheduled to run. -You can enable this behavior by passing `track_jobs: true`. Each record starts with a `pending` status. Make sure that your worker updates the job status to `succeeded` by calling `Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded` in the `perform` method of your background migration. - -```ruby -# Background migration code - -def perform(start_id, end_id) - # do work here - - mark_job_as_succeeded(start_id, end_id) -end - -private - -# Make sure that the arguments passed here match those passed to the background -# migration -def mark_job_as_succeeded(*arguments) - Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded( - self.class.name.demodulize, - arguments - ) -end -``` - -```ruby -# Post deployment migration -MIGRATION = 'YourBackgroundMigrationName' -DELAY_INTERVAL = 2.minutes.to_i # can be different -BATCH_SIZE = 10_000 # can be different - -disable_ddl_transaction! - -def up - queue_background_migration_jobs_by_range_at_intervals( - define_batchable_model('name_of_the_table_backing_the_model'), - MIGRATION, - DELAY_INTERVAL, - batch_size: BATCH_SIZE, - track_jobs: true - ) -end - -def down - # no-op -end -``` - -See [`lib/gitlab/background_migration/drop_invalid_vulnerabilities.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/background_migration/drop_invalid_vulnerabilities.rb) for a full example. - -#### Rescheduling pending jobs - -You can reschedule pending migrations from the `background_migration_jobs` table by creating a post-deployment migration and calling `requeue_background_migration_jobs_by_range_at_intervals` with the migration name and delay interval. - -```ruby -# Post deployment migration -MIGRATION = 'YourBackgroundMigrationName' -DELAY_INTERVAL = 2.minutes - -disable_ddl_transaction! - -def up - requeue_background_migration_jobs_by_range_at_intervals(MIGRATION, DELAY_INTERVAL) -end - -def down - # no-op -end -``` - -See [`db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb) for a full example. - -### Viewing failure error logs - -After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). -View the production Sidekiq log and filter for: - -- `json.class: BackgroundMigrationWorker` -- `json.job_status: fail` -- `json.meta.caller_id: <MyBackgroundMigrationSchedulingMigrationClassName>` -- `json.args: <MyBackgroundMigrationClassName>` - -Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. - -Depending on when and how the failure occurred, you may find other helpful information by filtering with `json.class: <MyBackgroundMigrationClassName>`. +<!-- This redirect file can be deleted after <2022-07-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/batched_background_migrations.md b/doc/development/batched_background_migrations.md new file mode 100644 index 00000000000..e7703b5dd2b --- /dev/null +++ b/doc/development/batched_background_migrations.md @@ -0,0 +1,319 @@ +--- +type: reference, dev +stage: Enablement +group: Database +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Batched background migrations + +Batched Background Migrations should be used to perform data migrations whenever a +migration exceeds [the time limits](migration_style_guide.md#how-long-a-migration-should-take) +in our guidelines. For example, you can use batched background +migrations to migrate data that's stored in a single JSON column +to a separate table instead. + +## When to use batched background migrations + +Use a batched background migration when you migrate _data_ in tables containing +so many rows that the process would exceed +[the time limits in our guidelines](migration_style_guide.md#how-long-a-migration-should-take) +if performed using a regular Rails migration. + +- Batched background migrations should be used when migrating data in + [high-traffic tables](migration_style_guide.md#high-traffic-tables). +- Batched background migrations may also be used when executing numerous single-row queries + for every item on a large dataset. Typically, for single-record patterns, runtime is + largely dependent on the size of the dataset. Split the dataset accordingly, + and put it into background migrations. +- Don't use batched background migrations to perform schema migrations. + +Background migrations can help when: + +- Migrating events from one table to multiple separate tables. +- Populating one column based on JSON stored in another column. +- Migrating data that depends on the output of external services. (For example, an API.) + +NOTE: +If the batched background migration is part of an important upgrade, it must be announced +in the release post. Discuss with your Project Manager if you're unsure if the migration falls +into this category. + +## Isolation + +Batched background migrations must be isolated and can not use application code. (For example, +models defined in `app/models`.). Because these migrations can take a long time to +run, it's possible for new versions to deploy while the migrations are still running. + +## Idempotence + +Batched background migrations are executed in a context of a Sidekiq process. +The usual Sidekiq rules apply, especially the rule that jobs should be small +and idempotent. Make sure that in case that your migration job is retried, data +integrity is guaranteed. + +See [Sidekiq best practices guidelines](https://github.com/mperham/sidekiq/wiki/Best-Practices) +for more details. + +## Batched background migrations for EE-only features + +All the background migration classes for EE-only features should be present in GitLab CE. +For this purpose, create an empty class for GitLab CE, and extend it for GitLab EE +as explained in the guidelines for +[implementing Enterprise Edition features](ee_features.md#code-in-libgitlabbackground_migration). + +Batched Background migrations are simple classes that define a `perform` method. A +Sidekiq worker then executes such a class, passing any arguments to it. All +migration classes must be defined in the namespace +`Gitlab::BackgroundMigration`. Place the files in the directory +`lib/gitlab/background_migration/`. + +## Queueing + +Queueing a batched background migration should be done in a post-deployment +migration. Use this `queue_batched_background_migration` example, queueing the +migration to be executed in batches. Replace the class name and arguments with the values +from your migration: + +```ruby +queue_batched_background_migration( + JOB_CLASS_NAME, + TABLE_NAME, + JOB_ARGUMENTS, + JOB_INTERVAL + ) +``` + +Make sure the newly-created data is either migrated, or +saved in both the old and new version upon creation. Removals in +turn can be handled by defining foreign keys with cascading deletes. + +### Requeuing batched background migrations + +If one of the batched background migrations contains a bug that is fixed in a patch +release, you must requeue the batched background migration so the migration +repeats on systems that already performed the initial migration. + +When you requeue the batched background migration, turn the original +queuing into a no-op by clearing up the `#up` and `#down` methods of the +migration performing the requeuing. Otherwise, the batched background migration is +queued multiple times on systems that are upgrading multiple patch releases at +once. + +When you start the second post-deployment migration, delete the +previously batched migration with the provided code: + +```ruby +Gitlab::Database::BackgroundMigration::BatchedMigration + .for_configuration(MIGRATION_NAME, TABLE_NAME, COLUMN, JOB_ARGUMENTS) + .delete_all +``` + +## Cleaning up + +NOTE: +Cleaning up any remaining background migrations must be done in either a major +or minor release. You must not do this in a patch release. + +Because background migrations can take a long time, you can't immediately clean +things up after queueing them. For example, you can't drop a column used in the +migration process, as jobs would fail. You must add a separate _post-deployment_ +migration in a future release that finishes any remaining +jobs before cleaning things up. (For example, removing a column.) + +To migrate the data from column `foo` (containing a big JSON blob) to column `bar` +(containing a string), you would: + +1. Release A: + 1. Create a migration class that performs the migration for a row with a given ID. + 1. Update new rows using one of these techniques: + - Create a new trigger for simple copy operations that don't need application logic. + - Handle this operation in the model/service as the records are created or updated. + - Create a new custom background job that updates the records. + 1. Queue the batched background migration for all existing rows in a post-deployment migration. +1. Release B: + 1. Add a post-deployment migration that checks if the batched background migration is completed. + 1. Deploy code so that the application starts using the new column and stops to update new records. + 1. Remove the old column. + +Bump to the [import/export version](../user/project/settings/import_export.md) may +be required, if importing a project from a prior version of GitLab requires the +data to be in the new format. + +## Example + +The table `integrations` has a field called `properties`, stored in JSON. For all rows, +extract the `url` key from this JSON object and store it in the `integrations.url` +column. Millions of integrations exist, and parsing JSON is slow, so you can't +do this work in a regular migration. + +1. Start by defining our migration class: + + ```ruby + class Gitlab::BackgroundMigration::ExtractIntegrationsUrl + class Integration < ActiveRecord::Base + self.table_name = 'integrations' + end + + def perform(start_id, end_id) + Integration.where(id: start_id..end_id).each do |integration| + json = JSON.load(integration.properties) + + integration.update(url: json['url']) if json['url'] + rescue JSON::ParserError + # If the JSON is invalid we don't want to keep the job around forever, + # instead we'll just leave the "url" field to whatever the default value + # is. + next + end + end + end + ``` + + NOTE: + To get a `connection` in the batched background migration,use an inheritance + relation using the following base class `Gitlab::BackgroundMigration::BaseJob`. + For example: `class Gitlab::BackgroundMigration::ExtractIntegrationsUrl < Gitlab::BackgroundMigration::BaseJob` + +1. Add a new trigger to the database to update newly created and updated integrations, + similar to this example: + + ```ruby + execute(<<~SQL) + CREATE OR REPLACE FUNCTION example() RETURNS trigger + LANGUAGE plpgsql + AS $$ + BEGIN + NEW."url" := NEW.properties -> "url" + RETURN NEW; + END; + $$; + SQL + ``` + +1. Create a post-deployment migration that queues the migration for existing data: + + ```ruby + class QueueExtractIntegrationsUrl < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + MIGRATION = 'ExtractIntegrationsUrl' + DELAY_INTERVAL = 2.minutes + + def up + queue_batched_background_migration( + MIGRATION, + :migrations, + :id, + job_interval: DELAY_INTERVAL + ) + end + + def down + Gitlab::Database::BackgroundMigration::BatchedMigration + .for_configuration(MIGRATION, :migrations, :id, []).delete_all + end + end + ``` + + After deployment, our application: + - Continues using the data as before. + - Ensures that both existing and new data are migrated. + +1. In the next release, remove the trigger. We must also add a new post-deployment migration + that checks that the batched background migration is completed. For example: + + ```ruby + class FinalizeExtractIntegrationsUrlJobs < Gitlab::Database::Migration[1.0] + MIGRATION = 'ExtractIntegrationsUrl' + disable_ddl_transaction! + + def up + ensure_batched_background_migration_is_finished( + job_class_name: MIGRATION, + table_name: :integrations, + column_name: :id, + job_arguments: [] + ) + end + + def down + # no-op + end + end + ``` + + If the application does not depend on the data being 100% migrated (for + instance, the data is advisory, and not mission-critical), then you can skip this + final step. This step confirms that the migration is completed, and all of the rows were migrated. + +After the batched migration is completed, you can safely remove the `integrations.properties` column. + +## Testing + +Writing tests is required for: + +- The batched background migrations' queueing migration. +- The batched background migration itself. +- A cleanup migration. + +The `:migration` and `schema: :latest` RSpec tags are automatically set for +background migration specs. Refer to the +[Testing Rails migrations](testing_guide/testing_migrations_guide.md#testing-a-non-activerecordmigration-class) +style guide. + +Remember that `before` and `after` RSpec hooks +migrate your database down and up. These hooks can result in other batched background +migrations being called. Using `spy` test doubles with +`have_received` is encouraged, instead of using regular test doubles, because +your expectations defined in a `it` block can conflict with what is +called in RSpec hooks. Refer to [issue #35351](https://gitlab.com/gitlab-org/gitlab/-/issues/18839) +for more details. + +## Best practices + +1. Know how much data you're dealing with. +1. Make sure the batched background migration jobs are idempotent. +1. Confirm the tests you write are not false positives. +1. If the data being migrated is critical and cannot be lost, the + clean-up migration must also check the final state of the data before completing. +1. Discuss the numbers with a database specialist. The migration may add + more pressure on DB than you expect. Measure on staging, + or ask someone to measure on production. +1. Know how much time is required to run the batched background migration. + +## Additional tips and strategies + +### Viewing failure error logs + +You can view failures in two ways: + +- Via GitLab logs: + 1. After running a batched background migration, if any jobs fail, + view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). + View the production Sidekiq log and filter for: + + - `json.new_state: failed` + - `json.job_class_name: <Batched Background Migration job class name>` + - `json.job_arguments: <Batched Background Migration job class arguments>` + + 1. Review the `json.exception_class` and `json.exception_message` values to help + understand why the jobs failed. + + 1. Remember the retry mechanism. Having a failure does not mean the job failed. + Always check the last status of the job. + +- Via database: + + 1. Get the batched background migration `CLASS_NAME`. + 1. Execute the following query in the PostgreSQL console: + + ```sql + SELECT migration.id, migration.job_class_name, transition_logs.exception_class, transition_logs.exception_message + FROM batched_background_migrations as migration + INNER JOIN batched_background_migration_jobs as jobs + ON jobs.batched_background_migration_id = migration.id + INNER JOIN batched_background_migration_job_transition_logs as transition_logs + ON transition_logs.batched_background_migration_job_id = jobs.id + WHERE transition_logs.next_status = '2' AND migration.job_class_name = "CLASS_NAME"; + ``` diff --git a/doc/development/cached_queries.md b/doc/development/cached_queries.md index 492c8d13600..8c69981b27a 100644 --- a/doc/development/cached_queries.md +++ b/doc/development/cached_queries.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 26fcf520393..e18fcb0061b 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -25,6 +25,7 @@ To request access to ChatOps on GitLab.com: - The same username you use on GitLab.com. You may have to choose a different username later. - Clicking the **Sign in with Google** button to sign in with your GitLab.com email address. + - Clicking the **Sign in with Okta** button to sign in with Okta. 1. Confirm that your username in [Internal GitLab for Operations](https://ops.gitlab.net/) is the same as your username in [GitLab.com](https://gitlab.com/). If the usernames diff --git a/doc/development/cicd/schema.md b/doc/development/cicd/schema.md index b63d951b881..0e456a25a7a 100644 --- a/doc/development/cicd/schema.md +++ b/doc/development/cicd/schema.md @@ -5,26 +5,26 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, howto --- -# Contribute to the CI Schema **(FREE)** +# Contribute to the CI/CD Schema **(FREE)** -The [pipeline editor](../../ci/pipeline_editor/index.md) uses a CI schema to enhance -the authoring experience of our CI configuration files. With the CI schema, the editor can: +The [pipeline editor](../../ci/pipeline_editor/index.md) uses a CI/CD schema to enhance +the authoring experience of our CI/CD configuration files. With the CI/CD schema, the editor can: -- Validate the content of the CI configuration file as it is being written in the editor. +- Validate the content of the CI/CD configuration file as it is being written in the editor. - Provide autocomplete functionality and suggest available keywords. - Provide definitions of keywords through annotations. -As the rules and keywords for configuring our CI configuration files change, so too -should our CI schema. +As the rules and keywords for configuring our CI/CD configuration files change, so too +should our CI/CD schema. This feature is behind the [`schema_linting`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/feature_flags/development/schema_linting.yml) feature flag for self-managed instances, and is enabled for GitLab.com. ## JSON Schemas -The CI schema follows the [JSON Schema Draft-07](https://json-schema.org/draft-07/json-schema-release-notes.html) -specification. Although the CI configuration file is written in YAML, it is converted -into JSON by using `monaco-yaml` before it is validated by the CI schema. +The CI/CD schema follows the [JSON Schema Draft-07](https://json-schema.org/draft-07/json-schema-release-notes.html) +specification. Although the CI/CD configuration file is written in YAML, it is converted +into JSON by using `monaco-yaml` before it is validated by the CI/CD schema. If you're new to JSON schemas, consider checking out [this guide](https://json-schema.org/learn/getting-started-step-by-step) for @@ -32,8 +32,8 @@ a step-by-step introduction on how to work with JSON schemas. ## Update Keywords -The CI schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json). -It contains all the keywords available for authoring CI configuration files. +The CI/CD schema is at [`app/assets/javascripts/editor/schema/ci.json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/editor/schema/ci.json). +It contains all the keywords available for authoring CI/CD configuration files. Check the [keyword reference](../../ci/yaml/index.md) for a comprehensive list of all available keywords. @@ -138,9 +138,72 @@ under the topmost **properties** key. ## Test the schema -For now, the CI schema can only be tested manually. To verify the behavior is correct: +### Verify changes 1. Enable the `schema_linting` feature flag. 1. Go to **CI/CD** > **Editor**. 1. Write your CI/CD configuration in the editor and verify that the schema validates it correctly. + +### Write specs + +All of the CI/CD schema specs are in [`spec/frontend/editor/schema/ci`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/frontend/editor/schema/ci). +Legacy tests are in JSON, but we recommend writing all new tests in YAML. +You can write them as if you're adding a new `.gitlab-ci.yml` configuration file. + +Tests are separated into **positive** tests and **negative** tests. Positive tests +are snippets of CI/CD configuration code that use the schema keywords as intended. +Conversely, negative tests give examples of the schema keywords being used incorrectly. +These tests ensure that the schema validates different examples of input as expected. + +`ci_schema_spec.js` is responsible for running all of the tests against the schema. + +A detailed explanation of how the tests are set up can be found in this +[merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83047). + +#### Update schema specs + +If a YAML test does not exist for the specified keyword, create new files in +`yaml_tests/positive_tests` and `yaml_tests/negative_tests`. Otherwise, you can update +the existing tests: + +1. Write both positive and negative tests to validate different kinds of input. +1. If you created new files, import them in `ci_schema_spec.js` and add each file to their + corresponding object entries. For example: + + ```javascript + import CacheYaml from './yaml_tests/positive_tests/cache.yml'; + import CacheNegativeYaml from './yaml_tests/negative_tests/cache.yml'; + + // import your new test files + import NewKeywordTestYaml from './yaml_tests/positive_tests/cache.yml'; + import NewKeywordTestNegativeYaml from './yaml_tests/negative_tests/cache.yml'; + + describe('positive tests', () => { + it.each( + Object.entries({ + CacheYaml, + NewKeywordTestYaml, // add positive test here + }), + )('schema validates %s', (_, input) => { + expect(input).toValidateJsonSchema(schema); + }); + }); + + describe('negative tests', () => { + it.each( + Object.entries({ + CacheNegativeYaml, + NewKeywordTestYaml, // add negative test here + }), + )('schema validates %s', (_, input) => { + expect(input).not.toValidateJsonSchema(schema); + }); + }); + ``` + +1. Run the command `yarn jest spec/frontend/editor/schema/ci/ci_schema_spec.js` + and verify that all the tests successfully pass. + +If the spec covers a change to an existing keyword and it affects the legacy JSON +tests, update them as well. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index ec913df8e4a..48bbe4c60ba 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -74,17 +74,13 @@ It picks reviewers and maintainers from the list at the page, with these behaviors: 1. It doesn't pick people whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status): - - Contains the string 'OOO', 'PTO', 'Parental Leave', or 'Friends and Family'. + - Contains the string `OOO`, `PTO`, `Parental Leave`, or `Friends and Family`. - GitLab user **Busy** indicator is set to `True`. - - Emoji is any of: - - 🌴 `:palm_tree:` - - 🏖️ `:beach:`, `:beach_umbrella:`, or `:beach_with_umbrella:` - - 🎡 `:ferris_wheel:` - - 🌡️ `:thermometer:` - - 🤒 `:face_with_thermometer:` - - 🔴 `:red_circle:` - - 💡 `:bulb:` - - 🌞 `:sun_with_face:` + - Emoji is from one of these categories: + - **On leave** - 🌴 `:palm_tree:`, 🏖️ `:beach:`, ⛱ `:beach_umbrella:`, 🏖 `:beach_with_umbrella:`, 🌞 `:sun_with_face:`, 🎡 `:ferris_wheel:` + - **Out sick** - 🌡️ `:thermometer:`, 🤒 `:face_with_thermometer:` + - **At capacity** - 🔴 `:red_circle:` + - **Focus mode** - 💡 `:bulb:` (focusing on their team's work) 1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. 1. Team members whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji @@ -92,12 +88,22 @@ page, with these behaviors: - Reviewers with 🔵 `:large_blue_circle:` are two times as likely to be picked as other reviewers. - Trainee maintainers with 🔵 `:large_blue_circle:` are four times as likely to be picked as other reviewers. 1. People whose [GitLab status](../user/profile/index.md#set-your-current-status) emoji - is 🔶 `:large_orange_diamond:` or 🔸 `:small_orange_diamond:` are half as likely to be picked. This applies to both reviewers and trainee maintainers. + is 🔶 `:large_orange_diamond:` or 🔸 `:small_orange_diamond:` are half as likely to be picked. 1. It always picks the same reviewers and maintainers for the same branch name (unless their out-of-office (OOO) status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so that it can be stable for backport branches. +The [Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette) contains: + +- Assignment events in the last 7 and 30 days. +- Currently assigned merge requests per person. +- Sorting by different criteria. +- A manual reviewer roulette. +- Local time information. + +For more information, review [the roulette README](https://gitlab.com/gitlab-org/gitlab-roulette). + ### Approval guidelines As described in the section on the responsibility of the maintainer below, you @@ -136,6 +142,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes Product Intelligence (telemetry or analytics) changes, it should be reviewed and approved by a [Product Intelligence engineer](https://gitlab.com/gitlab-org/growth/product-intelligence/engineers). 1. If your merge request includes an addition of, or changes to a [Feature spec](testing_guide/testing_levels.md#frontend-feature-tests), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa) or [Quality reviewer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_qa)**. 1. If your merge request introduces a new service to GitLab (Puma, Sidekiq, Gitaly are examples), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. See the [process for adding a service component to GitLab](adding_service_component.md) for details. +1. If your merge request includes changes related to authentication or authorization, it must be **approved by a [Manage:Authentication and Authorization team member](https://about.gitlab.com/company/team/)**. Check the [code review section on the group page](https://about.gitlab.com/handbook/engineering/development/dev/manage/authentication-and-authorization/#additional-considerations) for more details. Patterns for files known to require review from the team are listed in the in the `Authentication and Authorization` section of the [`CODEOWNERS`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/CODEOWNERS) file, and the team will be listed in the approvers section of all merge requests that modify these files. - (*1*): Specs other than JavaScript specs are considered backend code. - (*2*): We encourage you to seek guidance from a database maintainer if your merge @@ -154,7 +161,7 @@ Using checklists improves quality in software engineering. This checklist is a s ##### Quality -See the [test engineering process](https://about.gitlab.com/handbook/engineering/quality/test-engineering/) for further quality guidelines. +See the [test engineering process](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/) for further quality guidelines. 1. I have self-reviewed this MR per [code review guidelines](code_review.md). 1. For the code that this change impacts, I believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). @@ -240,6 +247,8 @@ warrant a comment could be: - Any benchmarking performed to complement the change. - Potentially insecure code. +If there are any projects, snippets, or other assets that are required for a reviewer to validate the solution, ensure they have access to those assets before requesting review. + Avoid: - Adding TODO comments (referenced above) directly to the source code unless the reviewer requires @@ -249,7 +258,7 @@ Avoid: [_explain why, not what_](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/). - Requesting maintainer reviews of merge requests with failed tests. If the tests are failing and you have to request a review, ensure you leave a comment with an explanation. - Excessively mentioning maintainers through email or Slack (if the maintainer is reachable -through Slack). If you can't add a reviewer for a merge request, `@` mentioning a maintainer in a comment is acceptable and in all other cases adding a reviewer is sufficient. +through Slack). If you can't add a reviewer for a merge request, it's acceptable to `@` mention a maintainer in a comment. In all other cases, it's sufficient to add a reviewer or [request their attention](../user/project/merge_requests/index.md#request-attention-to-a-merge-request) if they're already a reviewer. This saves reviewers time and helps authors catch mistakes earlier. @@ -259,10 +268,8 @@ This saves reviewers time and helps authors catch mistakes earlier. that it meets all requirements, you should: - Click the Approve button. -- `@` mention the author to generate a to-do notification, and advise them that their merge request has been reviewed and approved. -- Request a review from a maintainer. Default to requests for a maintainer with [domain expertise](#domain-experts), +- Request a review from a maintainer or [request their attention](../user/project/merge_requests/index.md#request-attention-to-a-merge-request) if they're already a reviewer. Default to requests for a maintainer with [domain expertise](#domain-experts), however, if one isn't available or you think the merge request doesn't need a review by a [domain expert](#domain-experts), feel free to follow the [Reviewer roulette](#reviewer-roulette) suggestion. -- Remove yourself as a reviewer. ### The responsibility of the maintainer @@ -290,7 +297,7 @@ If a developer who happens to also be a maintainer was involved in a merge reque as a reviewer, it is recommended that they are not also picked as the maintainer to ultimately approve and merge it. Maintainers should check before merging if the merge request is approved by the -required approvers. If still awaiting further approvals from others, remove yourself as a reviewer then `@` mention the author and explain why in a comment. Stay as reviewer if you're merging the code. +required approvers. If still awaiting further approvals from others, explain that in a comment and [request attention](../user/project/merge_requests/index.md#request-attention-to-a-merge-request) from other reviewers as appropriate. Do not remove yourself as a reviewer. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the merge request @@ -312,14 +319,20 @@ After merging, a maintainer should stay as the reviewer listed on the merge requ ### Dogfooding the Reviewers feature -On March 18th 2021, an updated process was put in place aimed at efficiently and consistently dogfooding the Reviewers feature. +Replaced with [dogfooding the attention request feature](#dogfooding-the-attention-request-feature). + +### Dogfooding the attention request feature + +In March of 2022, an updated process was put in place aimed at efficiently and consistently dogfooding the +[attention requests feature](../user/project/merge_requests/index.md#request-attention-to-a-merge-request) under `Merge requests` -> `Need your attention`. This replaces previous guidance on [dogfooding the reviewers feature](#dogfooding-the-reviewers-feature). Here is a summary of the changes, also reflected in this section above. -- Merge request authors and DRIs stay as Assignees -- Authors request a review from Reviewers when they are expected to review -- Reviewers remove themselves after they're done reviewing/approving -- The last approver stays as Reviewer upon merging +- Merge request authors and DRIs stay as assignees +- Assignees request a review from reviewer(s) when they are expected to review +- Reviewers stay assigned for the entire duration of the merge request +- Reviewers request attention from the assignee or other reviewer(s) after they've done reviewing, depending on who needs to take action +- Assignees request attention from the reviewer(s) when changes are made ## Best practices @@ -392,6 +405,11 @@ When you are ready to have your merge request reviewed, you should [request an initial review](../user/project/merge_requests/getting_started.md#reviewer) by selecting a reviewer from your group or team. However, you can also assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. +When a merge request has multiple areas for review, it is recommended you specify which area a reviewer should be reviewing, and at which stage (first or second). +This will help team members who qualify as a reviewer for multiple areas to know which area they're being requested to review. +For example, when a merge request has both `backend` and `frontend` concerns, you can mention the reviewer in this manner: +`@john_doe can you please review ~backend?` or `@jane_doe - could you please give this MR a ~frontend maintainer review?` + You can also use `workflow::ready for review` label. That means that your merge request is ready to be reviewed and any reviewer can pick it. It is recommended to use that label only if there isn't time pressure and make sure the merge request is assigned to a reviewer. When your merge request receives an approval from the first reviewer it can be passed to a maintainer. You should default to choosing a maintainer with [domain expertise](#domain-experts), and otherwise follow the Reviewer Roulette recommendation or use the label `ready for merge`. @@ -605,9 +623,9 @@ Enterprise Edition instance. This has some implications: migration on the staging environment if you aren't sure. 1. Categorized correctly: - Regular migrations run before the new code is running on the instance. - - [Post-deployment migrations](post_deployment_migrations.md) run _after_ + - [Post-deployment migrations](database/post_deployment_migrations.md) run _after_ the new code is deployed, when the instance is configured to do that. - - [Background migrations](background_migrations.md) run in Sidekiq, and + - [Background migrations](database/background_migrations.md) run in Sidekiq, and should only be done for migrations that would take an extreme amount of time at GitLab.com scale. 1. **Sidekiq workers** [cannot change in a backwards-incompatible way](sidekiq/compatibility_across_updates.md): diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 463a7ee0e0b..def39a960d8 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -49,7 +49,7 @@ Check these aspects both when _designing_ and _reviewing_ UI changes. ### Visual design Check visual design properties using your browser's _elements inspector_ ([Chrome](https://developer.chrome.com/docs/devtools/css/), -[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Open_the_Inspector)). +[Firefox](https://firefox-source-docs.mozilla.org/devtools-user/page_inspector/how_to/open_the_inspector/index.html)). - Use recommended [colors](https://design.gitlab.com/product-foundations/colors/) and [typography](https://design.gitlab.com/product-foundations/type-fundamentals/). @@ -66,7 +66,7 @@ Check visual design properties using your browser's _elements inspector_ ([Chrom Check states using your browser's _styles inspector_ to toggle CSS pseudo-classes like `:hover` and others ([Chrome](https://developer.chrome.com/docs/devtools/css/reference/#pseudo-class), -[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#viewing_common_pseudo-classes)). +[Firefox](https://firefox-source-docs.mozilla.org/devtools-user/page_inspector/how_to/examine_and_edit_css/index.html#viewing-common-pseudo-classes)). - Account for all applicable states ([error](https://design.gitlab.com/content/error-messages), rest, loading, focus, hover, selected, disabled). @@ -78,7 +78,7 @@ like `:hover` and others ([Chrome](https://developer.chrome.com/docs/devtools/cs ### Responsive Check responsive behavior using your browser's _responsive mode_ ([Chrome](https://developer.chrome.com/docs/devtools/device-mode/#viewport), -[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Responsive_Design_Mode)). +[Firefox](https://firefox-source-docs.mozilla.org/devtools-user/responsive_design_mode/index.html)). - Account for resizing, collapsing, moving, or wrapping of elements across all breakpoints (even if larger viewports are prioritized). @@ -99,7 +99,7 @@ Check accessibility using your browser's _accessibility inspector_ ([Chrome](htt When the design is ready, _before_ starting its implementation: - Share design specifications in the related issue, preferably through a [Figma link](https://help.figma.com/hc/en-us/articles/360040531773-Share-Files-with-anyone-using-Link-Sharing#Copy_links) - link or [GitLab Designs feature](../../user/project/issues/design_management.md#the-design-management-section). + link or [GitLab Designs feature](../../user/project/issues/design_management.md). See [when you should use each tool](https://about.gitlab.com/handbook/engineering/ux/product-designer/#deliver). - Document user flow and states (for example, using [Mermaid flowcharts in Markdown](../../user/markdown.md#mermaid)). - Document animations and transitions. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 4db686b9b1e..fe1549e7f34 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -31,11 +31,7 @@ on those issues. Please select someone with relevant experience from the If there is nobody mentioned with that expertise, look in the commit history for the affected files to find someone. -We also use [GitLab Triage](https://gitlab.com/gitlab-org/gitlab-triage) to automate -some triaging policies. This is currently set up as a scheduled pipeline -(`https://gitlab.com/gitlab-org/quality/triage-ops/-/pipeline_schedules/10512/edit`, -must have at least the Developer role in the project) running on [quality/triage-ops](https://gitlab.com/gitlab-org/quality/triage-ops) -project. +We also have triage automation in place, described [in our handbook](https://about.gitlab.com/handbook/engineering/quality/triage-operations/). ## Labels diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index a9b4d13ab06..5ed0885eed9 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -144,7 +144,7 @@ document from the Kubernetes team also has some great points regarding this. ### Commit messages guidelines -Commit messages should follow the guidelines below, for reasons explained by Chris Beams in [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/): +Commit messages should follow the guidelines below, for reasons explained by Chris Beams in [How to Write a Git Commit Message](https://cbea.ms/git-commit/): - The commit subject and body must be separated by a blank line. - The commit subject must start with a capital letter. @@ -203,7 +203,7 @@ Example commit message template that can be used on your machine that embodies t # Do not use Emojis # Use the body to explain what and why vs. how # Can use multiple lines with "-" for bullet points in body -# For more information: https://chris.beams.io/posts/git-commit/ +# For more information: https://cbea.ms/git-commit/ # -------------------- ``` @@ -286,8 +286,8 @@ requirements. ### Production use 1. Confirmed to be working in staging before implementing the change in production, where possible. -1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/#sentry) errors after the contribution is deployed. -1. Confirmed that the [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans) has been completed. +1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/monitoring/#sentry) errors after the contribution is deployed. +1. Confirmed that the [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans/) has been completed. 1. If there is a performance risk in the change, I have analyzed the performance of the system before and after the change. 1. *If the merge request uses feature flags, per-project or per-group enablement, and a staged rollout:* - Confirmed to be working on GitLab projects. diff --git a/doc/development/contributing/verify/index.md b/doc/development/contributing/verify/index.md index a2bb0eca733..828eb0a9598 100644 --- a/doc/development/contributing/verify/index.md +++ b/doc/development/contributing/verify/index.md @@ -55,7 +55,7 @@ and they serve us and our users well. Some examples of these principles are that - Feedback needs to be available when a user needs it and data can not disappear unexpectedly when engineers need it. - It all doesn’t matter if the platform is not secure and we are leaking credentials or secrets. -- When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. +- When a user provides a set of preconditions in a form of CI/CD configuration, the result should be deterministic each time a pipeline runs, because otherwise the platform might not be trustworthy. - If it is fast, simple to use and has a great UX it will serve our users well. ## Building things in Verify @@ -189,8 +189,7 @@ Slack channel (GitLab team members only). After your merge request is merged by a maintainer, it is time to release it to users and the wider community. We usually do this with feature flags. While not every merge request needs a feature flag, most merge -requests in Verify should have feature flags. [**TODO** link to docs about what -needs a feature flag and what doesn’t]. +requests in Verify should have [feature flags](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags). If you already follow the advice on this page, you probably already have a few metrics and perhaps a few loggers added that make your new code observable diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 9bf0fbe1d78..f941e0720c6 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -142,7 +142,7 @@ To enable the Dangerfile on another existing GitLab project, complete the follow 1. Create a `Dangerfile` with the following content: ```ruby - require_relative "lib/gitlab-dangerfiles" + require "gitlab-dangerfiles" Gitlab::Dangerfiles.for_project(self, &:import_defaults) ``` @@ -154,6 +154,8 @@ To enable the Dangerfile on another existing GitLab project, complete the follow - project: 'gitlab-org/quality/pipeline-common' file: - '/ci/danger-review.yml' + rules: + - if: '$CI_SERVER_HOST == "gitlab.com"' ``` 1. If your project is in the `gitlab-org` group, you don't need to set up any token as the `DANGER_GITLAB_API_TOKEN` diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index d74f826cc14..bfd455ef9da 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -123,7 +123,7 @@ end Validating the foreign key scans the whole table and makes sure that each relation is correct. NOTE: -When using [background migrations](../background_migrations.md), foreign key validation should happen in the next GitLab release. +When using [background migrations](background_migrations.md), foreign key validation should happen in the next GitLab release. Migration file for validating the foreign key: diff --git a/doc/development/database/avoiding_downtime_in_migrations.md b/doc/development/database/avoiding_downtime_in_migrations.md new file mode 100644 index 00000000000..ad2768397e6 --- /dev/null +++ b/doc/development/database/avoiding_downtime_in_migrations.md @@ -0,0 +1,491 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Avoiding downtime in migrations + +When working with a database certain operations may require downtime. Since we +cannot have downtime in migrations we need to use a set of steps to get the +same end result without downtime. This guide describes various operations that +may appear to need downtime, their impact, and how to perform them without +requiring downtime. + +## Dropping Columns + +Removing columns is tricky because running GitLab processes may still be using +the columns. To work around this safely, you will need three steps in three releases: + +1. Ignoring the column (release M) +1. Dropping the column (release M+1) +1. Removing the ignore rule (release M+2) + +The reason we spread this out across three releases is that dropping a column is +a destructive operation that can't be rolled back easily. + +Following this procedure helps us to make sure there are no deployments to GitLab.com +and upgrade processes for self-managed installations that lump together any of these steps. + +### Step 1: Ignoring the column (release M) + +The first step is to ignore the column in the application code. This is +necessary because Rails caches the columns and re-uses this cache in various +places. This can be done by defining the columns to ignore. For example, to ignore +`updated_at` in the User model you'd use the following: + +```ruby +class User < ApplicationRecord + include IgnorableColumns + ignore_column :updated_at, remove_with: '12.7', remove_after: '2020-01-22' +end +``` + +Multiple columns can be ignored, too: + +```ruby +ignore_columns %i[updated_at created_at], remove_with: '12.7', remove_after: '2020-01-22' +``` + +If the model exists in CE and EE, the column has to be ignored in the CE model. If the +model only exists in EE, then it has to be added there. + +We require indication of when it is safe to remove the column ignore with: + +- `remove_with`: set to a GitLab release typically two releases (M+2) after adding the + column ignore. +- `remove_after`: set to a date after which we consider it safe to remove the column + ignore, typically after the M+1 release date, during the M+2 development cycle. + +This information allows us to reason better about column ignores and makes sure we +don't remove column ignores too early for both regular releases and deployments to GitLab.com. For +example, this avoids a situation where we deploy a bulk of changes that include both changes +to ignore the column and subsequently remove the column ignore (which would result in a downtime). + +In this example, the change to ignore the column went into release 12.5. + +### Step 2: Dropping the column (release M+1) + +Continuing our example, dropping the column goes into a _post-deployment_ migration in release 12.6: + +```ruby + remove_column :user, :updated_at +``` + +### Step 3: Removing the ignore rule (release M+2) + +With the next release, in this example 12.7, we set up another merge request to remove the ignore rule. +This removes the `ignore_column` line and - if not needed anymore - also the inclusion of `IgnoreableColumns`. + +This should only get merged with the release indicated with `remove_with` and once +the `remove_after` date has passed. + +## Renaming Columns + +Renaming columns the normal way requires downtime as an application may continue +using the old column name during/after a database migration. To rename a column +without requiring downtime we need two migrations: a regular migration, and a +post-deployment migration. Both these migration can go in the same release. + +### Step 1: Add The Regular Migration + +First we need to create the regular migration. This migration should use +`Gitlab::Database::MigrationHelpers#rename_column_concurrently` to perform the +renaming. For example + +```ruby +# A regular migration in db/migrate +class RenameUsersUpdatedAtToUpdatedAtTimestamp < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end + + def down + undo_rename_column_concurrently :users, :updated_at, :updated_at_timestamp + end +end +``` + +This will take care of renaming the column, ensuring data stays in sync, and +copying over indexes and foreign keys. + +If a column contains one or more indexes that don't contain the name of the +original column, the previously described procedure will fail. In that case, +you'll first need to rename these indexes. + +### Step 2: Add A Post-Deployment Migration + +The renaming procedure requires some cleaning up in a post-deployment migration. +We can perform this cleanup using +`Gitlab::Database::MigrationHelpers#cleanup_concurrent_column_rename`: + +```ruby +# A post-deployment migration in db/post_migrate +class CleanupUsersUpdatedAtRename < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end + + def down + undo_cleanup_concurrent_column_rename :users, :updated_at, :updated_at_timestamp + end +end +``` + +If you're renaming a [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3), please carefully consider the state when the first migration has run but the second cleanup migration hasn't been run yet. +With [Canary](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/canary/) it is possible that the system runs in this state for a significant amount of time. + +## Changing Column Constraints + +Adding or removing a `NOT NULL` clause (or another constraint) can typically be +done without requiring downtime. However, this does require that any application +changes are deployed _first_. Thus, changing the constraints of a column should +happen in a post-deployment migration. + +Avoid using `change_column` as it produces an inefficient query because it re-defines +the whole column type. + +You can check the following guides for each specific use case: + +- [Adding foreign-key constraints](../migration_style_guide.md#adding-foreign-key-constraints) +- [Adding `NOT NULL` constraints](not_null_constraints.md) +- [Adding limits to text columns](strings_and_the_text_data_type.md) + +## Changing Column Types + +Changing the type of a column can be done using +`Gitlab::Database::MigrationHelpers#change_column_type_concurrently`. This +method works similarly to `rename_column_concurrently`. For example, let's say +we want to change the type of `users.username` from `string` to `text`. + +### Step 1: Create A Regular Migration + +A regular migration is used to create a new column with a temporary name along +with setting up some triggers to keep data in sync. Such a migration would look +as follows: + +```ruby +# A regular migration in db/migrate +class ChangeUsersUsernameStringToText < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + change_column_type_concurrently :users, :username, :text + end + + def down + undo_change_column_type_concurrently :users, :username + end +end +``` + +### Step 2: Create A Post Deployment Migration + +Next we need to clean up our changes using a post-deployment migration: + +```ruby +# A post-deployment migration in db/post_migrate +class ChangeUsersUsernameStringToTextCleanup < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + cleanup_concurrent_column_type_change :users, :username + end + + def down + undo_cleanup_concurrent_column_type_change :users, :username, :string + end +end +``` + +And that's it, we're done! + +### Casting data to a new type + +Some type changes require casting data to a new type. For example when changing from `text` to `jsonb`. +In this case, use the `type_cast_function` option. +Make sure there is no bad data and the cast will always succeed. You can also provide a custom function that handles +casting errors. + +Example migration: + +```ruby + def up + change_column_type_concurrently :users, :settings, :jsonb, type_cast_function: 'jsonb' + end +``` + +## Changing The Schema For Large Tables + +While `change_column_type_concurrently` and `rename_column_concurrently` can be +used for changing the schema of a table without downtime, it doesn't work very +well for large tables. Because all of the work happens in sequence the migration +can take a very long time to complete, preventing a deployment from proceeding. +They can also produce a lot of pressure on the database due to it rapidly +updating many rows in sequence. + +To reduce database pressure you should instead use a background migration +when migrating a column in a large table (for example, `issues`). This will +spread the work / load over a longer time period, without slowing down deployments. + +For more information, see [the documentation on cleaning up background +migrations](background_migrations.md#cleaning-up). + +## Adding Indexes + +Adding indexes does not require downtime when `add_concurrent_index` +is used. + +See also [Migration Style Guide](../migration_style_guide.md#adding-indexes) +for more information. + +## Dropping Indexes + +Dropping an index does not require downtime. + +## Adding Tables + +This operation is safe as there's no code using the table just yet. + +## Dropping Tables + +Dropping tables can be done safely using a post-deployment migration, but only +if the application no longer uses the table. + +## Renaming Tables + +Renaming tables requires downtime as an application may continue +using the old table name during/after a database migration. + +If the table and the ActiveRecord model is not in use yet, removing the old +table and creating a new one is the preferred way to "rename" the table. + +Renaming a table is possible without downtime by following our multi-release +[rename table process](rename_database_tables.md#rename-table-without-downtime). + +## Adding Foreign Keys + +Adding foreign keys usually works in 3 steps: + +1. Start a transaction +1. Run `ALTER TABLE` to add the constraint(s) +1. Check all existing data + +Because `ALTER TABLE` typically acquires an exclusive lock until the end of a +transaction this means this approach would require downtime. + +GitLab allows you to work around this by using +`Gitlab::Database::MigrationHelpers#add_concurrent_foreign_key`. This method +ensures that no downtime is needed. + +## Removing Foreign Keys + +This operation does not require downtime. + +## Migrating `integer` primary keys to `bigint` + +To [prevent the overflow risk](https://gitlab.com/groups/gitlab-org/-/epics/4785) for some tables +with `integer` primary key (PK), we have to migrate their PK to `bigint`. The process to do this +without downtime and causing too much load on the database is described below. + +### Initialize the conversion and start migrating existing data (release N) + +To start the process, add a regular migration to create the new `bigint` columns. Use the provided +`initialize_conversion_of_integer_to_bigint` helper. The helper also creates a database trigger +to keep in sync both columns for any new records ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072312_initialize_conversion_of_ci_stages_to_bigint.rb)): + +```ruby +class InitializeConversionOfCiStagesToBigint < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +Ignore the new `bigint` columns: + +```ruby +module Ci + class Stage < Ci::ApplicationRecord + include IgnorableColumns + ignore_column :id_convert_to_bigint, remove_with: '14.2', remove_after: '2021-08-22' + end +``` + +To migrate existing data, we introduced new type of _batched background migrations_. +Unlike the classic background migrations, built on top of Sidekiq, batched background migrations +don't have to enqueue and schedule all the background jobs at the beginning. +They also have other advantages, like automatic tuning of the batch size, better progress visibility, +and collecting metrics. To start the process, use the provided `backfill_conversion_of_integer_to_bigint` +helper ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072346_backfill_ci_stages_for_bigint_conversion.rb)): + +```ruby +class BackfillCiStagesForBigintConversion < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +### Monitor the background migration + +Check how the migration is performing while it's running. Multiple ways to do this are described below. + +#### High-level status of batched background migrations + +See how to [check the status of batched background migrations](../../update/index.md#checking-for-background-migrations-before-upgrading). + +#### Query the database + +We can query the related database tables directly. Requires access to read-only replica. +Example queries: + +```sql +-- Get details for batched background migration for given table +SELECT * FROM batched_background_migrations WHERE table_name = 'namespaces'\gx + +-- Get count of batched background migration jobs by status for given table +SELECT + batched_background_migrations.id, batched_background_migration_jobs.status, COUNT(*) +FROM + batched_background_migrations + JOIN batched_background_migration_jobs ON batched_background_migrations.id = batched_background_migration_jobs.batched_background_migration_id +WHERE + table_name = 'namespaces' +GROUP BY + batched_background_migrations.id, batched_background_migration_jobs.status; + +-- Batched background migration progress for given table (based on estimated total number of tuples) +SELECT + m.table_name, + LEAST(100 * sum(j.batch_size) / pg_class.reltuples, 100) AS percentage_complete +FROM + batched_background_migrations m + JOIN batched_background_migration_jobs j ON j.batched_background_migration_id = m.id + JOIN pg_class ON pg_class.relname = m.table_name +WHERE + j.status = 3 AND m.table_name = 'namespaces' +GROUP BY m.id, pg_class.reltuples; +``` + +#### Sidekiq logs + +We can also use the Sidekiq logs to monitor the worker that executes the batched background +migrations: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-sidekiq-inf-gprd*`. +1. Add filter for `json.queue: cronjob:database_batched_background_migration`. + +#### PostgreSQL slow queries log + +Slow queries log keeps track of low queries that took above 1 second to execute. To see them +for batched background migration: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-postgres-inf-gprd*`. +1. Add filter for `json.endpoint_id.keyword: Database::BatchedBackgroundMigrationWorker`. +1. Optional. To see only updates, add a filter for `json.command_tag.keyword: UPDATE`. +1. Optional. To see only failed statements, add a filter for `json.error_severity.keyword: ERROR`. +1. Optional. Add a filter by table name. + +#### Grafana dashboards + +To monitor the health of the database, use these additional metrics: + +- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that autovacuum cannot keep up. +- [PostgreSQL Overview](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1): if you see high system usage or transactions per second (TPS) on the primary database server, it might mean that the migration is causing problems. + +### Prometheus metrics + +Number of [metrics](https://gitlab.com/gitlab-org/gitlab/-/blob/294a92484ce4611f660439aa48eee4dfec2230b5/lib/gitlab/database/background_migration/batched_migration_wrapper.rb#L90-128) +for each batched background migration are published to Prometheus. These metrics can be searched for and +visualized in Thanos ([see an example](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20(rate(batched_migration_job_updated_tuples_total%7Benv%3D%22gprd%22%7D%5B5m%5D))%20by%20(migration_id)%20&g0.tab=0&g0.stacked=0&g0.range_input=3d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-06-13%2012%3A18%3A24&g0.moment_input=2021-06-13%2012%3A18%3A24)). + +### Swap the columns (release N + 1) + +After the background is completed and the new `bigint` columns are populated for all records, we can +swap the columns. Swapping is done with post-deployment migration. The exact process depends on the +table being converted, but in general it's done in the following steps: + +1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched +migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). +If the migration has not completed, the subsequent steps fail anyway. By checking in advance we +aim to have more helpful error message. +1. Create indexes using the `bigint` columns that match the existing indexes using the `integer` +column ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L28-34)). +1. Create foreign keys (FK) using the `bigint` columns that match the existing FKs using the +`integer` column. Do this both for FK referencing other tables, and FKs that reference the table +that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). +1. Inside a transaction, swap the columns: + 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). + 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) + 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). + 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). + 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). + 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). + 1. Remove old FKs (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + +See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). + +### Remove the trigger and old `integer` columns (release N + 2) + +Using post-deployment migration and the provided `cleanup_conversion_of_integer_to_bigint` helper, +drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69714)). + +### Remove ignore rules (release N + 3) + +In the next release after the columns were dropped, remove the ignore rules as we do not need them +anymore ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71161)). + +## Data migrations + +Data migrations can be tricky. The usual approach to migrate data is to take a 3 +step approach: + +1. Migrate the initial batch of data +1. Deploy the application code +1. Migrate any remaining data + +Usually this works, but not always. For example, if a field's format is to be +changed from JSON to something else we have a bit of a problem. If we were to +change existing data before deploying application code we'll most likely run +into errors. On the other hand, if we were to migrate after deploying the +application code we could run into the same problems. + +If you merely need to correct some invalid data, then a post-deployment +migration is usually enough. If you need to change the format of data (for example, from +JSON to something else) it's typically best to add a new column for the new data +format, and have the application use that. In such a case the procedure would +be: + +1. Add a new column in the new format +1. Copy over existing data to this new column +1. Deploy the application code +1. In a post-deployment migration, copy over any remaining data + +In general there is no one-size-fits-all solution, therefore it's best to +discuss these kind of migrations in a merge request to make sure they are +implemented in the best way possible. diff --git a/doc/development/database/background_migrations.md b/doc/development/database/background_migrations.md new file mode 100644 index 00000000000..1f7e0d76c89 --- /dev/null +++ b/doc/development/database/background_migrations.md @@ -0,0 +1,504 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Background migrations + +WARNING: +Background migrations are strongly discouraged in favor of the new [batched background migrations framework](../batched_background_migrations.md). +Please check that documentation and determine if that framework suits your needs and fall back +to these only if required. + +Background migrations should be used to perform data migrations whenever a +migration exceeds [the time limits in our guidelines](../migration_style_guide.md#how-long-a-migration-should-take). For example, you can use background +migrations to migrate data that's stored in a single JSON column +to a separate table instead. + +If the database cluster is considered to be in an unhealthy state, background +migrations automatically reschedule themselves for a later point in time. + +## When To Use Background Migrations + +You should use a background migration when you migrate _data_ in tables that have +so many rows that the process would exceed [the time limits in our guidelines](../migration_style_guide.md#how-long-a-migration-should-take) if performed using a regular Rails migration. + +- Background migrations should be used when migrating data in [high-traffic tables](../migration_style_guide.md#high-traffic-tables). +- Background migrations may also be used when executing numerous single-row queries +for every item on a large dataset. Typically, for single-record patterns, runtime is +largely dependent on the size of the dataset, hence it should be split accordingly +and put into background migrations. +- Background migrations should not be used to perform schema migrations. + +Some examples where background migrations can be useful: + +- Migrating events from one table to multiple separate tables. +- Populating one column based on JSON stored in another column. +- Migrating data that depends on the output of external services (for example, an API). + +NOTE: +If the background migration is part of an important upgrade, make sure it's announced +in the release post. Discuss with your Project Manager if you're not sure the migration falls +into this category. + +## Isolation + +Background migrations must be isolated and can not use application code (for example, +models defined in `app/models`). Since these migrations can take a long time to +run it's possible for new versions to be deployed while they are still running. + +It's also possible for different migrations to be executed at the same time. +This means that different background migrations should not migrate data in a +way that would cause conflicts. + +## Idempotence + +Background migrations are executed in a context of a Sidekiq process. +Usual Sidekiq rules apply, especially the rule that jobs should be small +and idempotent. + +See [Sidekiq best practices guidelines](https://github.com/mperham/sidekiq/wiki/Best-Practices) +for more details. + +Make sure that in case that your migration job is going to be retried data +integrity is guaranteed. + +## Background migrations for EE-only features + +All the background migration classes for EE-only features should be present in GitLab CE. +For this purpose, an empty class can be created for GitLab CE, and it can be extended for GitLab EE +as explained in the [guidelines for implementing Enterprise Edition features](../ee_features.md#code-in-libgitlabbackground_migration). + +## How It Works + +Background migrations are simple classes that define a `perform` method. A +Sidekiq worker will then execute such a class, passing any arguments to it. All +migration classes must be defined in the namespace +`Gitlab::BackgroundMigration`, the files should be placed in the directory +`lib/gitlab/background_migration/`. + +## Scheduling + +Scheduling a background migration should be done in a post-deployment +migration that includes `Gitlab::Database::MigrationHelpers` +To do so, simply use the following code while +replacing the class name and arguments with whatever values are necessary for +your migration: + +```ruby +migrate_in('BackgroundMigrationClassName', [arg1, arg2, ...]) +``` + +You can use the function `queue_background_migration_jobs_by_range_at_intervals` +to automatically split the job into batches: + +```ruby +queue_background_migration_jobs_by_range_at_intervals( + ClassName, + BackgroundMigrationClassName, + 2.minutes, + batch_size: 10_000 + ) +``` + +You'll also need to make sure that newly created data is either migrated, or +saved in both the old and new version upon creation. For complex and time +consuming migrations it's best to schedule a background job using an +`after_create` hook so this doesn't affect response timings. The same applies to +updates. Removals in turn can be handled by simply defining foreign keys with +cascading deletes. + +### Rescheduling background migrations + +If one of the background migrations contains a bug that is fixed in a patch +release, the background migration needs to be rescheduled so the migration would +be repeated on systems that already performed the initial migration. + +When you reschedule the background migration, make sure to turn the original +scheduling into a no-op by clearing up the `#up` and `#down` methods of the +migration performing the scheduling. Otherwise the background migration would be +scheduled multiple times on systems that are upgrading multiple patch releases at +once. + +When you start the second post-deployment migration, you should delete any +previously queued jobs from the initial migration with the provided +helper: + +```ruby +delete_queued_jobs('BackgroundMigrationClassName') +``` + +## Cleaning Up + +NOTE: +Cleaning up any remaining background migrations _must_ be done in either a major +or minor release, you _must not_ do this in a patch release. + +Because background migrations can take a long time you can't immediately clean +things up after scheduling them. For example, you can't drop a column that's +used in the migration process as this would cause jobs to fail. This means that +you'll need to add a separate _post deployment_ migration in a future release +that finishes any remaining jobs before cleaning things up (for example, removing a +column). + +As an example, say you want to migrate the data from column `foo` (containing a +big JSON blob) to column `bar` (containing a string). The process for this would +roughly be as follows: + +1. Release A: + 1. Create a migration class that performs the migration for a row with a given ID. + You can use [background jobs tracking](#background-jobs-tracking) to simplify cleaning up. + 1. Deploy the code for this release, this should include some code that will + schedule jobs for newly created data (for example, using an `after_create` hook). + 1. Schedule jobs for all existing rows in a post-deployment migration. It's + possible some newly created rows may be scheduled twice so your migration + should take care of this. +1. Release B: + 1. Deploy code so that the application starts using the new column and stops + scheduling jobs for newly created data. + 1. In a post-deployment migration, finalize all jobs that have not succeeded by now. + If you used [background jobs tracking](#background-jobs-tracking) in release A, + you can use `finalize_background_migration` from `BackgroundMigrationHelpers` to ensure no jobs remain. + This helper will: + 1. Use `Gitlab::BackgroundMigration.steal` to process any remaining + jobs in Sidekiq. + 1. Reschedule the migration to be run directly (that is, not through Sidekiq) + on any rows that weren't migrated by Sidekiq. This can happen if, for + instance, Sidekiq received a SIGKILL, or if a particular batch failed + enough times to be marked as dead. + 1. Remove `Gitlab::Database::BackgroundMigrationJob` rows where + `status = succeeded`. To retain diagnostic information that may + help with future bug tracking you can skip this step by specifying + the `delete_tracking_jobs: false` parameter. + 1. Remove the old column. + +This may also require a bump to the [import/export version](../../user/project/settings/import_export.md), if +importing a project from a prior version of GitLab requires the data to be in +the new format. + +## Example + +To explain all this, let's use the following example: the table `integrations` has a +field called `properties` which is stored in JSON. For all rows you want to +extract the `url` key from this JSON object and store it in the `integrations.url` +column. There are millions of integrations and parsing JSON is slow, thus you can't +do this in a regular migration. + +To do this using a background migration we'll start with defining our migration +class: + +```ruby +class Gitlab::BackgroundMigration::ExtractIntegrationsUrl + class Integration < ActiveRecord::Base + self.table_name = 'integrations' + end + + def perform(start_id, end_id) + Integration.where(id: start_id..end_id).each do |integration| + json = JSON.load(integration.properties) + + integration.update(url: json['url']) if json['url'] + rescue JSON::ParserError + # If the JSON is invalid we don't want to keep the job around forever, + # instead we'll just leave the "url" field to whatever the default value + # is. + next + end + end +end +``` + +Next we'll need to adjust our code so we schedule the above migration for newly +created and updated integrations. We can do this using something along the lines of +the following: + +```ruby +class Integration < ActiveRecord::Base + after_commit :schedule_integration_migration, on: :update + after_commit :schedule_integration_migration, on: :create + + def schedule_integration_migration + BackgroundMigrationWorker.perform_async('ExtractIntegrationsUrl', [id, id]) + end +end +``` + +We're using `after_commit` here to ensure the Sidekiq job is not scheduled +before the transaction completes as doing so can lead to race conditions where +the changes are not yet visible to the worker. + +Next we'll need a post-deployment migration that schedules the migration for +existing data. + +```ruby +class ScheduleExtractIntegrationsUrl < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + MIGRATION = 'ExtractIntegrationsUrl' + DELAY_INTERVAL = 2.minutes + + def up + queue_background_migration_jobs_by_range_at_intervals( + define_batchable_model('integrations'), + MIGRATION, + DELAY_INTERVAL) + end + + def down + end +end +``` + +Once deployed our application will continue using the data as before but at the +same time will ensure that both existing and new data is migrated. + +In the next release we can remove the `after_commit` hooks and related code. We +will also need to add a post-deployment migration that consumes any remaining +jobs and manually run on any un-migrated rows. Such a migration would look like +this: + +```ruby +class ConsumeRemainingExtractIntegrationsUrlJobs < Gitlab::Database::Migration[1.0] + disable_ddl_transaction! + + def up + # This must be included + Gitlab::BackgroundMigration.steal('ExtractIntegrationsUrl') + + # This should be included, but can be skipped - see below + define_batchable_model('integrations').where(url: nil).each_batch(of: 50) do |batch| + range = batch.pluck('MIN(id)', 'MAX(id)').first + + Gitlab::BackgroundMigration::ExtractIntegrationsUrl.new.perform(*range) + end + end + + def down + end +end +``` + +The final step runs for any un-migrated rows after all of the jobs have been +processed. This is in case a Sidekiq process running the background migrations +received SIGKILL, leading to the jobs being lost. (See +[more reliable Sidekiq queue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36791) for more information.) + +If the application does not depend on the data being 100% migrated (for +instance, the data is advisory, and not mission-critical), then this final step +can be skipped. + +This migration will then process any jobs for the ExtractIntegrationsUrl migration +and continue once all jobs have been processed. Once done you can safely remove +the `integrations.properties` column. + +## Testing + +It is required to write tests for: + +- The background migrations' scheduling migration. +- The background migration itself. +- A cleanup migration. + +The `:migration` and `schema: :latest` RSpec tags are automatically set for +background migration specs. +See the +[Testing Rails migrations](../testing_guide/testing_migrations_guide.md#testing-a-non-activerecordmigration-class) +style guide. + +Keep in mind that `before` and `after` RSpec hooks are going +to migrate you database down and up, which can result in other background +migrations being called. That means that using `spy` test doubles with +`have_received` is encouraged, instead of using regular test doubles, because +your expectations defined in a `it` block can conflict with what is being +called in RSpec hooks. See [issue #35351](https://gitlab.com/gitlab-org/gitlab/-/issues/18839) +for more details. + +## Best practices + +1. Make sure to know how much data you're dealing with. +1. Make sure that background migration jobs are idempotent. +1. Make sure that tests you write are not false positives. +1. Make sure that if the data being migrated is critical and cannot be lost, the + clean-up migration also checks the final state of the data before completing. +1. When migrating many columns, make sure it won't generate too many + dead tuples in the process (you may need to directly query the number of dead tuples + and adjust the scheduling according to this piece of data). +1. Make sure to discuss the numbers with a database specialist, the migration may add + more pressure on DB than you expect (measure on staging, + or ask someone to measure on production). +1. Make sure to know how much time it'll take to run all scheduled migrations. +1. Provide an estimation section in the description, estimating both the total migration + run time and the query times for each background migration job. Explain plans for each query + should also be provided. + + For example, assuming a migration that deletes data, include information similar to + the following section: + + ```plaintext + Background Migration Details: + + 47600 items to delete + batch size = 1000 + 47600 / 1000 = 48 batches + + Estimated times per batch: + - 820ms for select statement with 1000 items (see linked explain plan) + - 900ms for delete statement with 1000 items (see linked explain plan) + Total: ~2 sec per batch + + 2 mins delay per batch (safe for the given total time per batch) + + 48 batches * 2 min per batch = 96 mins to run all the scheduled jobs + ``` + + The execution time per batch (2 sec in this example) is not included in the calculation + for total migration time. The jobs are scheduled 2 minutes apart without knowledge of + the execution time. + +## Additional tips and strategies + +### Nested batching + +A strategy to make the migration run faster is to schedule larger batches, and then use `EachBatch` +within the background migration to perform multiple statements. + +The background migration helpers that queue multiple jobs such as +`queue_background_migration_jobs_by_range_at_intervals` use [`EachBatch`](../iterating_tables_in_batches.md). +The example above has batches of 1000, where each queued job takes two seconds. If the query has been optimized +to make the time for the delete statement within the [query performance guidelines](../query_performance.md), +1000 may be the largest number of records that can be deleted in a reasonable amount of time. + +The minimum and most common interval for delaying jobs is two minutes. This results in two seconds +of work for each two minute job. There's nothing that prevents you from executing multiple delete +statements in each background migration job. + +Looking at the example above, you could alternatively do: + +```plaintext +Background Migration Details: + +47600 items to delete +batch size = 10_000 +47600 / 10_000 = 5 batches + +Estimated times per batch: +- Records are updated in sub-batches of 1000 => 10_000 / 1000 = 10 total updates +- 820ms for select statement with 1000 items (see linked explain plan) +- 900ms for delete statement with 1000 items (see linked explain plan) +Sub-batch total: ~2 sec per sub-batch, +Total batch time: 2 * 10 = 20 sec per batch + +2 mins delay per batch + +5 batches * 2 min per batch = 10 mins to run all the scheduled jobs +``` + +The batch time of 20 seconds still fits comfortably within the two minute delay, yet the total run +time is cut by a tenth from around 100 minutes to 10 minutes! When dealing with large background +migrations, this can cut the total migration time by days. + +When batching in this way, it is important to look at query times on the higher end +of the table or relation being updated. `EachBatch` may generate some queries that become much +slower when dealing with higher ID ranges. + +### Delay time + +When looking at the batch execution time versus the delay time, the execution time +should fit comfortably within the delay time for a few reasons: + +- To allow for a variance in query times. +- To allow autovacuum to catch up after periods of high churn. + +Never try to optimize by fully filling the delay window even if you are confident +the queries themselves have no timing variance. + +### Background jobs tracking + +NOTE: +Background migrations with job tracking enabled must call `mark_all_as_succeeded` for its batch, even if no work is needed to be done. + +`queue_background_migration_jobs_by_range_at_intervals` can create records for each job that is scheduled to run. +You can enable this behavior by passing `track_jobs: true`. Each record starts with a `pending` status. Make sure that your worker updates the job status to `succeeded` by calling `Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded` in the `perform` method of your background migration. + +```ruby +# Background migration code + +def perform(start_id, end_id) + # do work here + + mark_job_as_succeeded(start_id, end_id) +end + +private + +# Make sure that the arguments passed here match those passed to the background +# migration +def mark_job_as_succeeded(*arguments) + Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded( + self.class.name.demodulize, + arguments + ) +end +``` + +```ruby +# Post deployment migration +MIGRATION = 'YourBackgroundMigrationName' +DELAY_INTERVAL = 2.minutes.to_i # can be different +BATCH_SIZE = 10_000 # can be different + +disable_ddl_transaction! + +def up + queue_background_migration_jobs_by_range_at_intervals( + define_batchable_model('name_of_the_table_backing_the_model'), + MIGRATION, + DELAY_INTERVAL, + batch_size: BATCH_SIZE, + track_jobs: true + ) +end + +def down + # no-op +end +``` + +See [`lib/gitlab/background_migration/drop_invalid_vulnerabilities.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/background_migration/drop_invalid_vulnerabilities.rb) for a full example. + +#### Rescheduling pending jobs + +You can reschedule pending migrations from the `background_migration_jobs` table by creating a post-deployment migration and calling `requeue_background_migration_jobs_by_range_at_intervals` with the migration name and delay interval. + +```ruby +# Post deployment migration +MIGRATION = 'YourBackgroundMigrationName' +DELAY_INTERVAL = 2.minutes + +disable_ddl_transaction! + +def up + requeue_background_migration_jobs_by_range_at_intervals(MIGRATION, DELAY_INTERVAL) +end + +def down + # no-op +end +``` + +See [`db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/db/post_migrate/20210604070207_retry_backfill_traversal_ids.rb) for a full example. + +### Viewing failure error logs + +After running a background migration, if any jobs have failed, you can view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). +View the production Sidekiq log and filter for: + +- `json.class: BackgroundMigrationWorker` +- `json.job_status: fail` +- `json.meta.caller_id: <MyBackgroundMigrationSchedulingMigrationClassName>` +- `json.args: <MyBackgroundMigrationClassName>` + +Looking at the `json.error_class`, `json.error_message` and `json.error_backtrace` values may be helpful in understanding why the jobs failed. + +Depending on when and how the failure occurred, you may find other helpful information by filtering with `json.class: <MyBackgroundMigrationClassName>`. diff --git a/doc/development/database/client_side_connection_pool.md b/doc/development/database/client_side_connection_pool.md index 8316a75ac8d..60c8665df87 100644 --- a/doc/development/database/client_side_connection_pool.md +++ b/doc/development/database/client_side_connection_pool.md @@ -1,8 +1,7 @@ --- -type: dev, reference -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Client-side connection-pool diff --git a/doc/development/database/database_lab.md b/doc/development/database/database_lab.md new file mode 100644 index 00000000000..1c8694b113d --- /dev/null +++ b/doc/development/database/database_lab.md @@ -0,0 +1,101 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Database Lab and Postgres.ai + +Internal users at GitLab have access to the Database Lab Engine (DLE) and +[postgres.ai](https://console.postgres.ai/) for testing performance of database queries +on replicated production data. Unlike a typical read-only production replica, in the DLE you can +also create, update, and delete rows. You can also test the performance of +schema changes, like additional indexes or columns, in an isolated copy of production data. + +## Access Database Lab Engine + +Access to the DLE is helpful for: + +- Database reviewers and maintainers. +- Engineers who work on merge requests that have large effects on databases. + +To access the DLE's services, you can: + +- Perform query testing in the `#database_lab` Slack channel, or in the Postgres.ai web console. + Employees access both services with their GitLab Google account. Query testing + provides `EXPLAIN` (analyze, buffers) plans for queries executed there. +- Migration testing by triggering a job as a part of a merge request. +- Direct `psql` access to DLE instead of a production replica. Available to authorized users only. + To request `psql` access, file an [access request](https://about.gitlab.com/handbook/business-technology/team-member-enablement/onboarding-access-requests/access-requests/#individual-or-bulk-access-request). + +For more assistance, use the `#database` Slack channel. + +NOTE: +If you need only temporary access to a production replica, instead of a Database Lab +clone, follow the runbook procedure for connecting to the +[database console with Teleport](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Database_Console_via_Teleport.md). +This procedure is similar to [Rails console access with Teleport](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console). + +### Query testing + +You can access Database Lab's query analysis features either: + +- In the `#database_lab` Slack channel. Shows everyone's commands and results, but + your own commands are still isolated in their own clone. +- In [the Postgres.ai web console](https://console.postgres.ai/GitLab/joe-instances). + Shows only the commands you run. + +#### Generate query plans + +Query plans are an essential part of the database review process. These plans +enable us to decide quickly if a given query can be performant on GitLab.com. +Running the `explain` command generates an `explain` plan and a link to the Postgres.ai +console with more query analysis. For example, running `EXPLAIN SELECT * FROM application_settings` +does the following: + +1. Runs `explain (analyze, buffers) select * from application_settings;` against a database clone. +1. Responds with timing and buffer details from the run. +1. Provides a [detailed, shareable report on the results](https://console.postgres.ai/shared/24d543c9-893b-4ff6-8deb-a8f902f85a53). + +#### Making schema changes + +Sometimes when testing queries, a contributor may realize that the query needs an index +or other schema change to make added queries more performant. To test the query, run the `exec` command. +For example, running this command: + +```sql +exec CREATE INDEX on application_settings USING btree (instance_administration_project_id) +``` + +creates the specified index on the table. You can [test queries](#generate-query-plans) leveraging +the new index. `exec` does not return any results, only the time required to execute the query. + +#### Reset the clone + +After many changes, such as after a destructive query or an ineffective index, +you must start over. To reset your designated clone, run `reset`. + +### Migration testing + +For information on testing migrations, review our +[database migration testing documentation](database_migration_pipeline.md). + +### Access the console with `psql` + +Team members with [`psql` access](#access-database-lab-engine), can gain direct access +to a clone via `psql`. Access to `psql` enables you to see data, not just metadata. + +To connect to a clone using `psql`: + +1. Create a clone from the [desired instance](https://console.postgres.ai/gitlab/instances/). + 1. Provide a **Clone ID**: Something that uniquely identifies your clone, such as `yourname-testing-gitlabissue`. + 1. Provide a **Database username** and **Database password**: Connects `psql` to your clone. + 1. Select **Enable deletion protection** if you want to preserve your clone. Avoid selecting this option. + Clones are removed after 12 hours. +1. In the **Clone details** page of the Postgres.ai web interface, copy and run + the command to start SSH port forwarding for the clone. +1. In the **Clone details** page of the Postgres.ai web interface, copy and run the `psql` connection string. + Use the password provided at setup. + +After you connect, use clone like you would any `psql` console in production, but with +the added benefit and safety of an isolated writeable environment. diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 9d5e4821c9f..ca9ca36b156 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -70,7 +70,7 @@ Finally, you can find various guides in the [Database guides](index.md) page tha topics and use cases. The most frequently required during database reviewing are the following: - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations. -- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md). +- [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md). - [SQL guidelines](../sql.md) for working with SQL queries. - [Guidelines for JiHu contributions with database migrations](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/jihu-database-change-process.html) diff --git a/doc/development/database/deleting_migrations.md b/doc/development/database/deleting_migrations.md new file mode 100644 index 00000000000..be9009f365d --- /dev/null +++ b/doc/development/database/deleting_migrations.md @@ -0,0 +1,39 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Delete existing migrations + +When removing existing migrations from the GitLab project, you have to take into account +the possibility of the migration already been included in past releases or in the current release, and thus already executed on GitLab.com and/or in self-managed instances. + +Because of it, it's not possible to delete existing migrations, as that could lead to: + +- Schema inconsistency, as changes introduced into the database were not rolled back properly. +- Leaving a record on the `schema_versions` table, that points out to migration that no longer exists on the codebase. + +Instead of deleting we can opt for disabling the migration. + +## Pre-requisites to disable a migration + +Migrations can be disabled if: + +- They caused a timeout or general issue on GitLab.com. +- They are obsoleted, for example, changes are not necessary due to a feature change. +- Migration is a data migration only, that is, the migration does not change the database schema. + +## How to disable a data migration? + +In order to disable a migration, the following steps apply to all types of migrations: + +1. Turn the migration into a no-op by removing the code inside `#up`, `#down` + or `#perform` methods, and adding `# no-op` comment instead. +1. Add a comment explaining why the code is gone. + +Disabling migrations requires explicit approval of Database Maintainer. + +## Examples + +- [Disable scheduling of productivity analytics](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17253) diff --git a/doc/development/database/index.md b/doc/development/database/index.md index efc48f72d00..0363d13ed4c 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -23,14 +23,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Migrations -- [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md) +- [Migrations for multiple databases](migrations_for_multiple_databases.md) +- [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md) - [SQL guidelines](../sql.md) for working with SQL queries - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations - [Testing Rails migrations](../testing_guide/testing_migrations_guide.md) guide -- [Post deployment migrations](../post_deployment_migrations.md) -- [Background migrations](../background_migrations.md) +- [Post deployment migrations](post_deployment_migrations.md) +- [Background migrations](background_migrations.md) - [Swapping tables](../swapping_tables.md) -- [Deleting migrations](../deleting_migrations.md) +- [Deleting migrations](deleting_migrations.md) - [Partitioning tables](table_partitioning.md) ## Debugging @@ -64,6 +65,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w - [Pagination guidelines](pagination_guidelines.md) - [Pagination performance guidelines](pagination_performance_guidelines.md) - [Efficient `IN` operator queries](efficient_in_operator_queries.md) +- [Data layout and access patterns](layout_and_access_patterns.md) ## Case studies diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index 4f0b353a37f..88928feb927 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -166,7 +166,7 @@ These order objects can be defined in the model classes as normal ActiveRecord s Consider the following scope: ```ruby -scope = Issue.where(project_id: 10).order(Gitlab::Database.nulls_last_order('relative_position', 'DESC')) +scope = Issue.where(project_id: 10).order(Issue.arel_table[:relative_position].desc.nulls_last) # SELECT "issues".* FROM "issues" WHERE "issues"."project_id" = 10 ORDER BY relative_position DESC NULLS LAST scope.keyset_paginate # raises: Gitlab::Pagination::Keyset::UnsupportedScopeOrder: The order on the scope does not support keyset pagination @@ -190,8 +190,8 @@ order = Gitlab::Pagination::Keyset::Order.build([ Gitlab::Pagination::Keyset::ColumnOrderDefinition.new( attribute_name: 'relative_position', column_expression: Issue.arel_table[:relative_position], - order_expression: Gitlab::Database.nulls_last_order('relative_position', 'DESC'), - reversed_order_expression: Gitlab::Database.nulls_first_order('relative_position', 'ASC'), + order_expression: Issue.arel_table[:relative_position].desc.nulls_last, + reversed_order_expression: Issue.arel_table[:relative_position].asc.nulls_first, nullable: :nulls_last, order_direction: :desc, distinct: false diff --git a/doc/development/database/layout_and_access_patterns.md b/doc/development/database/layout_and_access_patterns.md new file mode 100644 index 00000000000..a3e2fefb2a3 --- /dev/null +++ b/doc/development/database/layout_and_access_patterns.md @@ -0,0 +1,61 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Best practices for data layout and access patterns + +Certain patterns of data access, and especially data updates, can exacerbate strain +on the database. Avoid them if possible. + +This document lists some patterns to avoid, with recommendations for alternatives. + +## High-frequency updates, especially to the same row + +Avoid single database rows that are updated by many transactions at the same time. + +- If many processes attempt to update the same row simultaneously, they queue up + as each transaction locks the row for writing. As this can significantly increase + transaction timings, the Rails connection pools can saturate, leading to + application-wide downtime. +- For each row update, PostgreSQL inserts a new row version and deletes the old one. + In high-traffic scenarios, this approach can cause vacuum and WAL (write-ahead log) + pressure, reducing database performance. + +This pattern often happens when an aggregate is too expensive to compute for each +request, so a running tally is kept in the database. If you need such an aggregate, +consider keeping a running total in a single row, plus a small working set of +recently added data, such as individual increments: + +- When introducing new data, add it to the working set. These inserts do not + cause lock contention. +- When calculating the aggregate, combine the running total with a live aggregate + from the working set, providing an up-to-date result. +- Add a periodic job that incorporates the working set into the running total and + clears it in a transaction, bounding the amount of work needed by a reader. + +## Wide tables + +PostgreSQL organizes rows into 8 KB pages, and operates on one page at a time. +By minimizing the width of rows in a table, we improve the following: + +- Sequential and bitmap index scan performance, because fewer pages must be + scanned if each contains more rows. +- Vacuum performance, because vacuum can process more rows in each page. +- Update performance, because during a (non-HOT) update, each index must be + updated for every row update. + +Mitigating wide tables is one part of the database team's +[100 GB table initiative](../../architecture/blueprints/database_scaling/size-limits.md), +as wider tables can fit fewer rows in 100 GB. + +When adding columns to a table, consider if you intend to access the data in the +new columns by itself, in a one-to-one relationship with the other columns of the +table. If so, the new columns could be a good candidate for splitting to a new table. + +Several tables have already been split in this way. For example: + +- `search_data` is split from `issues`. +- `project_pages_metadata` is split from `projects`. +- `merge_request_diff_details` is split from `merge_request_diffs` diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index 17a825b4812..2bcdc91202a 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -95,27 +95,27 @@ Created database 'gitlabhq_test_ee' Created database 'gitlabhq_geo_test_ee' Showing cross-schema foreign keys (20): - ID | HAS_LFK | FROM | TO | COLUMN | ON_DELETE - 0 | N | ci_builds | projects | project_id | cascade - 1 | N | ci_job_artifacts | projects | project_id | cascade - 2 | N | ci_pipelines | projects | project_id | cascade - 3 | Y | ci_pipelines | merge_requests | merge_request_id | cascade - 4 | N | external_pull_requests | projects | project_id | cascade - 5 | N | ci_sources_pipelines | projects | project_id | cascade - 6 | N | ci_stages | projects | project_id | cascade - 7 | N | ci_pipeline_schedules | projects | project_id | cascade - 8 | N | ci_runner_projects | projects | project_id | cascade - 9 | Y | dast_site_profiles_pipelines | ci_pipelines | ci_pipeline_id | cascade - 10 | Y | vulnerability_feedback | ci_pipelines | pipeline_id | nullify - 11 | N | ci_variables | projects | project_id | cascade - 12 | N | ci_refs | projects | project_id | cascade - 13 | N | ci_builds_metadata | projects | project_id | cascade - 14 | N | ci_subscriptions_projects | projects | downstream_project_id | cascade - 15 | N | ci_subscriptions_projects | projects | upstream_project_id | cascade - 16 | N | ci_sources_projects | projects | source_project_id | cascade - 17 | N | ci_job_token_project_scope_links | projects | source_project_id | cascade - 18 | N | ci_job_token_project_scope_links | projects | target_project_id | cascade - 19 | N | ci_project_monthly_usages | projects | project_id | cascade + ID | HAS_LFK | FROM | TO | COLUMN | ON_DELETE + 0 | N | ci_builds | projects | project_id | cascade + 1 | N | ci_job_artifacts | projects | project_id | cascade + 2 | N | ci_pipelines | projects | project_id | cascade + 3 | Y | ci_pipelines | merge_requests | merge_request_id | cascade + 4 | N | external_pull_requests | projects | project_id | cascade + 5 | N | ci_sources_pipelines | projects | project_id | cascade + 6 | N | ci_stages | projects | project_id | cascade + 7 | N | ci_pipeline_schedules | projects | project_id | cascade + 8 | N | ci_runner_projects | projects | project_id | cascade + 9 | Y | dast_site_profiles_pipelines | ci_pipelines | ci_pipeline_id | cascade + 10 | Y | vulnerability_feedback | ci_pipelines | pipeline_id | nullify + 11 | N | ci_variables | projects | project_id | cascade + 12 | N | ci_refs | projects | project_id | cascade + 13 | N | ci_builds_metadata | projects | project_id | cascade + 14 | N | ci_subscriptions_projects | projects | downstream_project_id | cascade + 15 | N | ci_subscriptions_projects | projects | upstream_project_id | cascade + 16 | N | ci_sources_projects | projects | source_project_id | cascade + 17 | N | ci_job_token_project_scope_links | projects | source_project_id | cascade + 18 | N | ci_job_token_project_scope_links | projects | target_project_id | cascade + 19 | N | ci_project_monthly_usages | projects | project_id | cascade To match FK write one or many filters to match against FROM/TO/COLUMN: - scripts/decomposition/generate-loose-foreign-key <filter(s)...> @@ -191,7 +191,7 @@ ci_pipelines: ### Track record changes To know about deletions in the `projects` table, configure a `DELETE` trigger -using a [post-deployment migration](../post_deployment_migrations.md). The +using a [post-deployment migration](post_deployment_migrations.md). The trigger needs to be configured only once. If the model already has at least one `loose_foreign_key` definition, then this step can be skipped: @@ -226,7 +226,7 @@ ON DELETE CASCADE; The migration must run after the `DELETE` trigger is installed and the loose foreign key definition is deployed. As such, it must be a [post-deployment -migration](../post_deployment_migrations.md) dated after the migration for the +migration](post_deployment_migrations.md) dated after the migration for the trigger. If the foreign key is deleted earlier, there is a good chance of introducing data inconsistency which needs manual cleanup: @@ -480,3 +480,380 @@ it executes `occurrence.pipeline.created_at`. When looping through the vulnerability occurrences in the Sidekiq worker, we could try to load the corresponding pipeline and choose to skip processing that occurrence if pipeline is not found. + +## Architecture + +The loose foreign keys feature is implemented within the `LooseForeignKeys` Ruby namespace. The +code is isolated from the core application code and theoretically, it could be a standalone library. + +The feature is invoked solely in the [`LooseForeignKeys::CleanupWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/loose_foreign_keys/cleanup_worker.rb) worker class. The worker is scheduled via a +cron job where the schedule depends on the configuration of the GitLab instance. + +- Non-decomposed GitLab (1 database): invoked every minute. +- Decomposed GitLab (2 databases, CI and Main): invoked every minute, cleaning up one database +at a time. For example, the cleanup worker for the main database runs every two minutes. + +To avoid lock contention and the processing of the same database rows, the worker does not run +parallel. This behavior is ensured with a Redis lock. + +**Record cleanup procedure:** + +1. Acquire the Redis lock. +1. Determine which database to clean up. +1. Collect all database tables where the deletions are tracked (parent tables). + - This is achieved by reading the `config/gitlab_loose_foreign_keys.yml` file. + - A table is considered "tracked" when a loose foreign key definition exists for the table and + the `DELETE` trigger is installed. +1. Cycle through the tables with an infinite loop. +1. For each table, load a batch of deleted parent records to clean up. +1. Depending on the YAML configuration, build `DELETE` or `UPDATE` (nullify) queries for the +referenced child tables. +1. Invoke the queries. +1. Repeat until all child records are cleaned up or the maximum limit is reached. +1. Remove the deleted parent records when all child records are cleaned up. + +### Database structure + +The feature relies on triggers installed on the parent tables. When a parent record is deleted, +the trigger will automatically insert a new record into the `loose_foreign_keys_deleted_records` +database table. + +The inserted record will store the following information about the deleted record: + +- `fully_qualified_table_name`: name of the database table where the record was located. +- `primary_key_value`: the ID of the record, the value will be present in the child tables as +the foreign key value. At the moment, composite primary keys are not supported, the parent table +must have an `id` column. +- `status`: defaults to pending, represents the status of the cleanup process. +- `consume_after`: defaults to the current time. +- `cleanup_attempts`: defaults to 0. The number of times the worker tried to clean up this record. +A non-zero number would mean that this record has many child records and cleaning it up requires +several runs. + +#### Database decomposition + +The `loose_foreign_keys_deleted_records` table will exist on both database servers (Ci and Main) +after the [database decomposition](https://gitlab.com/groups/gitlab-org/-/epics/6168). The worker +ill determine which parent tables belong to which database by reading the +`lib/gitlab/database/gitlab_schemas.yml` YAML file. + +Example: + +- Main database tables + - `projects` + - `namespaces` + - `merge_requests` +- Ci database tables + - `ci_builds` + - `ci_pipelines` + +When the worker is invoked for the Ci database, the worker will load deleted records only from the +`ci_builds` and `ci_pipelines` tables. During the cleanup process, `DELETE` and `UPDATE` queries +will mostly run on tables located in the Main database. In this example, one `UPDATE` query will +nullify the `merge_requests.head_pipeline_id` column. + +#### Database partitioning + +Due to the large volume of inserts the database table receives daily, a special partitioning +strategy was implemented to address data bloat concerns. Originally, the +[time-decay](https://about.gitlab.com/company/team/structure/working-groups/database-scalability/time-decay.html) +strategy was considered for the feature but due to the large data volume we decided to implement a +new strategy. + +A deleted record is considered fully processed when all its direct children records have been +cleaned up. When this happens, the loose foreign key worker will update the `status` column of +the deleted record. After this step, the record is no longer needed. + +The sliding partitioning strategy provides an efficient way of cleaning up old, unused data by +adding a new database partition and removing the old one when certain conditions are met. +The `loose_foreign_keys_deleted_records` database table is list partitioned where most of the +time there is only one partition attached to the table. + +```sql + Partitioned table "public.loose_foreign_keys_deleted_records" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +----------------------------+--------------------------+-----------+----------+----------------------------------------------------------------+----------+--------------+------------- + id | bigint | | not null | nextval('loose_foreign_keys_deleted_records_id_seq'::regclass) | plain | | + partition | bigint | | not null | 84 | plain | | + primary_key_value | bigint | | not null | | plain | | + status | smallint | | not null | 1 | plain | | + created_at | timestamp with time zone | | not null | now() | plain | | + fully_qualified_table_name | text | | not null | | extended | | + consume_after | timestamp with time zone | | | now() | plain | | + cleanup_attempts | smallint | | | 0 | plain | | +Partition key: LIST (partition) +Indexes: + "loose_foreign_keys_deleted_records_pkey" PRIMARY KEY, btree (partition, id) + "index_loose_foreign_keys_deleted_records_for_partitioned_query" btree (partition, fully_qualified_table_name, consume_after, id) WHERE status = 1 +Check constraints: + "check_1a541f3235" CHECK (char_length(fully_qualified_table_name) <= 150) +Partitions: gitlab_partitions_dynamic.loose_foreign_keys_deleted_records_84 FOR VALUES IN ('84') +``` + +The `partition` column controls the insert direction, the `partition` value determines which +partition will get the deleted rows inserted via the trigger. Notice that the default value of +the `partition` table matches with the value of the list partition (84). In `INSERT` query +within the trigger thevalue of the `partition` is omitted, the trigger always relies on the +default value of the column. + +Example `INSERT` query for the trigger: + +```sql +INSERT INTO loose_foreign_keys_deleted_records +(fully_qualified_table_name, primary_key_value) +SELECT TG_TABLE_SCHEMA || '.' || TG_TABLE_NAME, old_table.id FROM old_table; +``` + +The partition "sliding" process is controlled by two, regularly executed callbacks. These +callbackes are defined within the `LooseForeignKeys::DeletedRecord` model. + +The `next_partition_if` callback controls when to create a new partition. A new partition will +be created when the current partition has at least one record older than 24 hours. A new partition +is added by the [`PartitionManager`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/partitioning/partition_manager.rb) +using the following steps: + +1. Create a new partition, where the `VALUE` for the partition is `CURRENT_PARTITION + 1`. +1. Update the default value of the `partition` column to `CURRENT_PARTITION + 1`. + +With these steps, new `INSERT`-s via the triggers will end up in the new partition. At this point, +the database table has two partitions. + +The `detach_partition_if` callback determines if the old partitions can be detached from the table. +A partition is detachable if there are no pending (unprocessed) records in the partition +(`status = 1`). The detached partitions will be available for some time, you can see the list +detached partitions in the `detached_partitions` table: + +```sql +select * from detached_partitions; +``` + +#### Cleanup queries + +The `LooseForeignKeys::CleanupWorker` has its database query builder which depends on `Arel`. +The feature doesn't reference any application-specific `ActiveRecord` models to avoid unexpected +side effects. The database queries are batched, which means that several parent records are being +cleaned up at the same time. + +Example `DELETE` query: + +```sql +DELETE +FROM "merge_request_metrics" +WHERE ("merge_request_metrics"."id") IN + (SELECT "merge_request_metrics"."id" + FROM "merge_request_metrics" + WHERE "merge_request_metrics"."pipeline_id" IN (1, 2, 10, 20) + LIMIT 1000 FOR UPDATE SKIP LOCKED) +``` + +The primary key values of the parent records are 1, 2, 10, and 20. + +Example `UPDATE` (nullify) query: + +```sql +UPDATE "merge_requests" +SET "head_pipeline_id" = NULL +WHERE ("merge_requests"."id") IN + (SELECT "merge_requests"."id" + FROM "merge_requests" + WHERE "merge_requests"."head_pipeline_id" IN (3, 4, 30, 40) + LIMIT 500 FOR UPDATE SKIP LOCKED) +``` + +These queries are batched, which means that in many cases, several invocations are needed to clean +up all associated child records. + +The batching is implemented with loops, the processing will stop when all associated child records +are cleaned up or the limit is reached. + +```ruby +loop do + modification_count = process_batch_with_skip_locked + + break if modification_count == 0 || over_limit? +end + +loop do + modification_count = process_batch + + break if modification_count == 0 || over_limit? +end +``` + +The loop-based batch processing is preferred over `EachBatch` for the following reasons: + +- The records in the batch are modified, so the next batch will contain different records. +- There is always an index on the foreign key column however, the column is usually not unique. +`EachBatch` requires a unique column for the iteration. +- The record order doesn't matter for the cleanup. + +Notice that we have two loops. The initial loop will process records with the `SKIP LOCKED` clause. +The query will skip rows that are locked by other application processes. This will ensure that the +cleanup worker will less likely to become blocked. The second loop will execute the database +queries without `SKIP LOCKED` to ensure that all records have been processed. + +#### Processing limits + +A constant, large volume of record updates or deletions can cause incidents and affect the +availability of GitLab: + +- Increased table bloat. +- Increased number of pending WAL files. +- Busy tables, difficulty when acquiring locks. + +To mitigate these issues, several limits are applied when the worker runs. + +- Each query has `LIMIT`, a query cannot process an unbounded number of rows. +- The maximum number of record deletions and record updates is limited. +- The maximum runtime (30 seconds) for the database queries is limited. + +The limit rules are implemented in the `LooseForeignKeys::ModificationTracker` class. When one of +the limits (record modification count, time limit) is reached the processing is stopped +immediately. After some time, the next scheduled worker will continue the cleanup process. + +#### Performance characteristics + +The database trigger on the parent tables will **decrease** the record deletion speed. Each +statement that removes rows from the parent table will invoke the trigger to insert records +into the `loose_foreign_keys_deleted_records` table. + +The queries within the cleanup worker are fairly efficient index scans, with limits in place +they're unlikely to affect other parts of the application. + +The database queries are not running in transaction, when an error happens for example a statement +timeout or a worker crash, the next job will continue the processing. + +## Troubleshooting + +### Accumulation of deleted records + +There can be cases where the workers need to process an unusually large amount of data. This can +happen under normal usage, for example when a large project or group is deleted. In this scenario, +there can be several million rows to be deleted or nullified. Due to the limits enforced by the +worker, processing this data will take some time. + +When cleaning up "heavy-hitters", the feature ensures fair processing by rescheduling larger +batches for later. This gives time for other deleted records to be processed. + +For example, a project with millions of `ci_builds` records is deleted. The `ci_builds` records +will be deleted by the loose foreign keys feature. + +1. The cleanup worker is scheduled and picks up a batch of deleted `projects` records. The large +project is part of the batch. +1. Deletion of the orphaned `ci_builds` rows has started. +1. The time limit is reached, but the cleanup is not complete. +1. The `cleanup_attempts` column is incremented for the deleted records. +1. Go to step 1. The next cleanup worker continues the cleanup. +1. When the `cleanup_attempts` reaches 3, the batch is re-scheduled 10 minutes later by updating +the `consume_after` column. +1. The next cleanup worker will process a different batch. + +We have Prometheus metrics in place to monitor the deleted record cleanup: + +- `loose_foreign_key_processed_deleted_records`: Number of processed deleted records. When large +cleanup happens, this number would decrease. +- `loose_foreign_key_incremented_deleted_records`: Number of deleted records which were not +finished processing. The `cleanup_attempts` column was incremented. +- `loose_foreign_key_rescheduled_deleted_records`: Number of deleted records that had to be +rescheduled at a later time after 3 cleanup attempts. + +Example Thanos query: + +```plaintext +loose_foreign_key_rescheduled_deleted_records{env="gprd", table="ci_runners"} +``` + +Another way to look at the situation is by running a database query. This query gives the exact +counts of the unprocessed records: + +```sql +SELECT partition, fully_qualified_table_name, count(*) +FROM loose_foreign_keys_deleted_records +WHERE +status = 1 +GROUP BY 1, 2; +``` + +Example output: + +```sql + partition | fully_qualified_table_name | count +-----------+----------------------------+------- + 87 | public.ci_builds | 874 + 87 | public.ci_job_artifacts | 6658 + 87 | public.ci_pipelines | 102 + 87 | public.ci_runners | 111 + 87 | public.merge_requests | 255 + 87 | public.namespaces | 25 + 87 | public.projects | 6 +``` + +The query includes the partition number which can be useful to detect if the cleanup process is +significantly lagging behind. When multiple different partition values are present in the list +that means the cleanup of some deleted records didn't finish in several days (1 new partition +is added every day). + +Steps to diagnose the problem: + +- Check which records are accumulating. +- Try to get an estimate of the number of remaining records. +- Looking into the worker performance stats (Kibana or Thanos). + +Possible solutions: + +- Short-term: increase the batch sizes. +- Long-term: invoke the worker more frequently. Parallelize the worker + +For a one-time fix, we can run the cleanup worker several times from the rails console. The worker +can run parallelly however, this can introduce lock contention and it could increase the worker +runtime. + +```ruby +LooseForeignKeys::CleanupWorker.new.perform +``` + +When the cleanup is done, the older partitions will be automatically detached by the +`PartitionManager`. + +### PartitionManager bug + +NOTE: +This issue happened in the past on Staging and it has been mitigated. + +When adding a new partition, the default value of the `partition` column is also updated. This is +a schema change that is executed in the same transaction as the new partition creation. It's highly +unlikely that the `partition` column goes outdated. + +However, if this happens then this can cause application-wide incidents because the `partition` +value points to a partition that doesn't exist. Symptom: deletion of records from tables where the +`DELETE` trigger is installed fails. + +```sql +\d+ loose_foreign_keys_deleted_records; + + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +----------------------------+--------------------------+-----------+----------+----------------------------------------------------------------+----------+--------------+------------- + id | bigint | | not null | nextval('loose_foreign_keys_deleted_records_id_seq'::regclass) | plain | | + partition | bigint | | not null | 4 | plain | | + primary_key_value | bigint | | not null | | plain | | + status | smallint | | not null | 1 | plain | | + created_at | timestamp with time zone | | not null | now() | plain | | + fully_qualified_table_name | text | | not null | | extended | | + consume_after | timestamp with time zone | | | now() | plain | | + cleanup_attempts | smallint | | | 0 | plain | | +Partition key: LIST (partition) +Indexes: + "loose_foreign_keys_deleted_records_pkey" PRIMARY KEY, btree (partition, id) + "index_loose_foreign_keys_deleted_records_for_partitioned_query" btree (partition, fully_qualified_table_name, consume_after, id) WHERE status = 1 +Check constraints: + "check_1a541f3235" CHECK (char_length(fully_qualified_table_name) <= 150) +Partitions: gitlab_partitions_dynamic.loose_foreign_keys_deleted_records_3 FOR VALUES IN ('3') +``` + +Check the default value of the `partition` column and compare it with the available partitions +(4 vs 3). The partition with the value of 4 does not exist. To mitigate the problem an emergency +schema change is required: + +```sql +ALTER TABLE loose_foreign_keys_deleted_records ALTER COLUMN partition SET DEFAULT 3; +``` diff --git a/doc/development/database/migrations_for_multiple_databases.md b/doc/development/database/migrations_for_multiple_databases.md new file mode 100644 index 00000000000..0ec4612e985 --- /dev/null +++ b/doc/development/database/migrations_for_multiple_databases.md @@ -0,0 +1,390 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Migrations for Multiple databases + +> Support for describing migration purposes was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73756) in GitLab 14.8. + +This document describes how to properly write database migrations +for [the decomposed GitLab application using multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). + +Learn more about general multiple databases support in a [separate document](multiple_databases.md). + +WARNING: +If you experience any issues using `Gitlab::Database::Migration[2.0]`, +you can temporarily revert back to the previous behavior by changing the version to `Gitlab::Database::Migration[1.0]`. +Please report any issues with `Gitlab::Database::Migration[2.0]` in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/358430). + +The design for multiple databases (except for the Geo database) assumes +that all decomposed databases have **the same structure** (for example, schema), but **the data is different** in each database. This means that some tables do not contain data on each database. + +## Operations + +Depending on the used constructs, we can classify migrations to be either: + +1. Modifying structure ([DDL - Data Definition Language](https://www.postgresql.org/docs/current/ddl.html)) (for example, `ALTER TABLE`). +1. Modifying data ([DML - Data Manipulation Language](https://www.postgresql.org/docs/current/dml.html)) (for example, `UPDATE`). +1. Performing [other queries](https://www.postgresql.org/docs/current/queries.html) (for example, `SELECT`) that are treated as **DML** for the purposes of our migrations. + +**The usage of `Gitlab::Database::Migration[2.0]` requires migrations to always be of a single purpose**. +Migrations cannot mix **DDL** and **DML** changes as the application requires the structure +(as described by `db/structure.sql`) to be exactly the same across all decomposed databases. + +### Data Definition Language (DDL) + +The DDL migrations are all migrations that: + +1. Create or drop a table (for example, `create_table`). +1. Add or remove an index (for example, `add_index`, `add_index_concurrently`). +1. Add or remove a foreign key (for example `add_foreign_key`, `add_foreign_key_concurrently`). +1. Add or remove a column with or without a default value (for example, `add_column`). +1. Create or drop trigger functions (for example, `create_trigger_function`). +1. Attach or detach triggers from tables (for example, `track_record_deletions`, `untrack_record_deletions`). +1. Prepare or not async indexes (for example, `prepare_async_index`, `unprepare_async_index_by_name`). + +As such DDL migrations **CANNOT**: + +1. Read or modify data in any form, via SQL statements or ActiveRecord models. +1. Update column values (for example, `update_column_in_batches`). +1. Schedule background migrations (for example, `queue_background_migration_jobs_by_range_at_intervals`). +1. Read the state of feature flags since they are stored in `main:` (a `features` and `feature_gates`). +1. Read application settings (as settings are stored in `main:`). + +As the majority of migrations in the GitLab codebase are of the DDL-type, +this is also the default mode of operation and requires no further changes +to the migrations files. + +#### Example: perform DDL on all databases + +Example migration adding a concurrent index that is treated as change of the structure (DDL) +that is executed on all configured databases. + +```ruby +class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + INDEX_NAME = 'index_on_merge_request_reviewers_user_id_and_state' + + def up + add_concurrent_index :merge_request_reviewers, [:user_id, :state], where: 'state = 2', name: INDEX_NAME + end + + def down + remove_concurrent_index_by_name :merge_request_reviewers, INDEX_NAME + end +end +``` + +### Data Manipulation Language (DML) + +The DML migrations are all migrations that: + +1. Read data via SQL statements (for example, `SELECT * FROM projects WHERE id=1`). +1. Read data via ActiveRecord models (for example, `User < MigrationRecord`). +1. Create, update or delete data via ActiveRecord models (for example, `User.create!(...)`). +1. Create, update or delete data via SQL statements (for example, `DELETE FROM projects WHERE id=1`). +1. Update columns in batches (for example, `update_column_in_batches(:projects, :archived, true)`). +1. Schedule background migrations (for example, `queue_background_migration_jobs_by_range_at_intervals`). +1. Access application settings (for example, `ApplicationSetting.last` if run for `main:` database). +1. Read and modify feature flags if run for the `main:` database. + +The DML migrations **CANNOT**: + +1. Make any changes to DDL since this breaks the rule of keeping `structure.sql` coherent across + all decomposed databases. +1. **Read data from another database**. + +To indicate the `DML` migration type, a migration must use the `restrict_gitlab_migration gitlab_schema:` +syntax in a migration class. This marks the given migration as DML and restricts access to it. + +#### Example: perform DML only in context of the database containing the given `gitlab_schema` + +Example migration updating `archived` column of `projects` that is executed +only for the database containing `gitlab_main` schema. + +```ruby +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + restrict_gitlab_migration gitlab_schema: :gitlab_main + + def up + update_column_in_batches(:projects, :archived, true) do |table, query| + query.where(table[:archived].eq(false)) # rubocop:disable CodeReuse/ActiveRecord + end + end + + def down + # no-op + end +end +``` + +#### Example: usage of `ActiveRecord` classes + +A migration using `ActiveRecord` class to perform data manipulation +must use the `MigrationRecord` class. This class is guaranteed to provide +a correct connection in a context of a given migration. + +Underneath the `MigrationRecord == ActiveRecord::Base`, as once the `db:migrate` +runs, it switches the active connection of `ActiveRecord::Base.establish_connection :ci`. +To avoid confusion to using the `ActiveRecord::Base`, `MigrationRecord` is required. + +This implies that DML migrations are forbidden to read data from other +databases. For example, running migration in context of `ci:` and reading feature flags +from `main:`, as no established connection to another database is present. + +```ruby +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + restrict_gitlab_migration gitlab_schema: :gitlab_main + + class Project < MigrationRecord + end + + def up + Project.where(archived: false).each_batch of |batch| + batch.update_all(archived: true) + end + end + + def down + end +end +``` + +### The special purpose of `gitlab_shared` + +As described in [gitlab_schema](multiple_databases.md#the-special-purpose-of-gitlab_shared), +the `gitlab_shared` tables are allowed to contain data across all databases. This implies +that such migrations should run across all databases to modify structure (DDL) or modify data (DML). + +As such migrations accessing `gitlab_shared` do not need to use `restrict_gitlab_migration gitlab_schema:`, +migrations without restriction run across all databases and are allowed to modify data on each of them. +If the `restrict_gitlab_migration gitlab_schema:` is specified, the `DML` migration +runs only in a context of a database containing the given `gitlab_schema`. + +#### Example: run DML `gitlab_shared` migration on all databases + +Example migration updating `loose_foreign_keys_deleted_records` table +that is marked in `lib/gitlab/database/gitlab_schemas.yml` as `gitlab_shared`. + +This migration is executed across all configured databases. + +```ruby +class DeleteAllLooseForeignKeyRecords < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + def up + execute("DELETE FROM loose_foreign_keys_deleted_records") + end + + def down + # no-op + end +end +``` + +#### Example: run DML `gitlab_shared` only on the database containing the given `gitlab_schema` + +Example migration updating `loose_foreign_keys_deleted_records` table +that is marked in `lib/gitlab/database/gitlab_schemas.yml` as `gitlab_shared`. + +This migration since it configures restriction on `gitlab_ci` is executed only +in context of database containing `gitlab_ci` schema. + +```ruby +class DeleteCiBuildsLooseForeignKeyRecords < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + restrict_gitlab_migration gitlab_schema: :gitlab_ci + + def up + execute("DELETE FROM loose_foreign_keys_deleted_records WHERE fully_qualified_table_name='ci_builds'") + end + + def down + # no-op + end +end +``` + +### The behavior of skipping migrations + +The only migrations that are skipped are the ones performing **DML** changes. +The **DDL** migrations are **always and unconditionally** executed. + +The implemented [solution](https://gitlab.com/gitlab-org/gitlab/-/issues/355014#solution-2-use-database_tasks) +uses the `database_tasks:` as a way to indicate which additional database configurations +(in `config/database.yml`) share the same primary database. The database configurations +marked with `database_tasks: false` are exempt from executing `db:migrate` for those +database configurations. + +If database configurations do not share databases (all do have `database_tasks: true`), +each migration runs for every database configuration: + +1. The DDL migration applies all structure changes on all databases. +1. The DML migration runs only in the context of a database containing the given `gitlab_schema:`. +1. If the DML migration is not eligible to run, it is skipped. It's still + marked as executed in `schema_migrations`. While running `db:migrate`, the skipped + migration outputs `Current migration is skipped since it modifies 'gitlab_ci' which is outside of 'gitlab_main, gitlab_shared`. + +To prevent loss of migrations if the `database_tasks: false` is configured, a dedicated +Rake task is used [`gitlab:db:validate_config`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83118). +The `gitlab:db:validate_config` validates the correctness of `database_tasks:` by checking database identifiers +of each underlying database configuration. The ones that share the database are required to have +the `database_tasks: false` set. `gitlab:db:validate_config` always runs before `db:migrate`. + +## Validation + +Validation in a nutshell uses [pg_query](https://github.com/pganalyze/pg_query) to analyze +each query and classify tables with information from [`gitlab_schema.yml`](multiple_databases.md#gitlab-schema). +The migration is skipped if the specified `gitlab_schema` is outside of a list of schemas +managed by a given database connection (`Gitlab::Database::gitlab_schemas_for_connection`). + +The `Gitlab::Database::Migration[2.0]` includes `Gitlab::Database::MigrationHelpers::RestrictGitlabSchema` +which extends the `#migrate` method. For the duration of a migration a dedicated query analyzer +is installed `Gitlab::Database::QueryAnalyzers::RestrictAllowedSchemas` that accepts +a list of allowed schemas as defined by `restrict_gitlab_migration:`. If the executed query +is outside of allowed schemas, it raises an exception. + +## Exceptions + +Depending on misuse or lack of `restrict_gitlab_migration` various exceptions can be raised +as part of the migration run and prevent the migration from being completed. + +### Exception 1: migration running in DDL mode does DML select + +```ruby +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + # Missing: + # restrict_gitlab_migration gitlab_schema: :gitlab_main + + def up + update_column_in_batches(:projects, :archived, true) do |table, query| + query.where(table[:archived].eq(false)) # rubocop:disable CodeReuse/ActiveRecord + end + end + + def down + # no-op + end +end +``` + +```plaintext +Select/DML queries (SELECT/UPDATE/DELETE) are disallowed in the DDL (structure) mode +Modifying of 'projects' (gitlab_main) with 'SELECT * FROM projects... +``` + +The current migration do not use `restrict_gitlab_migration`. The lack indicates a migration +running in **DDL** mode, but the executed payload appears to be reading data from `projects`. + +**The solution** is to add `restrict_gitlab_migration gitlab_schema: :gitlab_main`. + +### Exception 2: migration running in DML mode changes the structure + +```ruby +class AddUserIdAndStateIndexToMergeRequestReviewers < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + # restrict_gitlab_migration if defined indicates DML, it should be removed + restrict_gitlab_migration gitlab_schema: :gitlab_main + + INDEX_NAME = 'index_on_merge_request_reviewers_user_id_and_state' + + def up + add_concurrent_index :merge_request_reviewers, [:user_id, :state], where: 'state = 2', name: INDEX_NAME + end + + def down + remove_concurrent_index_by_name :merge_request_reviewers, INDEX_NAME + end +end +``` + +```plaintext +DDL queries (structure) are disallowed in the Select/DML (SELECT/UPDATE/DELETE) mode. +Modifying of 'merge_request_reviewers' with 'CREATE INDEX... +``` + +The current migration do use `restrict_gitlab_migration`. The presence indicates **DML** mode, +but the executed payload appears to be doing structure changes (DDL). + +**The solution** is to remove `restrict_gitlab_migration gitlab_schema: :gitlab_main`. + +### Exception 3: migration running in DML mode accesses data from a table in another schema + +```ruby +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + # Since it modifies `projects` it should use `gitlab_main` + restrict_gitlab_migration gitlab_schema: :gitlab_ci + + def up + update_column_in_batches(:projects, :archived, true) do |table, query| + query.where(table[:archived].eq(false)) # rubocop:disable CodeReuse/ActiveRecord + end + end + + def down + # no-op + end +end +``` + +```plaintext +Select/DML queries (SELECT/UPDATE/DELETE) do access 'projects' (gitlab_main) " \ +which is outside of list of allowed schemas: 'gitlab_ci' +``` + +The current migration do restrict the migration to `gitlab_ci`, but appears to modify +data in `gitlab_main`. + +**The solution** is to change `restrict_gitlab_migration gitlab_schema: :gitlab_ci`. + +### Exception 4: mixing DDL and DML mode + +```ruby +class UpdateProjectsArchivedState < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + # This migration is invalid regardless of specification + # as it cannot modify structure and data at the same time + restrict_gitlab_migration gitlab_schema: :gitlab_ci + + def up + add_concurrent_index :merge_request_reviewers, [:user_id, :state], where: 'state = 2', name: 'index_on_merge_request_reviewers' + update_column_in_batches(:projects, :archived, true) do |table, query| + query.where(table[:archived].eq(false)) # rubocop:disable CodeReuse/ActiveRecord + end + end + + def down + # no-op + end +end +``` + +The migrations mixing **DDL** and **DML** depending on ordering of operations raises +one of the prior exceptions. + +## Upcoming changes on multiple database migrations + +The `restrict_gitlab_migration` using `gitlab_schema:` is considered as a first iteration +of this feature for running migrations selectively depending on a context. It is possible +to add additional restrictions to DML-only migrations (as the structure coherency is likely +to stay as-is until further notice) to restrict when they run. + +A Potential extension is to limit running DML migration only to specific environments: + +```ruby +restrict_gitlab_migration gitlab_schema: :gitlab_main, gitlab_env: :gitlab_com +``` diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index c9bbf73be55..3b1b06b557c 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -9,141 +9,86 @@ info: To determine the technical writer assigned to the Stage/Group associated w To scale GitLab, the we are [decomposing the GitLab application database into multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). -## CI/CD Database +## GitLab Schema -> Support for configuring the GitLab Rails application to use a distinct -database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) -in GitLab 14.1. This feature is still under development, and is not ready for production use. +For properly discovering allowed patterns between different databases +the GitLab application implements the `lib/gitlab/database/gitlab_schemas.yml` YAML file. -### Development setup +This file provides a virtual classification of tables into a `gitlab_schema` +which conceptually is similar to [PostgreSQL Schema](https://www.postgresql.org/docs/current/ddl-schemas.html). +We decided as part of [using database schemas to better isolated CI decomposed features](https://gitlab.com/gitlab-org/gitlab/-/issues/333415) +that we cannot use PostgreSQL schema due to complex migration procedures. Instead we implemented +the concept of application-level classification. +Each table of GitLab needs to have a `gitlab_schema` assigned: -By default, GitLab is configured to use only one main database. To -opt-in to use a main database, and CI database, modify the -`config/database.yml` file to have a `main` and a `ci` database -configurations. +- `gitlab_main`: describes all tables that are being stored in the `main:` database (for example, like `projects`, `users`). +- `gitlab_ci`: describes all CI tables that are being stored in the `ci:` database (for example, `ci_pipelines`, `ci_builds`). +- `gitlab_shared`: describe all application tables that contain data across all decomposed databases (for example, `loose_foreign_keys_deleted_records`). +- `...`: more schemas to be introduced with additional decomposed databases -You can set this up using [GDK](#gdk-configuration) or by -[manually configuring `config/database.yml`](#manually-set-up-the-cicd-database). +The usage of schema enforces the base class to be used: -#### GDK configuration +- `ApplicationRecord` for `gitlab_main` +- `Ci::ApplicationRecord` for `gitlab_ci` +- `Gitlab::Database::SharedModel` for `gitlab_shared` -If you are using GDK, you can follow the following steps: +### The impact of `gitlab_schema` -1. On the GDK root directory, run: +The usage of `gitlab_schema` has a significant impact on the application. +The `gitlab_schema` primary purpose is to introduce a barrier between different data access patterns. - ```shell - gdk config set gitlab.rails.databases.ci.enabled true - ``` +This is used as a primary source of classification for: -1. Open your `gdk.yml`, and confirm that it has the following lines: +- [Discovering cross-joins across tables from different schemas](#removing-joins-between-ci_-and-non-ci_-tables) +- [Discovering cross-database transactions across tables from different schemas](#removing-cross-database-transactions) - ```yaml - gitlab: - rails: - databases: - ci: - enabled: true - ``` +### The special purpose of `gitlab_shared` -1. Reconfigure GDK: +`gitlab_shared` is a special case describing tables or views that by design contain data across +all decomposed databases. This does describe application-defined tables (like `loose_foreign_keys_deleted_records`), +Rails-defined tables (like `schema_migrations` or `ar_internal_metadata` as well as internal PostgreSQL tables +(for example, `pg_attribute`). - ```shell - gdk reconfigure - ``` +**Be careful** to use `gitlab_shared` as it requires special handling while accessing data. +Since `gitlab_shared` shares not only structure but also data, the application needs to be written in a way +that traverses all data from all databases in sequential manner. -1. [Create the new CI/CD database](#create-the-new-database). +```ruby +Gitlab::Database::EachDatabase.each_model_connection([MySharedModel]) do |connection, connection_name| + MySharedModel.select_all_data... +end +``` -#### Manually set up the CI/CD database +As such, migrations modifying data of `gitlab_shared` tables are expected to run across +all decomposed databases. -You can manually edit `config/database.yml` to split the databases. -To do so, consider a `config/database.yml` file like the example below: +## Migrations -```yaml -development: - main: - adapter: postgresql - encoding: unicode - database: gitlabhq_development - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s - -test: &test - main: - adapter: postgresql - encoding: unicode - database: gitlabhq_test - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s -``` +Read [Migrations for Multiple Databases](migrations_for_multiple_databases.md). -Edit it to split the databases into `main` and `ci`: +## CI/CD Database -```yaml -development: - main: - adapter: postgresql - encoding: unicode - database: gitlabhq_development - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s - ci: - adapter: postgresql - encoding: unicode - database: gitlabhq_development_ci - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s - -test: &test - main: - adapter: postgresql - encoding: unicode - database: gitlabhq_test - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s - ci: - adapter: postgresql - encoding: unicode - database: gitlabhq_test_ci - host: /path/to/gdk/postgresql - pool: 10 - prepared_statements: false - variables: - statement_timeout: 120s -``` +> Support for configuring the GitLab Rails application to use a distinct +database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) +in GitLab 14.1. This feature is still under development, and is not ready for production use. -Next, [create the new CI/CD database](#create-the-new-database). +### Configure single database -#### Create the new database +By default, GDK is configured to run with multiple databases. To configure GDK to use a single database: -After configuring GitLab for the two databases, create the new CI/CD database: +1. On the GDK root directory, run: -1. Create the new `ci:` database, load the DB schema into the `ci:` database, - and run any pending migrations: + ```shell + gdk config set gitlab.rails.databases.ci.enabled false + ``` - ```shell - bundle exec rails db:create db:schema:load:ci db:migrate - ``` +1. Reconfigure GDK: -1. Restart GDK: + ```shell + gdk reconfigure + ``` - ```shell - gdk restart - ``` +To switch back to using multiple databases, set `gitlab.rails.databases.ci.enabled` to `true` and run `gdk reconfigure`. <!-- NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references @@ -167,9 +112,9 @@ already many such examples that need to be fixed in The following steps are the process to remove cross-database joins between `ci_*` and non `ci_*` tables: -1. **{check-circle}** Add all failing specs to the [`cross-join-allowlist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/f5de89daeb468fc45e1e95a76d1b5297aa53da11/spec/support/database/cross-join-allowlist.yml) +1. **{check-circle}** Add all failing specs to the [`cross-join-allowlist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/database/cross-join-allowlist.yml) file. -1. **{dotted-circle}** Find the code that caused the spec failure and wrap the isolated code +1. **{check-circle}** Find the code that caused the spec failure and wrap the isolated code in [`allow_cross_joins_across_databases`](#allowlist-for-existing-cross-joins). Link to a new issue assigned to the correct team to remove the specs from the `cross-join-allowlist.yml` file. diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index de070f7e434..af7d569e282 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -197,7 +197,7 @@ end If you have to clean up a nullable column for a [high-traffic table](../migration_style_guide.md#high-traffic-tables) (for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and -it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up) +it will need an additional [background migration cleaning up](background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you will need 3 releases end-to-end: diff --git a/doc/development/database/post_deployment_migrations.md b/doc/development/database/post_deployment_migrations.md new file mode 100644 index 00000000000..799eefdb875 --- /dev/null +++ b/doc/development/database/post_deployment_migrations.md @@ -0,0 +1,81 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Post Deployment Migrations + +Post deployment migrations are regular Rails migrations that can optionally be +executed after a deployment. By default these migrations are executed alongside +the other migrations. To skip these migrations you must set the +environment variable `SKIP_POST_DEPLOYMENT_MIGRATIONS` to a non-empty value +when running `rake db:migrate`. + +For example, this would run all migrations including any post deployment +migrations: + +```shell +bundle exec rake db:migrate +``` + +This however skips post deployment migrations: + +```shell +SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rake db:migrate +``` + +## Deployment Integration + +Say you're using Chef for deploying new versions of GitLab and you'd like to run +post deployment migrations after deploying a new version. Let's assume you +normally use the command `chef-client` to do so. To make use of this feature +you'd have to run this command as follows: + +```shell +SKIP_POST_DEPLOYMENT_MIGRATIONS=true sudo chef-client +``` + +Once all servers have been updated you can run `chef-client` again on a single +server _without_ the environment variable. + +The process is similar for other deployment techniques: first you would deploy +with the environment variable set, then you re-deploy a single +server but with the variable _unset_. + +## Creating Migrations + +To create a post deployment migration you can use the following Rails generator: + +```shell +bundle exec rails g post_deployment_migration migration_name_here +``` + +This generates the migration file in `db/post_migrate`. These migrations +behave exactly like regular Rails migrations. + +## Use Cases + +Post deployment migrations can be used to perform migrations that mutate state +that an existing version of GitLab depends on. For example, say you want to +remove a column from a table. This requires downtime as a GitLab instance +depends on this column being present while it's running. Normally you'd follow +these steps in such a case: + +1. Stop the GitLab instance +1. Run the migration removing the column +1. Start the GitLab instance again + +Using post deployment migrations we can instead follow these steps: + +1. Deploy a new version of GitLab while ignoring post deployment migrations +1. Re-run `rake db:migrate` but without the environment variable set + +Here we don't need any downtime as the migration takes place _after_ a new +version (which doesn't depend on the column anymore) has been deployed. + +Some other examples where these migrations are useful: + +- Cleaning up data generated due to a bug in GitLab +- Removing tables +- Migrating jobs from one Sidekiq queue to another diff --git a/doc/development/database/rename_database_tables.md b/doc/development/database/rename_database_tables.md index 881adf00ad0..7a76c028042 100644 --- a/doc/development/database/rename_database_tables.md +++ b/doc/development/database/rename_database_tables.md @@ -82,7 +82,7 @@ when naming indexes, so there is a possibility that not all indexes are properly the migration locally, check if there are inconsistently named indexes (`db/structure.sql`). Those can be renamed manually in a separate migration, which can be also part of the release M.N+1. - Foreign key columns might still contain the old table name. For smaller tables, follow our [standard column -rename process](../avoiding_downtime_in_migrations.md#renaming-columns) +rename process](avoiding_downtime_in_migrations.md#renaming-columns) - Avoid renaming database tables which are using with triggers. - Table modifications (add or remove columns) are not allowed during the rename process, please make sure that all changes to the table happen before the rename migration is started (or in the next release). - As the index names might change, verify that the model does not use bulk insert diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 9674deb4603..4ed7cf1b4de 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -229,7 +229,7 @@ end To keep this guide short, we skipped the definition of the background migration and only provided a high level example of the post-deployment migration that is used to schedule the batches. -You can find more information on the guide about [background migrations](../background_migrations.md) +You can find more information on the guide about [background migrations](background_migrations.md) #### Validate the text limit (next release) @@ -277,7 +277,7 @@ end If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3) (for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and -it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up) +it will need an additional [background migration cleaning up](background_migrations.md#cleaning-up) in the release after adding the data migration. In that rare case you will need 3 releases end-to-end: diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md index 5319c73aad0..ec768136404 100644 --- a/doc/development/database/table_partitioning.md +++ b/doc/development/database/table_partitioning.md @@ -214,7 +214,7 @@ end ``` This step uses the same mechanism as any background migration, so you -may want to read the [Background Migration](../background_migrations.md) +may want to read the [Background Migration](background_migrations.md) guide for details on that process. Background jobs are scheduled every 2 minutes and copy `50_000` records at a time, which can be used to estimate the timing of the background migration portion of the diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 4b5845992b9..fd0e2e17623 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -125,7 +125,7 @@ the following preparations into account. test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slack channel and add the execution time to the MR description: - Execution time largely varies between `#database-lab` and GitLab.com, but an elevated execution time from `#database-lab` can give a hint that the execution on GitLab.com will also be considerably high. - - If the execution from `#database-lab` is longer than `1h`, the index should be moved to a [post-migration](post_deployment_migrations.md). + - If the execution from `#database-lab` is longer than `1h`, the index should be moved to a [post-migration](database/post_deployment_migrations.md). Keep in mind that in this case you may need to split the migration and the application changes in separate releases to ensure the index will be in place when the code that needs it will be deployed. - Manually trigger the [database testing](database/database_migration_pipeline.md) job (`db:gitlabcom-database-testing`) in the `test` stage. @@ -212,7 +212,7 @@ Include in the MR description: #### Preparation when removing columns, tables, indexes, or other structures -- Follow the [guidelines on dropping columns](avoiding_downtime_in_migrations.md#dropping-columns). +- Follow the [guidelines on dropping columns](database/avoiding_downtime_in_migrations.md#dropping-columns). - Generally it's best practice (but not a hard rule) to remove indexes and foreign keys in a post-deployment migration. - Exceptions include removing indexes and foreign keys for small tables. - If you're adding a composite index, another index might become redundant, so remove that in the same migration. @@ -222,6 +222,7 @@ Include in the MR description: - Check migrations - Review relational modeling and design choices + - Consider [access patterns and data layout](database/layout_and_access_patterns.md) if new tables or columns are added. - Review migrations follow [database migration style guide](migration_style_guide.md), for example - [Check ordering of columns](ordering_table_columns.md) @@ -235,8 +236,8 @@ Include in the MR description: - Check that the relevant version files under `db/schema_migrations` were added or removed. - Check queries timing (If any): In a single transaction, cumulative query time executed in a migration needs to fit comfortably within `15s` - preferably much less than that - on GitLab.com. - - For column removals, make sure the column has been [ignored in a previous release](avoiding_downtime_in_migrations.md#dropping-columns) -- Check [background migrations](background_migrations.md): + - For column removals, make sure the column has been [ignored in a previous release](database/avoiding_downtime_in_migrations.md#dropping-columns) +- Check [background migrations](database/background_migrations.md): - Establish a time estimate for execution on GitLab.com. For historical purposes, it's highly recommended to include this estimation on the merge request description. - If a single `update` is below than `1s` the query can be placed @@ -249,6 +250,8 @@ Include in the MR description: it's suggested to treat background migrations as post migrations: place them in `db/post_migrate` instead of `db/migrate`. Keep in mind that post migrations are executed post-deployment in production. + - If a migration [has tracking enabled](database/background_migrations.md#background-jobs-tracking), + ensure `mark_all_as_succeeded` is called even if no work is done. - Check [timing guidelines for migrations](migration_style_guide.md#how-long-a-migration-should-take) - Check migrations are reversible and implement a `#down` method - Check new table migrations: diff --git a/doc/development/deleting_migrations.md b/doc/development/deleting_migrations.md index 25ec1c08335..5d5ca431598 100644 --- a/doc/development/deleting_migrations.md +++ b/doc/development/deleting_migrations.md @@ -1,39 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'database/deleting_migrations.md' +remove_date: '2022-07-08' --- -# Delete existing migrations +This document was moved to [another location](database/deleting_migrations.md). -When removing existing migrations from the GitLab project, you have to take into account -the possibility of the migration already been included in past releases or in the current release, and thus already executed on GitLab.com and/or in self-managed instances. - -Because of it, it's not possible to delete existing migrations, as that could lead to: - -- Schema inconsistency, as changes introduced into the database were not rolled back properly. -- Leaving a record on the `schema_versions` table, that points out to migration that no longer exists on the codebase. - -Instead of deleting we can opt for disabling the migration. - -## Pre-requisites to disable a migration - -Migrations can be disabled if: - -- They caused a timeout or general issue on GitLab.com. -- They are obsoleted, for example, changes are not necessary due to a feature change. -- Migration is a data migration only, that is, the migration does not change the database schema. - -## How to disable a data migration? - -In order to disable a migration, the following steps apply to all types of migrations: - -1. Turn the migration into a no-op by removing the code inside `#up`, `#down` - or `#perform` methods, and adding `# no-op` comment instead. -1. Add a comment explaining why the code is gone. - -Disabling migrations requires explicit approval of Database Maintainer. - -## Examples - -- [Disable scheduling of productivity analytics](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17253) +<!-- This redirect file can be deleted after <2022-07-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/diffs.md b/doc/development/diffs.md index d61de740f15..5f03ba93a4d 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Create +group: Code Review info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 1e4698ff867..fb58851e93f 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -19,8 +19,29 @@ must be documented. For context, see the When you document feature flags, you must: -- [Add a note at the start of the topic](#use-a-note-to-describe-the-state-of-the-feature-flag). - [Add version history text](#add-version-history-text). +- [Add a note at the start of the topic](#use-a-note-to-describe-the-state-of-the-feature-flag). + +## Add version history text + +When the state of a flag changes (for example, disabled by default to enabled by default), add the change to the version history. + +Possible version history entries are: + +```markdown +> - [Introduced](issue-link) in GitLab X.X [with a flag](../../administration/feature_flags.md) named <flag name>. Disabled by default. +> - [Enabled on GitLab.com](issue-link) in GitLab X.X. +> - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. +> - [Enabled on self-managed](issue-link) in GitLab X.X. +> - [Generally available](issue-link) in GitLab X.Y. [Feature flag <flag name>](issue-link) removed. +``` + +You can combine entries if they happened in the same release: + +```markdown +> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. +``` ## Use a note to describe the state of the feature flag @@ -30,7 +51,8 @@ The note has three parts, and follows this structure: ```markdown FLAG: -<Self-managed GitLab availability information.> <GitLab.com availability information.> +<Self-managed GitLab availability information.> +<GitLab.com availability information.> <This feature is not ready for production use.> ``` @@ -61,27 +83,6 @@ If needed, you can add this sentence: `The feature is not ready for production use.` -## Add version history text - -When the state of a flag changes (for example, disabled by default to enabled by default), add the change to the version history. - -Possible version history entries are: - -```markdown -> - [Introduced](issue-link) in GitLab X.X [with a flag](../../administration/feature_flags.md) named <flag name>. Disabled by default. -> - [Enabled on GitLab.com](issue-link) in GitLab X.X. -> - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. -> - [Enabled on self-managed](issue-link) in GitLab X.X. -> - [Generally available](issue-link) in GitLab X.Y. [Feature flag <flag name>](issue-link) removed. -``` - -You can combine entries if they happened in the same release: - -```markdown -> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. -> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. -``` - ## Feature flag documentation examples The following examples show the progression of a feature flag. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 66d6beb821f..c6afcdbddd0 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -20,7 +20,7 @@ In addition to this page, the following resources can help you craft and contrib ## Source files and rendered web locations -Documentation for GitLab, GitLab Runner, Omnibus GitLab, and Charts is published to <https://docs.gitlab.com>. Documentation for GitLab is also published within the application at `/help` on the domain of the GitLab instance. +Documentation for GitLab, GitLab Runner, GitLab Operator, Omnibus GitLab, and Charts is published to <https://docs.gitlab.com>. Documentation for GitLab is also published within the application at `/help` on the domain of the GitLab instance. At `/help`, only help for your current edition and version is included. Help for other versions is available at <https://docs.gitlab.com/archives/>. The source of the documentation exists within the codebase of each GitLab application in the following repository locations: @@ -31,6 +31,7 @@ The source of the documentation exists within the codebase of each GitLab applic | [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) | | [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc) | | [Charts](https://gitlab.com/gitlab-org/charts/gitlab) | [`/doc`](https://gitlab.com/gitlab-org/charts/gitlab/tree/master/doc) | +| [GitLab Operator](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator) | [`/doc`](https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/tree/master/doc) | Documentation issues and merge requests are part of their respective repositories and all have the label `Documentation`. diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index 4d654b6b901..8a505ed84a8 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -83,23 +83,25 @@ to describe the GitLab release that introduced the API call. Use the following table headers to describe the methods. Attributes should always be in code blocks using backticks (`` ` ``). -Sort the attributes in the table: first, required, then alphabetically. +Sort the table by required attributes first, then alphabetically. ```markdown | Attribute | Type | Required | Description | |:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| `title` | string | **{check-circle}** Yes | Title of the issue. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Sets the issue to confidential. Default is `false`. | ``` Rendered example: | Attribute | Type | Required | Description | |:-----------------------------|:--------------|:-----------------------|:-----------------------------------------------------| -| `user` | string | **{check-circle}** Yes | The GitLab username. | -| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | -| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | +| `title` | string | **{check-circle}** Yes | Title of the issue. | +| `assignee_ids` **(PREMIUM)** | integer array | **{dotted-circle}** No | IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Sets the issue to confidential. Default is `false`. | + +For information about writing attribute descriptions, see the [GraphQL API description style guide](../api_graphql_styleguide.md#description-style-guide). ## cURL commands diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index e7a915eab09..bdda15e2064 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -71,7 +71,7 @@ GitLab Docs is built with a combination of external: - [Schema.org](https://schema.org/) - [Google Analytics](https://marketingplatform.google.com/about/analytics/) -- [Google Tag Manager](https://developers.google.com/tag-manager/) +- [Google Tag Manager](https://developers.google.com/tag-platform/tag-manager) ## Global navigation diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 91e9d0c703d..7bfc0320d02 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -349,17 +349,15 @@ Follow these guidelines for punctuation: <!-- vale gitlab.Repetition = NO --> -| Rule | Example | -|------------------------------------------------------------------|--------------------------------------------------------| -| Avoid semicolons. Use two sentences instead. | That's the way that the world goes 'round. You're up one day and the next you're down. -| Always end full sentences with a period. | For a complete overview, read through this document. | -| Always add a space after a period when beginning a new sentence. | For a complete overview, check this doc. For other references, check out this guide. | -| Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) | --- | -| Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. | --- | -| Use serial commas (Oxford commas) before the final **and** or **or** in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) | You can create new issues, merge requests, and milestones. | -| Always add a space before and after dashes when using it in a sentence (for replacing a comma, for example). | You should try this - or not. | -| When a colon is part of a sentence, always use lowercase after the colon. | Linked issues: a way to create a relationship between issues. | -| Do not use typographer's quotes. Use straight quotes instead. (Tested in [`NonStandardQuotes.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/NonStandardQuotes.yml).) | "It's the questions we can't answer that teach us the most"---Patrick Rothfuss | +- End full sentences with a period. +- Use one space between sentences. +- Do not use semicolons. Use two sentences instead. +- Do not use double spaces. (Tested in [`SentenceSpacing.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/SentenceSpacing.yml).) +- Do not use non-breaking spaces. Use standard spaces instead. (Tested in [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh).) +- Do not use tabs for indentation. Use spaces instead. You can configure your code editor to output spaces instead of tabs when pressing the tab key. +- Use serial (Oxford) commas before the final **and** or **or** in a list of three or more items. (Tested in [`OxfordComma.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/OxfordComma.yml).) +- Avoid dashes. Use separate sentences, or commas, instead. +- Do not use typographer's ("curly") quotes. Use straight quotes instead. (Tested in [`NonStandardQuotes.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/NonStandardQuotes.yml).) <!-- vale gitlab.Repetition = YES --> @@ -403,17 +401,6 @@ Backticks are more precise than quotes. For example, in this string: It's not clear whether the user should include the period in the string. -### Spaces between words - -Use only standard spaces between words. The search engine for the documentation -website doesn't split words separated with -[non-breaking spaces](https://en.wikipedia.org/wiki/Non-breaking_space) when -indexing, and fails to create expected individual search terms. Tests that search -for certain words separated by regular spaces can't find words separated by -non-breaking spaces. - -Tested in [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh). - ## Lists - Always start list items with a capital letter, unless they're parameters or @@ -421,30 +408,30 @@ Tested in [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scr - Always leave a blank line before and after a list. - Begin a line with spaces (not tabs) to denote a [nested sub-item](#nesting-inside-a-list-item). -### Ordered vs. unordered lists +### Choose between an ordered or unordered list -Only use ordered lists when their items describe a sequence of steps to follow. - -Do: +Use ordered lists for a sequence of steps. For example: ```markdown -These are the steps to do something: +Follow these steps to do something. 1. First, do the first step. 1. Then, do the next step. 1. Finally, do the last step. ``` -Don't: +Use an unordered lists when the steps do not need to be completed in order. For example: ```markdown -This is a list of available features: +These things are imported: -1. Feature 1 -1. Feature 2 -1. Feature 3 +- Thing 1 +- Thing 2 +- Thing 3 ``` +You can choose to introduce either list with a colon, but you do not have to. + ### Markup - Use dashes (`-`) for unordered lists instead of asterisks (`*`). @@ -454,12 +441,8 @@ This is a list of available features: ### Punctuation - Don't add commas (`,`) or semicolons (`;`) to the ends of list items. -- Only add periods to the end of a list item if the item consists of a complete - sentence (with a subject and a verb). -- Be consistent throughout the list: if the majority of the items do not end in - a period, do not end any of the items in a period, even if they consist of a - complete sentence. The opposite is also valid: if the majority of the items - end with a period, end all with a period. +- If a list item is a complete sentence (with a subject and a verb), add a period at the end. +- Majority rules. If the majority of items do not end in a period, do not end any of the items in a period. - Separate list items from explanatory text with a colon (`:`). For example: ```markdown @@ -469,32 +452,6 @@ This is a list of available features: - Second item: this explains the second item. ``` -**Examples:** - -Do: - -- First list item -- Second list item -- Third list item - -Don't: - -- First list item -- Second list item -- Third list item. - -Do: - -- Let's say this is a complete sentence. -- Let's say this is also a complete sentence. -- Not a complete sentence. - -Don't (vary use of periods; majority rules): - -- Let's say this is a complete sentence. -- Let's say this is also a complete sentence. -- Not a complete sentence - ### Nesting inside a list item It's possible to nest items under a list item, so that they render with the same @@ -686,7 +643,7 @@ For the heading text, **do not**: - Use words that might change in the future. Changing a heading changes its anchor URL, which affects other linked pages. - Repeat text from earlier headings. For example, instead of `Troubleshooting merge requests`, - use `Troubleshooting`. + use `Troubleshooting`. - Use links. ### Anchor links @@ -746,7 +703,7 @@ We include guidance for links in these categories: - Use inline link Markdown markup `[Text](https://example.com)`. It's easier to read, review, and maintain. Do not use `[Text][identifier]` reference-style links. -- Use [meaningful anchor text](https://www.futurehosting.com/blog/links-should-have-meaningful-anchor-text-heres-why/). +- Use meaningful anchor text. For example, instead of writing something like `Read more about merge requests [here](LINK)`, write `Read more about [merge requests](LINK)`. @@ -1561,6 +1518,47 @@ The voting strategy in GitLab 13.4 and later requires the primary and secondary voters to agree. ``` +#### Deprecated features + +When a feature is deprecated, add `(DEPRECATED)` to the page title or to +the heading of the section documenting the feature, immediately before +the tier badge: + +```markdown +<!-- Page title example: --> +# Feature A (DEPRECATED) **(ALL TIERS)** + +<!-- Doc section example: --> +## Feature B (DEPRECATED) **(PREMIUM SELF)** +``` + +Add the deprecation to the version history note (you can include a link +to a replacement when available): + +```markdown +> - [Deprecated](<link-to-issue>) in GitLab 11.3. Replaced by [meaningful text](<link-to-appropriate-documentation>). +``` + +You can also describe the replacement in surrounding text, if available. If the +deprecation isn't obvious in existing text, you may want to include a warning: + +```markdown +WARNING: +This feature was [deprecated](link-to-issue) in GitLab 12.3 and replaced by +[Feature name](link-to-feature-documentation). +``` + +If you add `(DEPRECATED)` to the page's title and the document is linked from the docs +navigation, either remove the page from the nav or update the nav item to include the +same text before the feature name: + +```yaml + - doc_title: (DEPRECATED) Feature A +``` + +In the first major GitLab version after the feature was deprecated, be sure to +remove information about that deprecated feature. + #### End-of-life for features or products When a feature or product enters its end-of-life, indicate its status by @@ -1572,7 +1570,7 @@ For example: ```markdown WARNING: This feature is in its end-of-life process. It is [deprecated](link-to-issue) -for use in GitLab X.X, and is planned for [removal](link-to-issue) in GitLab X.X. +in GitLab X.X, and is planned for [removal](link-to-issue) in GitLab X.X. ``` After the feature or product is officially deprecated and removed, remove @@ -1647,47 +1645,6 @@ To view historical information about a feature, review GitLab [release posts](https://about.gitlab.com/releases/), or search for the issue or merge request where the work was done. -### Deprecated features - -When a feature is deprecated, add `(DEPRECATED)` to the page title or to -the heading of the section documenting the feature, immediately before -the tier badge: - -```markdown -<!-- Page title example: --> -# Feature A (DEPRECATED) **(ALL TIERS)** - -<!-- Doc section example: --> -## Feature B (DEPRECATED) **(PREMIUM SELF)** -``` - -Add the deprecation to the version history note (you can include a link -to a replacement when available): - -```markdown -> - [Deprecated](<link-to-issue>) in GitLab 11.3. Replaced by [meaningful text](<link-to-appropriate-documentation>). -``` - -You can also describe the replacement in surrounding text, if available. If the -deprecation isn't obvious in existing text, you may want to include a warning: - -```markdown -WARNING: -This feature was [deprecated](link-to-issue) in GitLab 12.3 and replaced by -[Feature name](link-to-feature-documentation). -``` - -If you add `(DEPRECATED)` to the page's title and the document is linked from the docs -navigation, either remove the page from the nav or update the nav item to include the -same text before the feature name: - -```yaml - - doc_title: (DEPRECATED) Feature A -``` - -In the first major GitLab version after the feature was deprecated, be sure to -remove information about that deprecated feature. - ## Products and features Refer to the information in this section when describing products and features diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index c38c6586c3a..65f6a0a328b 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -97,6 +97,12 @@ The token generated when you create an agent for Kubernetes. Use **agent access - secret token - authentication token +## air gap, air-gapped + +Use **offline environment** to describe installations that have physical barriers or security policies that prevent or limit internet access. Do not use **air gap**, **air gapped**, or **air-gapped**. For example: + +- The firewall policies in an offline environment prevent the computer from accessing the internet. + ## allow, enable Try to avoid **allow** and **enable**, unless you are talking about security-related features. @@ -261,11 +267,17 @@ Do not use **Developer permissions**. A user who is assigned the Developer role See [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/d/disable-disabled) for guidance on **disable**. Use **inactive** or **off** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) - ## disallow Use **prevent** instead of **disallow**. ([Vale](../testing.md#vale) rule: [`Substitutions.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Substitutions.yml)) +## downgrade + +To be more upbeat and precise, do not use **downgrade**. Focus instead on the action the user is taking. + +- For changing to earlier GitLab versions, use [**roll back**](#roll-back). +- For changing to lower GitLab tiers, use **change the subscription tier**. + ## dropdown list Use **dropdown list** to refer to the UI element. Do not use **dropdown** without **list** after it. @@ -729,12 +741,31 @@ Do not use **Reporter permissions**. A user who is assigned the Reporter role ha Use title case for **Repository Mirroring**. +## respectively + +Avoid **respectively** and be more precise instead. + +Use: + +- To create a user, select **Create user**. For an existing user, select **Save changes**. + +Instead of: + +- Select **Create user** or **Save changes** if you created a new user or + edited an existing one respectively. + ## roles Do not use **roles** and [**permissions**](#permissions) interchangeably. Each user is assigned a role. Each role includes a set of permissions. Roles are not the same as [**access levels**](#access-level). +## roll back + +Use **roll back** for changing a GitLab version to an earlier one. + +Do not use **roll back** for licensing or subscriptions. Use **change the subscription tier** instead. + ## runner, runners Use lowercase for **runners**. These are the agents that run CI/CD jobs. See also [GitLab Runner](#gitlab-runner) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). @@ -921,6 +952,33 @@ Use [**2FA** and **two-factor authentication**](#2fa-two-factor-authentication) Do not use **type** if you can avoid it. Use **enter** instead. +## update + +Use **update** for installing a newer **patch** version of the software only. For example: + +- Update GitLab from 14.9 to 14.9.1. + +Do not use **update** for any other case. Instead, use **upgrade**. + +## upgrade + +Use **upgrade** for: + +- Choosing a higher subscription tier (Premium or Ultimate). +- Installing a newer **major** (13.0, 14.0) or **minor** (13.8, 14.5) version of GitLab. + +For example: + +- Upgrade to GitLab Ultimate. +- Upgrade GitLab from 14.0 to 14.1. +- Upgrade GitLab from 14.0 to 15.0. + +Use caution with the phrase **Upgrade GitLab** without any other text. +Ensure the surrounding text clarifies whether +you're talking about the product version or the subscription tier. + +See also [downgrade](#downgrade) and [roll back](#roll-back). + ## useful Do not use **useful**. If the user doesn't find the process to be useful, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) @@ -971,7 +1029,7 @@ Do not use **yet** when talking about the product or its features. The documenta Sometimes you might need to use **yet** when writing a task. If you use **yet**, ensure the surrounding phrases are written -in present tense, active voice. +in present tense, active voice. [View guidance about how to write about future features](index.md#promising-features-in-future-versions). ([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index 49fe0aff3c6..9facb22669b 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -189,7 +189,7 @@ English language. Vale's configuration is stored in the [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) file located in the root directory of projects. -Vale supports creating [custom tests](https://errata-ai.github.io/vale/styles/) that extend any of +Vale supports creating [custom tests](https://docs.errata.ai/vale/styles) that extend any of several types of checks, which we store in the `.linting/vale/styles/gitlab` directory in the documentation directory of projects. @@ -365,6 +365,35 @@ file for the [`gitlab`](https://gitlab.com/gitlab-org/gitlab) project. To set up `lefthook` for documentation linting, see [Pre-push static analysis](../contributing/style_guides.md#pre-push-static-analysis-with-lefthook). +#### Show Vale warnings on push + +By default, `lefthook` shows only Vale errors when pushing changes to a branch. The default branches +have no Vale errors, so any errors listed here are introduced by commits to the branch. + +To also see the Vale warnings when pushing to a branch, set a local environment variable: `VALE_WARNINGS=true`. + +Enable Vale warnings on push to improve the documentation suite by: + +- Detecting warnings you might be introducing with your commits. +- Identifying warnings that already exist in the page, which you can resolve to reduce technical debt. + +These warnings: + +- Don't stop the push from working. +- Don't result in a broken pipeline. +- Include all warnings for a file, not just warnings that are introduced by the commits. + +To enable Vale warnings on push: + +- Automatically, add `VALE_WARNINGS=true` to your shell configuration. +- Manually, prepend `VALE_WARNINGS=true` to invocations of `lefthook`. For example: + + ```shell + VALE_WARNINGS=true bundle exec lefthook run pre-push + ``` + +You can also [configure your editor](#configure-editors) to show Vale warnings. + ### Show subset of Vale alerts You can set Visual Studio Code to display only a subset of Vale alerts when viewing files: diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 5bd830715f5..019dbb13599 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -21,7 +21,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w When developing locally, there are times when you need your instance to act like the SaaS version of the product. In those instances, you can simulate SaaS by exporting an environment variable as seen below: -`export GITLAB_SIMULATE_SAAS=1` +```shell +export GITLAB_SIMULATE_SAAS=1 +``` + +There are many ways to pass an environment variable to your local GitLab instance. +For example, you can create a `env.runit` file in the root of your GDK with the above snippet. ## Act as CE when unlicensed diff --git a/doc/development/event_store.md b/doc/development/event_store.md index b00a824e2eb..967272dcf2e 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -290,3 +290,42 @@ executed synchronously every time the given event is published. For complex conditions it's best to subscribe to all the events and then handle the logic in the `handle_event` method of the subscriber worker. + +## Testing + +The publisher's responsibility is to ensure that the event is published correctly. + +To test that an event has been published correctly, we can use the RSpec matcher `:publish_event`: + +```ruby +it 'publishes a ProjectDeleted event with project id and namespace id' do + expected_data = { project_id: project.id, namespace_id: project.namespace_id } + + # The matcher verifies that when the block is called, the block publishes the expected event and data. + expect { destroy_project(project, user, {}) } + .to publish_event(Projects::ProjectDeletedEvent) + .with(expected_data) +end +``` + +The subscriber must ensure that a published event can be consumed correctly. For this purpose +we have added helpers and shared examples to standardize the way we test subscribers: + +```ruby +RSpec.describe MergeRequests::UpdateHeadPipelineWorker do + let(:event) { Ci::PipelineCreatedEvent.new(data: ({ pipeline_id: pipeline.id })) } + + # This shared example ensures that an event is published and correctly processed by + # the current subscriber (`described_class`). + it_behaves_like 'consumes the published event' do + let(:event) { event } + end + + it 'does something' do + # This helper directly executes `perform` ensuring that `handle_event` is called correctly. + consume_event(subscriber: described_class, event: event) + + # run expectations + end +end +``` diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index c34e5eb36dc..f7af1113b6e 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -64,8 +64,8 @@ We recommend the following workflow: 1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the [Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2). - -## Turn off all experiments + +## Turn off all experiments When there is a case on GitLab.com (SaaS) that necessitates turning off all experiments, we have this control. diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index e71e414002a..2a1083d031f 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -32,7 +32,7 @@ By default, macOS limits the <kbd>tab</kbd> key to **Text boxes and lists only** 1. Open the **Shortcuts** tab. 1. Enable the setting **Use keyboard navigation to move focus between controls**. -You can read more about enabling browser-specific keyboard navigation on [a11yproject](https://www.a11yproject.com/posts/2017-12-29-macos-browser-keyboard-navigation/). +You can read more about enabling browser-specific keyboard navigation on [a11yproject](https://www.a11yproject.com/posts/macos-browser-keyboard-navigation/). ## Quick checklist diff --git a/doc/development/fe_guide/emojis.md b/doc/development/fe_guide/emojis.md index 2dedbc8f19d..7ef88c5ca19 100644 --- a/doc/development/fe_guide/emojis.md +++ b/doc/development/fe_guide/emojis.md @@ -25,7 +25,7 @@ when your platform does not support it. - `app/assets/images/emoji.png` - `app/assets/images/emoji@2x.png` 1. Ensure you see new individual images copied into `app/assets/images/emoji/` - 1. Ensure you can see the new emojis and their aliases in the GitLab Flavored Markdown (GFM) Autocomplete + 1. Ensure you can see the new emojis and their aliases in the GitLab Flavored Markdown (GLFM) Autocomplete 1. Ensure you can see the new emojis and their aliases in the award emoji menu 1. You might need to add new emoji Unicode support checks and rules for platforms that do not support a certain emoji and we need to fallback to an image. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index e79a473df9e..ddd99f3614d 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -264,9 +264,15 @@ Read more about [Vue Apollo](https://github.com/vuejs/vue-apollo) in the [Vue Ap ### Local state with Apollo -It is possible to manage an application state with Apollo by passing -in a resolvers object when creating the default client. The default state can be set by writing -to the cache after setting up the default client. In the example below, we are using query with `@client` Apollo directive to write the initial data to Apollo cache and then get this state in the Vue component: +It is possible to manage an application state with Apollo by using [client-site resolvers](#using-client-side-resolvers) +or [type policies with reactive variables](#using-type-policies-with-reactive-variables) when creating your default +client. + +#### Using client-side resolvers + +The default state can be set by writing to the cache after setting up the default client. In the +example below, we are using query with `@client` Apollo directive to write the initial data to +Apollo cache and then get this state in the Vue component: ```javascript // user.query.graphql @@ -322,7 +328,7 @@ export default { Along with creating local data, we can also extend existing GraphQL types with `@client` fields. This is extremely helpful when we need to mock an API response for fields not yet added to our GraphQL API. -#### Mocking API response with local Apollo cache +##### Mocking API response with local Apollo cache Using local Apollo Cache is helpful when we have a need to mock some GraphQL API responses, queries, or mutations locally (such as when they're still not added to our actual API). @@ -384,6 +390,108 @@ For each attempt to fetch a version, our client fetches `id` and `sha` from the Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.app/guide/local-state.html#local-state). +#### Using type policies with reactive variables + +Apollo Client 3 offers an alternative to [client-side resolvers](#using-client-side-resolvers) by using +[reactive variables to store client state](https://www.apollographql.com/docs/react/local-state/reactive-variables/). + +**NOTE:** +We are still learning the best practices for both **type policies** and **reactive vars**. +Take a moment to improve this guide or [leave a comment](https://gitlab.com/gitlab-org/frontend/rfcs/-/issues/100) +if you use it! + +In the example below we define a `@client` query and its `typedefs`: + +```javascript +// ./graphql/typedefs.graphql +extend type Query { + localData: String! +} +``` + +```javascript +// ./graphql/get_local_data.query.graphql +query getLocalData { + localData @client +} +``` + +Similar to resolvers, your `typePolicies` will execute when the `@client` query is used. However, +using `makeVar` will trigger every relevant active Apollo query to reactively update when the state +mutates. + +```javascript +// ./graphql/local_state.js + +import { makeVar } from '@apollo/client/core'; +import typeDefs from './typedefs.graphql'; + +export const createLocalState = () => { + // set an initial value + const localDataVar = makeVar(''); + + const cacheConfig = { + typePolicies: { + Query: { + fields: { + localData() { + // obtain current value + // triggers when `localDataVar` is updated + return localDataVar(); + }, + }, + }, + }, + }; + + // methods that update local state + const localMutations = { + setLocalData(newData) { + localDataVar(newData); + }, + clearData() { + localDataVar(''); + }, + }; + + return { + cacheConfig, + typeDefs, + localMutations, + }; +}; +``` + +Pass the cache config to your Apollo Client: + +```javascript +// index.js + +// ... +import createDefaultClient from '~/lib/graphql'; +import { createLocalState } from './graphql/local_state'; + +const { cacheConfig, typeDefs, localMutations } = createLocalState(); + +const apolloProvider = new VueApollo({ + defaultClient: createDefaultClient({}, { cacheConfig, typeDefs }), +}); + +return new Vue({ + el, + apolloProvider, + provide: { + // inject local state mutations to your app + localMutations, + }, + render(h) { + return h(MyApp); + }, +}); +``` + +Wherever used, the local query will update as the state updates thanks to the **reactive variable**. + ### Using with Vuex When the Apollo Client is used in Vuex and fetched data is stored in the Vuex store, the Apollo Client cache does not need to be enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. With Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache by passing a valid `fetchPolicy` option to its constructor: @@ -583,7 +691,7 @@ we want to fetch after or before a given endpoint. For example, here we're fetching 10 designs after a cursor (let us call this `projectQuery`): ```javascript -#import "~/graphql_shared/fragments/pageInfo.fragment.graphql" +#import "~/graphql_shared/fragments/page_info.fragment.graphql" query { project(fullPath: "root/my-project") { @@ -606,7 +714,7 @@ query { } ``` -Note that we are using the [`pageInfo.fragment.graphql`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/graphql_shared/fragments/pageInfo.fragment.graphql) to populate the `pageInfo` information. +Note that we are using the [`page_info.fragment.graphql`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/graphql_shared/fragments/page_info.fragment.graphql) to populate the `pageInfo` information. #### Using `fetchMore` method in components @@ -869,7 +977,7 @@ You'd then be able to retrieve the data without providing any pagination-specifi Here's an example of a query using the `@connection` directive: ```graphql -#import "~/graphql_shared/fragments/pageInfo.fragment.graphql" +#import "~/graphql_shared/fragments/page_info.fragment.graphql" query DastSiteProfiles($fullPath: ID!, $after: String, $before: String, $first: Int, $last: Int) { project(fullPath: $fullPath) { diff --git a/doc/development/fe_guide/performance.md b/doc/development/fe_guide/performance.md index 94beecf6168..bcdc49a1070 100644 --- a/doc/development/fe_guide/performance.md +++ b/doc/development/fe_guide/performance.md @@ -457,6 +457,6 @@ General tips: ## Additional Resources - [WebPage Test](https://www.webpagetest.org) for testing site loading time and size. -- [Google PageSpeed Insights](https://developers.google.com/speed/pagespeed/insights/) grades web pages and provides feedback to improve the page. +- [Google PageSpeed Insights](https://pagespeed.web.dev/) grades web pages and provides feedback to improve the page. - [Profiling with Chrome DevTools](https://developer.chrome.com/docs/devtools/) - [Browser Diet](https://browserdiet.com/) is a community-built guide that catalogues practical tips for improving web page performance. diff --git a/doc/development/fe_guide/registry_architecture.md b/doc/development/fe_guide/registry_architecture.md new file mode 100644 index 00000000000..47a6dc40e19 --- /dev/null +++ b/doc/development/fe_guide/registry_architecture.md @@ -0,0 +1,90 @@ +--- +stage: Package +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Registry architecture + +GitLab has several registry applications. Given that they all leverage similar UI, UX, and business +logic, they are all built with the same architecture. In addition, a set of shared components +already exists to unify the user and developer experiences. + +Existing registries: + +- Package Registry +- Container Registry +- Infrastructure Registry +- Dependency Proxy + +## Frontend architecture + +### Component classification + +All the registries follow an architecture pattern that includes four component types: + +- Pages: represent an entire app, or for the registries using [vue-router](https://v3.router.vuejs.org/) they represent one router + route. +- Containers: represent a single piece of functionality. They contain complex logic and may + connect to the API. +- Presentationals: represent a portion of the UI. They receive all their data with `props` or through + `inject`, and do not connect to the API. +- Shared components: presentational components that accept a various array of configurations and are + shared across all of the registries. + +### Communicating with the API + +The complexity and communication with the API should be concentrated in the pages components, and +in the container components when needed. This makes it easier to: + +- Handle concurrent requests, loading states, and user messages. +- Maintain the code, especially to estimate work. If it touches a page or functional component, + expect it to be more complex. +- Write fast and consistent unit tests. + +### Best practices + +- Use [`provide` or `inject`](https://v2.vuejs.org/v2/api/?redirect=true#provide-inject) + to pass static, non-reactive values coming from the app initialization. +- When passing data, prefer `props` over nested queries or Vuex bindings. Only pages and + container components should be aware of the state and API communication. +- Don't repeat yourself. If one registry receives functionality, the likelihood of the rest needing + it in the future is high. If something seems reusable and isn't bound to the state, create a + shared component. +- Try to express functionality and logic with dedicated components. It's much easier to deal with + events and properties than callbacks and asynchronous code (see + [`delete_package.vue`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/packages_and_registries/package_registry/components/functional/delete_package.vue)). +- Leverage [startup for GraphQL calls](graphql.md#making-initial-queries-early-with-graphql-startup-calls). + +## Shared compoenents library + +Inside `vue_shared/components/registry` and `packages_and_registries/shared`, there's a set of +shared components that you can use to implement registry functionality. These components build the +main pieces of the desired UI and UX of a registry page. The most important components are: + +- `code-instruction`: represents a copyable box containing code. Supports multiline and single line + code boxes. Snowplow tracks the code copy event. +- `details-row`: represents a row of details. Used to add additional info in the details area of + the `list-item` component. +- `history-item`: represents a history list item used to build a timeline. +- `list-item`: represents a list element in the registry. It supports: left action, left primary and + secondary content, right primary and secondary content, right action, and details slots. +- `metadata-item`: represents one piece of metadata, with an icon or a link. Used primarily in the + title area. +- `persisted-dropdown-selection`: represents a dropdown menu that stores the user selection in the + `localStorage`. +- `registry-search`: implements `gl-filtered-search` with a sorting section on the right. +- `title-area`: implements the top title area of the registry. Includes: a main title, an avatar, a + subtitle, a metadata row, and a right actions slot. + +## Adding a new registry page + +When adding a new registry: + +- Leverage the shared components that already exist. It's good to look at how the components are + structured and used in the more mature registries (for example, the Package Registry). +- If it's in line with the backend requirements, we suggest using GraphQL for the API. This helps in + dealing with the innate performance issue of registries. +- If possible, we recommend using [Vue Router](https://v3.router.vuejs.org/) + and frontend routing. Coupled with Apollo, the caching layer helps with the perceived page + performance. diff --git a/doc/development/fe_guide/source_editor.md b/doc/development/fe_guide/source_editor.md index 2ff0bacfc3a..b06e341630f 100644 --- a/doc/development/fe_guide/source_editor.md +++ b/doc/development/fe_guide/source_editor.md @@ -35,7 +35,7 @@ Vue component, but the integration of Source Editor is generally straightforward const editor = new SourceEditor({ // Editor Options. // The list of all accepted options can be found at - // https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.editoroption.html + // https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.EditorOption.html }); ``` @@ -56,19 +56,19 @@ An instance of Source Editor accepts the following configuration options: | `blobContent` | `false` | `String`: The initial content to render in the editor. | | `extensions` | `false` | `Array`: Extensions to use in this instance. | | `blobGlobalId` | `false` | `String`: An auto-generated property.<br>**Note:** This property may go away in the future. Do not pass `blobGlobalId` unless you know what you're doing.| -| Editor Options | `false` | `Object(s)`: Any property outside of the list above is treated as an Editor Option for this particular instance. Use this field to override global Editor Options on the instance level. A full [index of Editor Options](https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.editoroption.html) is available. | +| Editor Options | `false` | `Object(s)`: Any property outside of the list above is treated as an Editor Option for this particular instance. Use this field to override global Editor Options on the instance level. A full [index of Editor Options](https://microsoft.github.io/monaco-editor/api/enums/monaco.editor.EditorOption.html) is available. | ## API The editor uses the same public API as -[provided by Monaco editor](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html) +[provided by Monaco editor](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) with additional functions on the instance level: | Function | Arguments | Description | --------------------- | ----- | ----- | | `updateModelLanguage` | `path`: String | Updates the instance's syntax highlighting to follow the extension of the passed `path`. Available only on the instance level.| | `use` | Array of objects | Array of extensions to apply to the instance. Accepts only the array of _objects_. You must fetch the extensions' ES6 modules must be fetched and resolved in your views or components before they are passed to `use`. This property is available on _instance_ (applies extension to this particular instance) and _global editor_ (applies the same extension to all instances) levels. | -| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html) | Default Monaco editor options | +| Monaco Editor options | See [documentation](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html) | Default Monaco editor options | ## Tips @@ -202,7 +202,7 @@ export default { In the code example, `this` refers to the instance. By referring to the instance, we can access the complete underlying -[Monaco editor API](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.istandalonecodeeditor.html), +[Monaco editor API](https://microsoft.github.io/monaco-editor/api/interfaces/monaco.editor.IStandaloneCodeEditor.html), which includes functions like `getValue()`. Now let's use our extension: diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index b947d90cc11..fecb0af936d 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -123,6 +123,10 @@ Using dependency injection to provide values from HAML is ideal when: prop-drilling becomes an inconvenience. Prop-drilling when the same prop is passed through all components in the hierarchy until the component that is genuinely using it. +Dependency injection can potentially break a child component (either an immediate child or multiple levels deep) if the value declared in the `inject` configuration doesn't have defaults defined and the parent component has not provided the value using the `provide` configuration. + +- A [default value](https://vuejs.org/guide/components/provide-inject.html#injection-default-values) might be useful in contexts where it makes sense. + ##### props If the value from HAML doesn't fit the criteria of dependency injection, use `props`. @@ -499,7 +503,7 @@ component under test, with the `computed` property, for example). Remember to us We should test for events emitted in response to an action in our component. This is used to verify the correct events are being fired with the correct arguments. -For any DOM events we should use [`trigger`](https://vue-test-utils.vuejs.org/api/wrapper/#trigger) +For any DOM events we should use [`trigger`](https://v1.test-utils.vuejs.org/api/wrapper/#trigger) to fire out event. ```javascript @@ -530,7 +534,7 @@ it('should fire the itemClicked event', () => { ``` We should verify an event has been fired by asserting against the result of the -[`emitted()`](https://vue-test-utils.vuejs.org/api/wrapper/#emitted) method. +[`emitted()`](https://v1.test-utils.vuejs.org/api/wrapper/#emitted) method. ## Vue.js Expert Role diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index f174408c946..8c8bb36d962 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -156,6 +156,6 @@ export default { </template> ``` -[In Vue 3](https://v3.vuejs.org/guide/migration/props-default-this.html#props-default-function-this-access), +[In Vue 3](https://v3-migration.vuejs.org/breaking-changes/props-default-this.html), the props default value factory is passed the raw props as an argument, and can also access injections. diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index d6b64001e13..b2d141798fa 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -58,7 +58,8 @@ not, the specs will fail. ### Excluding Sidekiq workers from feature categorization A few Sidekiq workers, that are used across all features, cannot be mapped to a -single category. These should be declared as such using the `feature_category_not_owned!` +single category. These should be declared as such using the +`feature_category :not_owned` declaration, as shown below: ```ruby @@ -66,7 +67,7 @@ class SomeCrossCuttingConcernWorker include ApplicationWorker # Declares that this worker does not map to a feature category - feature_category_not_owned! + feature_category :not_owned # rubocop:disable Gitlab/AvoidFeatureCategoryNotOwned # ... end diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md deleted file mode 100644 index f98366beb6b..00000000000 --- a/doc/development/feature_flags/process.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/' -remove_date: '2022-03-01' ---- - -This document was moved to [another location](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/). - -<!-- This redirect file can be deleted after 2022-03-01. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index 5ff1bc7b127..e0f5a905831 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -58,9 +58,9 @@ to a gem, go through these steps: the [`gitlab-org/ruby/gems` namespace](https://gitlab.com/gitlab-org/ruby/gems/). - To create this project: - 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/#creating-a-new-project). - 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/#cicd-configuration). - 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/#publishing-a-project). + 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#creating-a-new-project). + 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#cicd-configuration). + 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/gitlab-repositories/#publishing-a-project). - See [issue #325463](https://gitlab.com/gitlab-org/gitlab/-/issues/325463) for an example. diff --git a/doc/development/gitlab_flavored_markdown/index.md b/doc/development/gitlab_flavored_markdown/index.md new file mode 100644 index 00000000000..682d8011cd8 --- /dev/null +++ b/doc/development/gitlab_flavored_markdown/index.md @@ -0,0 +1,20 @@ +--- +stage: Create +group: Editor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Markdown developer documentation **(FREE)** + +This page contains the MVC for the developer documentation for GitLab Flavored Markdown. +For the user documentation about Markdown in GitLab, refer to +[GitLab Flavored Markdown](../../user/markdown.md). + +## GitLab Flavored Markdown specification guide + +The [specification guide](specification_guide/index.md) includes: + +- [Terms and definitions](specification_guide/index.md#terms-and-definitions). +- [Parsing and rendering](specification_guide/index.md#parsing-and-rendering). +- [Goals](specification_guide/index.md#goals). +- [Implementation](specification_guide/index.md#implementation) of the spec. diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md new file mode 100644 index 00000000000..021f7bafce9 --- /dev/null +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -0,0 +1,717 @@ +--- +stage: Create +group: Editor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Flavored Markdown (GLFM) Specification Guide **(FREE)** + +GitLab supports Markdown in various places. The Markdown dialect we use is called +GitLab Flavored Markdown, or GLFM. + +The specification for the GLFM dialect is based on the +[GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/), +which is in turn based on the [CommonMark specification](https://spec.commonmark.org/current/). +The GLFM specification includes +[several extensions](../../../user/markdown.md#differences-between-gitlab-flavored-markdown-and-standard-markdown) +to the GFM specification. + +See the [section on acronyms](#acronyms-glfm-ghfm-gfm-commonmark) for a +detailed explanation of the various acronyms used in this document. +This guide is a developer-facing document that describes the various terms and +definitions, goals, tools, and implementations related to the GLFM specification. +It is intended to support and augment the [user-facing documentation](../../../user/markdown.md) +for GitLab Flavored Markdown. + +NOTE: +In this document, _GFM_ refers to _GitHub_ Flavored Markdown, not _GitLab_ Flavored Markdown. +Refer to the [section on acronyms](#acronyms-glfm-ghfm-gfm-commonmark) +for a detailed explanation of the various acronyms used in this document. + +NOTE: +This guide and the implementation and files described in it are still a work in +progress. As the work progresses, rewrites and consolidation +between this guide and the [user-facing documentation](../../../user/markdown.md) +for GitLab Flavored Markdown are likely. + +## Terms and definitions + +### Acronyms: GLFM, GHFM, GFM, CommonMark + +[_GitHub_ Flavored Markdown](https://github.github.com/gfm/) is widely referred +to by the acronym GFM, and this document follows that convention as well. +_GitLab_ Flavored Markdown is referred to as GLFM in this document, +to distinguish it from GitHub Flavored Markdown. + +Unfortunately, this convention is not followed consistently in the rest +of the documentation or GitLab codebase. In many places, the GFM +acronym is used to refer to _GitLab_ Flavored Markdown. An +[open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24592) exists to resolve +this inconsistency. + +Some places in the code refer to both the GitLab and GitHub specifications +simultaneous in the same areas of logic. In these situations, +_GitHub_ Flavored Markdown may be referred to with variable or constant names like +`ghfm_` to avoid confusion. + +The original CommonMark specification is referred to as _CommonMark_ (no acronym). + +### Various Markdown specifications + +The specification format we use is based on the approach used in CommonMark, where +a `spec.txt` file serves as documentation, as well as being in a format that can +serve as input to automated conformance tests. It is +[explained in the CommonMark specification](https://spec.commonmark.org/0.30/#about-this-document): + +> This document attempts to specify Markdown syntax unambiguously. It contains many +> examples with side-by-side Markdown and HTML. These are intended to double as conformance tests. + +The HTML-rendered versions of the specifications: + +- [GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html), which extends the: +- [GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/), which extends the: +- [CommonMark specification](https://spec.commonmark.org/0.30/) + +NOTE: +The creation of the +[GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html) +file is still pending. + +However, GLFM has more complex parsing, rendering, and testing requirements than +GFM or CommonMark. Therefore, +it does not have a static, hardcoded, manually updated `spec.txt`. Instead, the +GLFM `spec.txt` is automatically generated based on other input files. This process +is explained in detail in the [Implementation](#implementation) sections below. + +### Markdown examples + +Everywhere in the context of the specification and this guide, the term +_examples_ is specifically used to refer to the Markdown + HTML pairs used +to illustrate the canonical parsing (or rendering) behavior of various Markdown source +strings in the standard +[CommonMark specification format](https://spec.commonmark.org/0.30/#example-1). + +In this context, it should not be confused with other similar or related meanings of +_example_, such as +[RSpec examples](https://relishapp.com/rspec/rspec-core/docs/example-groups/basic-structure-describe-it). + +### Parsers and renderers + +To understand the various ways in which a specification is used, and how it related +to a given Markdown dialect, it's important to understand the distinction between +a _parser_ and a _renderer_: + +- A Markdown _parser_ accepts Markdown as input and produces a Markdown + Abstract Syntax Tree (AST) as output. +- A Markdown _renderer_ accepts the AST produced by a parser, and produces HTML + (or a PDF, or any other relevant rendering format) as output. + +### Types of Markdown tests driven by the GLFM specification + +The two main types of automated testing are driven by the Markdown +examples and data contained in the GLFM specification. We refer to them as: + +- Markdown conformance testing. +- Markdown snapshot testing. + +Many other types of tests also occur in the GitLab +codebase, and some of these tests are also related to the GLFM Markdown dialect. +Therefore, to avoid confusion, we use these standard terms for the two types +of specification-driven testing referred to in this documentation and elsewhere. + +#### Markdown conformance testing + +_Markdown conformance testing_ refers to the standard testing method used by +all CommonMark Markdown dialects to verify that a specific implementation conforms +to the CommonMark Markdown specification. It is enforced by running the standard +CommonMark tool [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py) +against a given `spec.txt` specification and the implementation. + +NOTE: +`spec_tests.py` may eventually be re-implemented in Ruby, to not have a dependency on Python. + +#### Markdown snapshot testing + +_Markdown snapshot testing_ refers to the automated testing performed in +the GitLab codebase, which is driven by snapshot fixture data derived from the +GLFM specification. It consists of both backend RSpec tests and frontend Jest tests +which use the fixture data. This fixture data is contained in YAML files. These files +can be generated and updated based on the Markdown examples in the specification, +and the existing GLFM parser and render implementations. They may also be +manually updated as necessary to test-drive incomplete implementations. +Regarding the terminology used here: + +1. The Markdown snapshot tests can be considered a form of the + [Golden Master Testing approach](https://www.google.com/search?q=golden+master+testing), + which is also referred to as Approval Testing or Characterization Testing. + 1. The term Golden Master originally comes from the recording industry, and + refers to the process of mastering, or making a final mix from which all + other copies are produced. + 1. For more information and background, you can read about + [Characterization Tests](https://en.wikipedia.org/wiki/Characterization_test) and + [Golden Masters](https://en.wikipedia.org/wiki/Gold_master_(disambiguation)). +1. The usage of the term _snapshot_ does not refer to the approach of + [Jest snapshot testing](https://jestjs.io/docs/snapshot-testing), as used elsewhere + in the GitLab frontend testing suite. However, the Markdown snapshot testing does + follow the same philosophy and patterns as Jest snapshot testing: + 1. Snapshot fixture data is represented as files which are checked into source control. + 1. The files can be automatically generated and updated based on the implementation + of the code under tests. + 1. The files can also be manually updated when necessary, for example, to test-drive + changes to an incomplete or buggy implementation. +1. The usage of the term _fixture_ does not refer to standard + [Rails database fixture files](https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html). + It instead refers to _test fixtures_ in the + [more generic definition](https://en.wikipedia.org/wiki/Test_fixture#Software), + as input data to support automated testing. However, fixture files still exist, so + they are colocated under the `spec/fixtures` directory with the rest of + the fixture data for the GitLab Rails application. + +## Parsing and Rendering + +The Markdown dialect used in the GitLab application has a dual requirement for rendering: + +1. Rendering to static read-only HTML format, to be displayed in various + places throughout the application. +1. Rendering editable content in the + [Content Editor](https://about.gitlab.com/direction/create/editor/content_editor/), + a ["What You See Is What You Get" (WYSIWYG)](https://en.wikipedia.org/wiki/WYSIWYG) + editor. The Content Editor supports real-time instant switching between an editable + Markdown source and an editable WYSIWYG document. + +These requirements means that GitLab has two independent parser and renderer +implementations: + +1. The backend parser / renderer supports parsing and rendering to _static_ + read-only HTML. It is [implemented in Ruby](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/banzai). + It leverages the [`commonmarker`](https://github.com/gjtorikian/commonmarker) gem, + which is a Ruby wrapper for [`libcmark-gfm`](https://github.com/github/cmark), + GitHub's fork of the reference parser for CommonMark. `libcmark-gfm` is an extended + version of the C reference implementation of [CommonMark](http://commonmark.org/) +1. The frontend parser / renderer supports parsing and _WYSIWYG_ rendering for + the Content Editor. It is implemented in JavaScript. Parsing is based on the + [Remark](https://github.com/remarkjs/remark) Markdown parser, which produces a + MDAST Abstract Syntax Tree (MDAST). Rendering is the process of turning + an MDAST into a [ProseMirror document](../../fe_guide/content_editor.md). Then, + ProseMirror is used to render a ProseMirror document to WYSIWYG HTML. In this + document, we refer to the process of turning Markdown into an MDAST as the + _frontend / JavaScript parser_, and the entire process of rendering Markdown + to WYSIWYG HTML in ProseMirror as the _Content Editor_. Several + requirements drive the need for an independent frontend parser / renderer + implementation, including: + 1. Lack of necessary support for accurate source mapping in the HTML renderer + implementation used on the backend. + 1. Latency and bandwidth concerns: eliminating the need for a round-trip to the backend + every time the user switches between the Markdown source and the WYSIWYG document. + 1. Different HTML and browser rendering requirements for WYSIWYG documents. For example, + displaying read-only elements such as diagrams and references in an editable form. + +### Multiple versions of rendered HTML + +Both of these GLFM renderer implementations (static and WYSIWYG) produce +HTML which differs from the canonical HTML examples from the specification. +For every Markdown example in the GLFM specification, three +versions of HTML can potentially be rendered from the example: + +1. **Static HTML**: HTML produced by the backend (Ruby) renderer, which + contains extra styling and behavioral HTML. For example, **Create task** buttons + added for dynamically creating an issue from a task list item. + The GitLab [Markdown API](../../../api/markdown.md) generates HTML + for a given Markdown string using this method. +1. **WYSIWYG HTML**: HTML produced by the frontend (JavaScript) Content Editor, + which includes parsing and rendering logic. Used to present an editable document + in the ProseMirror WYSIWYG editor. +1. **Canonical HTML**: The clean, basic version of HTML rendered from Markdown. + 1. For the examples which come from the CommonMark specification and + GFM extensions specification, + the canonical HTML is the exact identical HTML found in the + GFM + `spec.txt` example blocks. + 1. For GLFM extensions to the <abbr title="GitHub Flavored Markdown">GFM</abbr> / CommonMark + specification, a `glfm_canonical_examples.txt` + [input specification file](#input-specification-files) contains the + Markdown examples and corresponding canonical HTML examples. + +As the rendered static and WYSIWYG HTML from the backend (Ruby) and frontend (JavaScript) +renderers contain extra HTML, their rendered HTML can be converted to canonical HTML +by a [canonicalization](#canonicalization-of-html) process. + +#### Canonicalization of HTML + +Neither the backend (Ruby) nor the frontend (JavaScript) rendered can directly render canonical HTML. +Nor should they be able to, because: + +- It's not a direct requirement to support any GitLab application feature. +- Adding this feature adds unnecessary requirements and complexity to the implementations. + +Instead, the rendered static or WYSIWYG HTML is converted to canonical HTML by a +_canonicalization_ process. This process can strip all the extra styling and behavioral +HTML from the static or WYSIWYG HTML, resulting in canonical HTML which exactly +matches the Markdown + HTML examples in a standard `spec.txt` specification. + +Use the [`canonicalize-html.rb` script](#canonicalize-htmlrb-script) for this process. +More explanation about this canonicalization process in the sections below. + +NOTE: +Some of the static or WYSIWYG HTML examples may not be representable as canonical +HTML. (For example, when they are represented as an image.) In these cases, the Markdown +conformance test for the example can be skipped by setting `skip_update_example_snapshots: true` +for the example in `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml`. + +## Goals + +Given the constraints above, we have a few goals related to the GLFM +specification and testing infrastructure: + +1. A canonical `spec.txt` exists, and represents the official specification for + GLFM, which meets these requirements: + 1. The spec is a strict superset of the GitHub Flavored Markdown + (GFM) specification, just as + <abbr title="GitHub Flavored Markdown">GFM</abbr> is a strict superset + [of the CommonMark specification](https://github.github.com/gfm/#what-is-github-flavored-markdown-). + Therefore, it contains the superset of all canonical Markdown + HTML examples + for CommonMark, GFM, and GLFM. + 1. It contains a prose introduction section which is specific to GitLab and GLFM. + 1. It contains all other non-introduction sections verbatim from the + GFM + `spec.txt`. + 1. It contains a new extra section for the GLFM GitLab-specific extensions, + with both prose and examples describing the extensions. + 1. It should be in the standard format which can processed by the standard + CommonMark tools [`spec_tests.py`](https://github.com/github/cmark-gfm/blob/master/test/spec_tests.py), + which is a [script used to run the Markdown conformance tests](https://github.github.com/gfm/#about-this-document) + against all examples contained in a `spec.txt`. +1. The GLFM parsers and HTML renderers for + both the static backend (Ruby) and WYSIWYG frontend (JavaScript) implementations + support _consistent_ rendering of all canonical Markdown + HTML examples in the + GLFM `spec.txt` specification, as verified by `spec_tests.py`. + + NOTE: + Consistent does not mean that both of these implementations render + to the identical HTML. They each have different implementation-specific additions + to the HTML they render, so therefore their rendered HTML is + ["canonicalized"](#canonicalization-of-html) to canonical HTML prior running + the Markdown conformance tests. +1. For _both_ the static backend (Ruby) and WYSIWYG frontend (JavaScript) implementations, + a set of example snapshots exists in the form of YAML files, which + correspond to every Markdown example in the GLFM `spec.txt`. These example snapshots + support the following usages for every GLFM Markdown example: + 1. The backend (Ruby) parser and renderer can convert Markdown to the + expected custom static HTML. + 1. The frontend (JavaScript) parser and renderer (which includes GitLab custom + code and Remark) can convert Markdown to the expected ProseMirror JSON + representing a ProseMirror document. + 1. The **Content Editor** (which includes the frontend (JavaScript) parser and renderer, + and ProseMirror) can convert Markdown to the expected custom WYSIWYG HTML as rendered by ProseMirror. + 1. The **Content Editor** can complete a round-trip test, which involves converting + from Markdown, to MDAST, to ProseMirror Document, then back to Markdown. It ensures + the resulting Markdown is exactly identical, with no differences. + +## Implementation + +The following set of scripts and files is complex. However, it allows us to meet +all of the goals listed above, and is carefully designed to meet the following +implementation goals: + +1. Minimize the amount of manual editing, curation, and maintenance of the GLFM specification + and related files. +1. Automate and simplify the process of updating the GLFM specification and related + files when there are changes to the upstream CommonMark spec, + GFM extensions, or the GLFM extensions. +1. Support partial or incomplete implementations of the GLFM specification, whether + due to in-progress work, bugs, or new future Markdown support, while still + performing all functionality for the existing implementations. +1. Automate, simplify, and support running various tests, including the standard + CommonMark conformance tests and GLFM-implementation-specific unit/acceptance + Markdown snapshot tests. +1. Provide a rich set of extensible metadata around all GLFM specification examples + to support current and future requirements, such as automated acceptance + testing and automated documentation updates. + +The documentation on the implementation is split into three sections: + +1. [Scripts](#scripts). +1. [Specification files](#specification-files). +1. Example snapshot files: These YAML files are used as input data + or fixtures to drive the various tests, and are located under + `spec/fixtures/glfm/example_snapshots`. All example snapshot files are automatically + generated based on the specification files and the implementation of the parsers and renderers. + However, they can also be directly edited if necessary, such as to + test-drive an incomplete implementation. + +### Scripts + +These executable scripts perform various tasks related to maintaining +the specification and running tests. Each script has a shell-executable entry point +file located under `scripts/glfm`, but the actual implementation is in unit-tested +classes under `scripts/lib/glfm`. + +NOTE: +Some of these scripts are implemented in Ruby, and others are shell scripts. +Ruby scripts are used for more complex custom scripts, to enable easier unit testing +and debugging. Shell scripts are used for simpler scripts which primarily invoke +other shell commands, to avoid the challenges related to +[running other shell sub-processes](https://github.com/thewoolleyman/process_helper#why-yet-another-ruby-process-wrapper-library) +from Ruby scripts. + +NOTE: +The Ruby executable scripts under `scripts/glfm` have dashes instead of underscores +in the filenames. This naming is non-standard for a Ruby file, but is used to distinguish +them from the corresponding implementation class entry point files under +`scripts/lib/glfm` when searching by filename. + +#### `update-specification.rb` script + +The `scripts/glfm/update-specification.rb` script uses specification input files to +generate and update `spec.txt` (Markdown) and `spec.html` (HTML). The `spec.html` is +generated by passing the generated (or updated) `spec.txt` Markdown to the backend API +for rendering to static HTML: + +```mermaid +graph LR +subgraph script: + A{update-specification.rb} + A --> B{Backend Markdown API} +end +subgraph input:<br/>input specification files + C[gfm_spec_v_0.29.txt] --> A + D[glfm_intro.txt] --> A + E[glfm_canonical_examples.txt] --> A +end +subgraph output:<br/>GLFM specification files + A --> F[spec.txt] + F --> B + B --> G[spec.html] +end +``` + +#### `update-example-snapshots.rb` script + +The `scripts/glfm/update-example-snapshots.rb` script uses input specification +files to update example snapshots: + +```mermaid +graph LR +subgraph script: + A{update-example-snapshots.rb} +end +subgraph input:<br/>input specification files + B[downloaded gfm_spec_v_0.29.txt] --> A + C[glfm_canonical_examples.txt] --> A + D[glfm_example_status.yml] --> A +end +subgraph output:<br/>example snapshot files + A --> E[examples_index.yml] + A --> F[markdown.yml] + A --> G[html.yml] + A --> H[prosemirror_json.yml] +end +``` + +#### `run-snapshot-tests.sh` script + +The `scripts/glfm/run-snapshot-tests.sh` convenience shell script runs all relevant +Markdown snapshot testing RSpec and Jest `*_spec` files (from main app `spec` folder) +which are driven by `example_snapshot` YAML files. + +The actual RSpec and Jest test `*_spec` files (frontend and backend) live +under the normal relevant locations under `spec`, matching the location of their +corresponding implementations. They can be run either: + +- As part of the normal pipelines. +- From the command line or an IDE, just like any other file under `spec`. + +However, they are spread across four different locations: + +- Backend tests under `spec/requests`. +- Backend EE tests under `ee/spec/requests`. +- Frontend tests under `spec/frontend`. +- Frontend EE tests under `ee/spec/frontend`. + +Therefore, this convenience script is intended to only be used in local +development. It simplifies running all tests at once and returning a single return +code. It contains only shell scripting commands for the relevant +`bundle exec rspec ...` and `yarn jest ...` commands. + +```mermaid +graph LR +subgraph script: + A{run-snapshopt-tests.sh} --> B + B[relevant rspec/jest test files] +end +subgraph input:<br/>YAML + C[examples_index.yml] --> B + D[markdown.yml] --> B + E[html.yml] --> B + F[prosemirror_json.yml] --> B +end +subgraph output:<br/>test results/output + B --> G[rspec/jest output] +end +``` + +#### `canonicalize-html.rb` script + +The `scripts/glfm/canonicalize-html.rb` handles the +["canonicalization" of HTML](#canonicalization-of-html). It is a pipe-through +helper script which takes as input a static or WYSIWYG HTML string containing +extra HTML, and outputs a canonical HTML string. + +It is implemented as a standalone, modular, single-purpose script, based on the +[Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy#:~:text=The%20Unix%20philosophy%20emphasizes%20building,developers%20other%20than%20its%20creators.). +It's easy to use when running the standard CommonMark `spec_tests.py` +script, which expects canonical HTML, against the GitLab renderer implementations. + +#### `run-spec-tests.sh` script + +`scripts/glfm/run-spec-tests.sh` is a convenience shell script which runs +conformance specs via the CommonMark standard `spec_tests.py` script, +which uses the `glfm_specification/output/spec.txt` file and `scripts/glfm/canonicalize-html.rb` +helper script to test the GLFM renderer implementations' support for rendering Markdown +specification examples to canonical HTML. + +```mermaid +graph LR +subgraph scripts: + A{run-spec-tests.sh} --> C + subgraph specification testing process + B[canonicalize-html.sh] --> C + C[spec_tests.py] + end +end +subgraph input + D[spec.txt GLFM specification] --> C + E((GLFM static<br/>renderer implementation)) --> B + F((GLFM WYSIWYG<br/>renderer implementation)) --> B +end +subgraph output:<br/>test results/output + C --> G[spec_tests.py output] +end +``` + +### Specification files + +These files represent the GLFM specification itself. They are all +located under the root `glfm_specification`, and are further divided into two +subfolders: + +- `input`: Contains files which are imported or manually edited. +- `output`: Contains files which are automatically generated. + +#### Input specification files + +The `glfm_specification/input` directory contains files which are the original +input to drive all other automated GLFM specification scripts/processes/tests. +They are either downloaded, as in the case of the +GFM `spec.txt` file, or manually +updated, as in the case of all GFM files. + +- `glfm_specification/input/github_flavored_markdown/gfm_spec_v_0.29.txt` - + official latest [GFM spec.txt](https://github.com/github/cmark-gfm/blob/master/test/spec.txt), + automatically downloaded and updated by `update-specification.rb` script. +- `glfm_specification/input/gitlab_flavored_markdown/glfm_intro.txt` - + Manually updated text of intro section for generated GLFM `spec.txt`. + - Replaces GFM version of introductory + section in `spec.txt`. +- `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` - + Manually updated canonical Markdown+HTML examples for GLFM extensions. + - Standard backtick-delimited `spec.txt` examples format with Markdown + canonical HTML. + - Inserted as a new section before the appendix of generated `spec.txt`. +- `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` - + Manually updated status of automatic generation of files based on Markdown + examples. + - Allows example snapshot generation, Markdown conformance tests, or + Markdown snapshot tests to be skipped for individual examples. For example, if + they are unimplemented, broken, or cannot be tested for some reason. + +`glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` sample entry: + +```yaml +07_99_an_example_with_incomplete_wysiwyg_implementation_1: + skip_update_example_snapshots: true + skip_running_snapshot_static_html_tests: false + skip_running_snapshot_wysiwyg_html_tests: true + skip_running_snapshot_prosemirror_json_tests: true + skip_running_conformance_static_tests: false + skip_running_conformance_wysiwyg_tests: true +``` + +#### Output specification files + +The `glfm_specification/output` directory contains the CommonMark standard format +`spec.txt` file which represents the canonical GLFM specification which is generated +by the `update-specification.rb` script. It also contains the rendered `spec.html` +and `spec.pdf` which are generated from with the `spec.txt` as input. + +- `glfm_specification/output/spec.txt` - A Markdown file, in the standard format + with prose and Markdown + canonical HTML examples, generated (or updated) by the + `update-specification.rb` script. +- `glfm_specification/output/spec.html` - An HTML file, rendered based on `spec.txt`, + also generated (or updated) by the `update-specification.rb` script at the same time as + `spec.txt`. It corresponds to the HTML-rendered versions of the + "GitHub Flavored Markdown" (<abbr title="GitHub Flavored Markdown">GFM</abbr>) + [specification](https://github.github.com/gfm/) + and the [CommonMark specification](https://spec.commonmark.org/0.30/). + +These output `spec.**` files, which represent the official, canonical GLFM specification +are colocated under the same parent folder `glfm_specification` with the other +`input` specification files. They're located here both for convenience, and because they are all +a mix of manually edited and generated files. In GFM, +`spec.txt` is [located in the test dir](https://github.com/github/cmark-gfm/blob/master/test/spec.txt), +and in CommonMark it's located +[in the project root](https://github.com/github/cmark-gfm/blob/master/test/spec.txt). +No precedent exists for a standard location. In the future, we may decide to +move or copy a hosted version of the rendered HTML `spec.html` version to another location or site. + +### Example snapshot files + +The `example_snapshots` directory contains files which are generated by the +`update-example-snapshots.rb` script based off of the files in the +`glfm_specification/input` directory. They are used as fixtures to drive the +various Markdown snapshot tests. + +After the entire GLFM implementation is complete for both backend (Ruby) and +frontend (JavaScript), all of these YAML files can be automatically generated. +However, while the implementations are still in progress, the `skip_update_example_snapshots` +key in `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` +can be used to disable automatic generation of some examples, and they can instead +be manually edited as necessary to help drive the implementations. + +#### `spec/fixtures/glfm/example_snapshots/examples_index.yml` + +`spec/fixtures/glfm/example_snapshots/examples_index.yml` is the main list of all +CommonMark, GFM, and GLFM example names, each with a unique canonical name. + +- It is generated from the hierarchical sections and examples in the + GFM `spec.txt` specification. +- For CommonMark and GFM examples, + these sections originally came from the GFM `spec.txt`. +- For GLFM examples, it is generated from `glfm_canonical_examples.txt`, which is + the additional Section 7 in the GLFM `spec.txt`. +- It also contains extra metadata about each example, such as: + 1. `spec_txt_example_position` - The position of the example in the generated GLFM `spec.txt` file. + 1. `source_specification` - Which specification the example originally came from: + `commonmark`, `github`, or `gitlab`. +- The naming convention for example entry names is based on nested header section + names and example index within the header. + - This naming convention should result in fairly stable names and example positions. + The CommonMark / GLFM specification rarely changes, and most GLFM + examples where multiple examples exist for the same Section 7 subsection are + added to the end of the sub-section. + +`spec/fixtures/glfm/example_snapshots/examples_index.yml` sample entries: + +```yaml +02_01_preliminaries_characters_and_lines_1: + spec_txt_example_position: 1 + source_specification: commonmark +03_01_blocks_and_inlines_precedence_1: + spec_txt_example_position: 12 + source_specification: commonmark +05_03_container_blocks_task_list_items_1: + spec_txt_example_position: 279 + source_specification: github +06_04_inlines_emphasis_and_strong_emphasis_1: + spec_txt_example_position: 360 + source_specification: github +07_01_audio_link_1: + spec_txt_example_position: 301 + source_specification: gitlab +``` + +#### `spec/fixtures/glfm/example_snapshots/markdown.yml` + +`spec/fixtures/glfm/example_snapshots/markdown.yml` contains the original Markdown +for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` + +- For CommonMark and GFM Markdown, + it is generated (or updated) from the standard GFM + `spec.txt` using the `update-example-snapshots.rb` script. +- For GLFM, it is generated (or updated) from the + `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` + input specification file. + +`spec/fixtures/glfm/example_snapshots/markdown.yml` sample entry: + +```yaml +06_04_inlines_emphasis_and_strong_emphasis_1: |- + *foo bar* +``` + +#### `spec/fixtures/glfm/example_snapshots/html.yml` + +`spec/fixtures/glfm/example_snapshots/html.yml` contains the HTML for each entry in +`spec/fixtures/glfm/example_snapshots/examples_index.yml` + +Three types of entries exist, with different HTML for each: + +- **Canonical** + - The ["Canonical"](#canonicalization-of-html) HTML. + - For CommonMark and GFM examples, the HTML comes from the examples in `spec.txt`. + - For GLFM examples, it is generated/updated from + `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt`. +- **Static** + - This is the static (backend (Ruby)-generated) HTML for each entry in + `spec/fixtures/glfm/example_snapshots/examples_index.yml`. + - It is generated/updated from backend [Markdown API](../../../api/markdown.md) + (or the underlying internal classes) via the `update-example-snapshots.rb` script, + but can be manually updated for static examples with incomplete implementations. +- **WYSIWYG** + - The WYSIWYG (frontend, JavaScript-generated) HTML for each entry in + `spec/fixtures/glfm/example_snapshots/examples_index.yml`. + - It is generated (or updated) from the frontend Content Editor implementation via the + `update-example-snapshots.rb` script. It can be manually updated for WYSIWYG + examples with incomplete implementations. + +Any exceptions or failures which occur when generating HTML are replaced with an +`Error - check implementation` value. + +`spec/fixtures/glfm/example_snapshots/html.yml` sample entry: + +```yaml +06_04_inlines_emphasis_and_strong_emphasis_1: + canonical: |- + <p><em>foo bar</em></p> + static: |- + <p data-sourcepos="1:1-1:9" dir="auto"><strong>foo bar</strong></p> + wysiwyg: |- + <p><strong>foo bar</strong></p> +``` + +NOTE: +The actual `static` or `WYSIWYG` entries may differ from the example `html.yml`, +depending on how the implementations evolve. + +#### `spec/fixtures/glfm/example_snapshots/prosemirror_json.yml` + +`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml` contains the ProseMirror +JSON for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` + +- It is generated (or updated) from the frontend code via the `update-example-snapshots.rb` + script, but can be manually updated for examples with incomplete implementations. +- Any exceptions or failures when generating are replaced with a `Error - check implementation` value. + +`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml` sample entry: + +```yaml +06_04_inlines_emphasis_and_strong_emphasis_1: |- + { + "type": "doc", + "content": [ + { + "type": "paragraph", + "content": [ + { + "type": "text", + "marks": [ + { + "type": "bold" + } + ], + "text": "foo bar" + }, + ] + }, + ] + } +``` diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index a99253b9723..3267d1262f0 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -76,9 +76,27 @@ if you need help finding the correct person or labels: 1. Create the epic in `gitlab-org` group: - Title the epic `Update Go version to <VERSION_NUMBER>`. - Ping the engineering managers responsible for [the projects listed below](#known-dependencies-using-go). + - Most engineering managers can be identified on + [the product page](https://about.gitlab.com/handbook/product/categories/) or the + [feature page](https://about.gitlab.com/handbook/product/categories/features/). + - If you still can't find the engineering manager, use + [Git blame](/ee/user/project/repository/git_blame.md) to identify a maintainer + involved in the project. + +1. Create an upgrade issue for each dependency in the + [location indicated below](#known-dependencies-using-go) titled + `Support building with Go <VERSION_NUMBER>`. Add the proper labels to each issue + for easier triage. These should include the stage, group and section. + - The issue should be assigned by a member of the maintaining group. + - The milestone should be assigned by a member of the maintaining group. -1. Create an upgrade issue for each dependency in the [location indicated below](#known-dependencies-using-go) - titled `Support building with Go <VERSION_NUMBER>`. Add the proper label to each issue for easier triage. + NOTE: + Some overlap exists between project dependencies. When creating an issue for a + dependency that is part of a larger product, note the relationship in the issue + body. For example: Projects built in the context of Omnibus GitLab have their + runtime Go version managed by Omnibus, but "support" and compatibility should + be a concern of the individual project. Issues in the parent project's dependencies + issue should be about adding support for the updated Go version. NOTE: The upgrade issues must include [upgrade validation items](#upgrade-validation) @@ -94,9 +112,10 @@ if you need help finding the correct person or labels: - [Composition Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - [Container Security tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - NOTE: - Updates to these Security analyzers should not block upgrades to Charts or Omnibus since - the analyzers are built independently as separate container images. + NOTE: + Updates to these Security analyzers should not block upgrades to Charts or Omnibus since + the analyzers are built independently as separate container images. + 1. Schedule builder updates with Distribution projects: - Dependency and GitLab Development Kit issues created in previous steps should be set as blockers. - Each issue should have the title `Support building with Go <VERSION_NUMBER>` and description as noted: diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 65cf8911e12..2aea15de443 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -786,7 +786,7 @@ The reasoning behind this is that in some languages words change depending on co in Japanese は is added to the subject of a sentence and を to the object. This is impossible to translate correctly if you extract individual words from the sentence. -When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). +When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://mozilla-l10n.github.io/documentation/localization/dev_best_practices.html#splitting-and-composing-sentences). ### Always pass string literals to the translation helpers diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 7c9777527ef..afc04045763 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -20,7 +20,7 @@ are very appreciative of the work done by translators and proofreaders! - Arabic - Proofreaders needed. - Bosnian - - Proofreaders needed. + - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) - Bulgarian - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) - Catalan @@ -38,7 +38,7 @@ are very appreciative of the work done by translators and proofreaders! - Victor Wu - [GitLab](https://gitlab.com/_victorwu_), [Crowdin](https://crowdin.com/profile/victorwu) - Ivan Ip - [GitLab](https://gitlab.com/lifehome), [Crowdin](https://crowdin.com/profile/lifehome) - Croatian - - Proofreaders needed. + - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) - Czech - Jan Urbanec - [GitLab](https://gitlab.com/TatranskyMedved), [Crowdin](https://crowdin.com/profile/Tatranskymedved) - Danish @@ -111,7 +111,7 @@ are very appreciative of the work done by translators and proofreaders! - Andrey Komarov - [GitLab](https://gitlab.com/elkamarado), [Crowdin](https://crowdin.com/profile/kamarado) - Iaroslav Postovalov - [GitLab](https://gitlab.com/CMDR_Tvis), [Crowdin](https://crowdin.com/profile/CMDR_Tvis) - Serbian (Latin and Cyrillic) - - Proofreaders needed. + - Haris Delalić - [GitLab](https://gitlab.com/haris.delalic), [Crowdin](https://crowdin.com/profile/haris.delalic) - Sinhalese/Sinhala සිංහල - හෙළබස (HelaBasa) - [GitLab](https://gitlab.com/helabasa), [Crowdin](https://crowdin.com/profile/helabasa) - Slovak diff --git a/doc/development/img/merge_request_reports_v14_7.png b/doc/development/img/merge_request_reports_v14_7.png Binary files differindex 282d6f96aa6..1c06e7f4fd0 100644 --- a/doc/development/img/merge_request_reports_v14_7.png +++ b/doc/development/img/merge_request_reports_v14_7.png diff --git a/doc/development/img/merge_widget_v14_7.png b/doc/development/img/merge_widget_v14_7.png Binary files differindex d5e8ed8df52..86bc11802d1 100644 --- a/doc/development/img/merge_widget_v14_7.png +++ b/doc/development/img/merge_widget_v14_7.png diff --git a/doc/development/index.md b/doc/development/index.md index 048112215fc..5c0cc7f9718 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -97,9 +97,8 @@ a given group, request an engineering review from one of the group's members. After the engineering review is complete, assign the MR to the [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) in the modified documentation page's metadata. - -If you have questions or need further input, request a review from the -Technical Writer assigned to the [Development Guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). +If the page is not assigned to a specific group, follow the +[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). #### Broader changes @@ -139,12 +138,13 @@ In these cases, use the following workflow: and approval from the VP of Development, the DRI for Development Guidelines, @clefelhocz1. -1. After all approvals are complete, review the page's metadata to - [find a Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - who can help you merge the changes. - for final content review and merge. The Technical Writer may ask for - additional approvals as previously suggested before merging the MR. - +1. After all approvals are complete, assign the MR to the + [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + in the modified documentation page's metadata. + If the page is not assigned to a specific group, follow the + [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + The Technical Writer may ask for additional approvals as previously suggested before merging the MR. + ### Reviewer values > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1. diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md index 5391b2c119e..26ef67c937c 100644 --- a/doc/development/integrations/jira_connect.md +++ b/doc/development/integrations/jira_connect.md @@ -79,7 +79,7 @@ If you use Gitpod and you get an error about Jira not being able to access the d > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81126) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `jira_connect_oauth`. Disabled by default. -GitLab for Jira users can authenticate with GitLab using GitLab OAuth. +GitLab for Jira users can authenticate with GitLab using GitLab OAuth. WARNING: This feature is not ready for production use. The feature flag should only be enabled in development. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 11fb06bd128..5f7cccdab64 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -327,6 +327,21 @@ You can find the schemas for these scanners here: - [Coverage Fuzzing](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json) - [Secret Detection](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/secret-detection-report-format.json) +### Retention period for vulnerabilities + +GitLab has the following retention policies for vulnerabilities on non-default branches. Vulnerabilities are no longer available: + +- When the related CI job artifact expires. +- 90 days after the pipeline is created, even if the related CI job artifacts are locked. + +To view vulnerabilities, either: + +- Re-run the pipeline. +- Download the related CI job artifacts if they are available. + +NOTE: +This does not apply for the vulnerabilities existing on the default branch. + ### Enable report validation > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/354928) in GitLab 14.9, and planned for removal in GitLab 15.0. diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index ef58d6c2c44..cdbc674e0a5 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -621,7 +621,7 @@ Example response: "name":"premium", "trial":false, "auto_renew":null, - "upgradable":false + "upgradable":false, }, "usage": { "seats_in_subscription":10, @@ -672,7 +672,7 @@ Example response: "name":"premium", "trial":false, "auto_renew":null, - "upgradable":false + "upgradable":false, }, "usage": { "seats_in_subscription":80, @@ -711,7 +711,8 @@ Example response: "name":"premium", "trial":false, "auto_renew":null, - "upgradable":false + "upgradable":false, + "exclude_guests":false, }, "usage": { "seats_in_subscription":80, diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index 38cdbdf5b79..8813fe560db 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -93,7 +93,7 @@ falling into an endless loop as described in following When dealing with data migrations the preferred way to iterate over a large volume of data is using `EachBatch`. -A special case of data migration is a [background migration](background_migrations.md#scheduling) +A special case of data migration is a [background migration](database/background_migrations.md#scheduling) where the actual data modification is executed in a background job. The migration code that determines the data ranges (slices) and schedules the background jobs uses `each_batch`. diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 0de3f94cf70..6df5c2164e8 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -1,6 +1,6 @@ --- stage: Fulfillment -group: License +group: Provision info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/merge_request_concepts/index.md b/doc/development/merge_request_concepts/index.md index f1dab69543a..90e8ff41368 100644 --- a/doc/development/merge_request_concepts/index.md +++ b/doc/development/merge_request_concepts/index.md @@ -49,7 +49,7 @@ When all of the required merge checks are satisfied a merge request becomes merg ### Approvals -Approval rules specify users that are required to or can optionally approve a merge request based on some kind of organizational policy. When approvals are required, they effectively become a required merge check. The key differentiator between merge checks and approval rules is that users **do** interact with approval rules, by deciding to approve the merge request. +Approval rules specify users that are required to or can optionally approve a merge request based on some kind of organizational policy. When approvals are required, they effectively become a required merge check. The key differentiator between merge checks and approval rules is that users **do** interact with approval rules, by deciding to approve the merge request. Additionally, approval settings provide configuration options to define how those approval rules are applied in a merge request. They can set limitations, add requirements, or modify approvals. @@ -58,5 +58,5 @@ Examples of approval rules and settings include: 1. [merge request approval rules](../../user/project/merge_requests/approvals/rules.md) 1. [code owner approvals](../../user/project/code_owners.md) 1. [security approvals](../../user/application_security/index.md#security-approvals-in-merge-requests) -1. [prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests)] +1. [prevent editing approval rules](../../user/project/merge_requests/approvals/settings.md#prevent-editing-approval-rules-in-merge-requests) 1. [remove all approvals when commits are added](../../user/project/merge_requests/approvals/settings.md#remove-all-approvals-when-commits-are-added-to-the-source-branch) diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 40f02f4fb6f..fe8e730d64e 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -16,7 +16,7 @@ with and agreed upon by backend maintainers and performance specialists. It's also highly recommended that you read the following guides: - [Performance Guidelines](performance.md) -- [Avoiding downtime in migrations](avoiding_downtime_in_migrations.md) +- [Avoiding downtime in migrations](database/avoiding_downtime_in_migrations.md) ## Definition diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index d85b7372814..086e061452b 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -45,14 +45,14 @@ work it needs to perform and how long it takes to complete: One exception is a migration that takes longer but is absolutely critical for the application to operate correctly. For example, you might have indices that enforce unique tuples, or that are needed for query performance in critical parts of the application. In cases where the migration would be unacceptably slow, however, a better option might be to guard the feature with a [feature flag](feature_flags/index.md) and perform a post-deployment migration instead. The feature can then be turned on after the migration finishes. -1. [**Post-deployment migrations.**](post_deployment_migrations.md) These are Rails migrations in `db/post_migrate` and +1. [**Post-deployment migrations.**](database/post_deployment_migrations.md) These are Rails migrations in `db/post_migrate` and run _after_ new application code has been deployed (for GitLab.com after the production deployment has finished). They can be used for schema changes that aren't critical for the application to operate, or data migrations that take at most a few minutes. Common examples for schema changes that should run post-deploy include: - Clean-ups, like removing unused columns. - Adding non-critical indices on high-traffic tables. - Adding non-critical indices that take a long time to create. -1. [**Background migrations.**](background_migrations.md) These aren't regular Rails migrations, but application code that is +1. [**Background migrations.**](database/background_migrations.md) These aren't regular Rails migrations, but application code that is executed via Sidekiq jobs, although a post-deployment migration is used to schedule them. Use them only for data migrations that exceed the timing guidelines for post-deploy migrations. Background migrations should _not_ change the schema. @@ -129,13 +129,13 @@ TARGET=12-9-stable-ee scripts/regenerate-schema ## Avoiding downtime -The document ["Avoiding downtime in migrations"](avoiding_downtime_in_migrations.md) specifies +The document ["Avoiding downtime in migrations"](database/avoiding_downtime_in_migrations.md) specifies various database operations, such as: -- [dropping and renaming columns](avoiding_downtime_in_migrations.md#dropping-columns) -- [changing column constraints and types](avoiding_downtime_in_migrations.md#changing-column-constraints) -- [adding and dropping indexes, tables, and foreign keys](avoiding_downtime_in_migrations.md#adding-indexes) -- [migrating `integer` primary keys to `bigint`](avoiding_downtime_in_migrations.md#migrating-integer-primary-keys-to-bigint) +- [dropping and renaming columns](database/avoiding_downtime_in_migrations.md#dropping-columns) +- [changing column constraints and types](database/avoiding_downtime_in_migrations.md#changing-column-constraints) +- [adding and dropping indexes, tables, and foreign keys](database/avoiding_downtime_in_migrations.md#adding-indexes) +- [migrating `integer` primary keys to `bigint`](database/avoiding_downtime_in_migrations.md#migrating-integer-primary-keys-to-bigint) and explains how to perform them without requiring downtime. @@ -219,7 +219,7 @@ in that limit. Singular query timings should fit within the [standard limit](que In case you need to insert, update, or delete a significant amount of data, you: - Must disable the single transaction with `disable_ddl_transaction!`. -- Should consider doing it in a [Background Migration](background_migrations.md). +- Should consider doing it in a [Background Migration](database/background_migrations.md). ## Migration helpers and versioning @@ -240,7 +240,7 @@ of migration helpers. In this example, we use version 1.0 of the migration class: ```ruby -class TestMigration < Gitlab::Database::Migration[1.0] +class TestMigration < Gitlab::Database::Migration[2.0] def change end end @@ -253,7 +253,7 @@ version of migration helpers automatically. Migration helpers and versioning were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68986) in GitLab 14.3. For merge requests targeting previous stable branches, use the old format and still inherit from -`ActiveRecord::Migration[6.1]` instead of `Gitlab::Database::Migration[1.0]`. +`ActiveRecord::Migration[6.1]` instead of `Gitlab::Database::Migration[2.0]`. ## Retry mechanism when acquiring database locks @@ -535,7 +535,7 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < Gitlab::Database::Migration[1.0] +class MyMigration < Gitlab::Database::Migration[2.0] disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -586,7 +586,7 @@ by calling the method `disable_ddl_transaction!` in the body of your migration class like so: ```ruby -class MyMigration < Gitlab::Database::Migration[1.0] +class MyMigration < Gitlab::Database::Migration[2.0] disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -629,7 +629,7 @@ The easiest way to test for existence of an index by name is to use the be used with a name option. For example: ```ruby -class MyMigration < Gitlab::Database::Migration[1.0] +class MyMigration < Gitlab::Database::Migration[2.0] INDEX_NAME = 'index_name' def up @@ -664,7 +664,7 @@ Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. ```ruby -class Migration < Gitlab::Database::Migration[1.0] +class Migration < Gitlab::Database::Migration[2.0] def change add_reference :model, :other_model, index: true, foreign_key: { on_delete: :cascade } @@ -710,7 +710,7 @@ expensive and disruptive operation for larger tables, but in reality it's not. Take the following migration as an example: ```ruby -class DefaultRequestAccessGroups < Gitlab::Database::Migration[1.0] +class DefaultRequestAccessGroups < Gitlab::Database::Migration[2.0] def change change_column_default(:namespaces, :request_access_enabled, from: false, to: true) end @@ -943,7 +943,7 @@ The Rails 5 natively supports `JSONB` (binary JSON) column type. Example migration adding this column: ```ruby -class AddOptionsToBuildMetadata < Gitlab::Database::Migration[1.0] +class AddOptionsToBuildMetadata < Gitlab::Database::Migration[2.0] def change add_column :ci_builds_metadata, :config_options, :jsonb end @@ -975,7 +975,7 @@ Do not store `attr_encrypted` attributes as `:text` in the database; use efficient: ```ruby -class AddSecretToSomething < Gitlab::Database::Migration[1.0] +class AddSecretToSomething < Gitlab::Database::Migration[2.0] def change add_column :something, :encrypted_secret, :binary add_column :something, :encrypted_secret_iv, :binary @@ -1033,8 +1033,8 @@ If you need more complex logic, you can define and use models local to a migration. For example: ```ruby -class MyMigration < Gitlab::Database::Migration[1.0] - class Project < ActiveRecord::Base +class MyMigration < Gitlab::Database::Migration[2.0] + class Project < MigrationRecord self.table_name = 'projects' end @@ -1114,7 +1114,7 @@ by an integer. For example: `users` would turn into `users0` ## Using models in migrations (discouraged) The use of models in migrations is generally discouraged. As such models are -[contraindicated for background migrations](background_migrations.md#isolation), +[contraindicated for background migrations](database/background_migrations.md#isolation), the model needs to be declared in the migration. If using a model in the migrations, you should first @@ -1132,8 +1132,8 @@ in a previous migration. It is important not to leave out the `User.reset_column_information` command, in order to ensure that the old schema is dropped from the cache and ActiveRecord loads the updated schema information. ```ruby -class AddAndSeedMyColumn < Gitlab::Database::Migration[1.0] - class User < ActiveRecord::Base +class AddAndSeedMyColumn < Gitlab::Database::Migration[2.0] + class User < MigrationRecord self.table_name = 'users' end diff --git a/doc/development/new_fe_guide/modules/widget_extensions.md b/doc/development/new_fe_guide/modules/widget_extensions.md index d3cd839464d..638a0a2a85b 100644 --- a/doc/development/new_fe_guide/modules/widget_extensions.md +++ b/doc/development/new_fe_guide/modules/widget_extensions.md @@ -40,6 +40,7 @@ export default { summary(data) {}, // Required: Level 1 summary text statusIcon(data) {}, // Required: Level 1 status icon tertiaryButtons() {}, // Optional: Level 1 action buttons + shouldCollapse() {}, // Optional: Add logic to determine if the widget can expand or not }, methods: { fetchCollapsedData(props) {}, // Required: Fetches data required for collapsed state diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 2aef0e10314..e0b236bc5fc 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -187,7 +187,7 @@ See the [experiment issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/is #### Automatic retry of failing tests in a separate process -When the `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `true`, RSpec tests that failed are automatically retried once in a separate +Unless `$RETRY_FAILED_TESTS_IN_NEW_PROCESS` variable is set to `false` (`true` by default), RSpec tests that failed are automatically retried once in a separate RSpec process. The goal is to get rid of most side-effects from previous tests that may lead to a subsequent test failure. We keep track of retried tests in the `$RETRIED_TESTS_REPORT_FILE` file saved as artifact by the `rspec:flaky-tests-report` job. diff --git a/doc/development/post_deployment_migrations.md b/doc/development/post_deployment_migrations.md index 6ab3620c197..c3922718e77 100644 --- a/doc/development/post_deployment_migrations.md +++ b/doc/development/post_deployment_migrations.md @@ -1,81 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'database/post_deployment_migrations.md' +remove_date: '2022-07-08' --- -# Post Deployment Migrations +This document was moved to [another location](database/post_deployment_migrations.md). -Post deployment migrations are regular Rails migrations that can optionally be -executed after a deployment. By default these migrations are executed alongside -the other migrations. To skip these migrations you must set the -environment variable `SKIP_POST_DEPLOYMENT_MIGRATIONS` to a non-empty value -when running `rake db:migrate`. - -For example, this would run all migrations including any post deployment -migrations: - -```shell -bundle exec rake db:migrate -``` - -This however skips post deployment migrations: - -```shell -SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rake db:migrate -``` - -## Deployment Integration - -Say you're using Chef for deploying new versions of GitLab and you'd like to run -post deployment migrations after deploying a new version. Let's assume you -normally use the command `chef-client` to do so. To make use of this feature -you'd have to run this command as follows: - -```shell -SKIP_POST_DEPLOYMENT_MIGRATIONS=true sudo chef-client -``` - -Once all servers have been updated you can run `chef-client` again on a single -server _without_ the environment variable. - -The process is similar for other deployment techniques: first you would deploy -with the environment variable set, then you re-deploy a single -server but with the variable _unset_. - -## Creating Migrations - -To create a post deployment migration you can use the following Rails generator: - -```shell -bundle exec rails g post_deployment_migration migration_name_here -``` - -This generates the migration file in `db/post_migrate`. These migrations -behave exactly like regular Rails migrations. - -## Use Cases - -Post deployment migrations can be used to perform migrations that mutate state -that an existing version of GitLab depends on. For example, say you want to -remove a column from a table. This requires downtime as a GitLab instance -depends on this column being present while it's running. Normally you'd follow -these steps in such a case: - -1. Stop the GitLab instance -1. Run the migration removing the column -1. Start the GitLab instance again - -Using post deployment migrations we can instead follow these steps: - -1. Deploy a new version of GitLab while ignoring post deployment migrations -1. Re-run `rake db:migrate` but without the environment variable set - -Here we don't need any downtime as the migration takes place _after_ a new -version (which doesn't depend on the column anymore) has been deployed. - -Some other examples where these migrations are useful: - -- Cleaning up data generated due to a bug in GitLab -- Removing tables -- Migrating jobs from one Sidekiq queue to another +<!-- This redirect file can be deleted after <2022-07-08>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index 6943f931d79..2395689ada2 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -10,6 +10,41 @@ The Product Qualified Lead (PQL) funnel connects our users with our team members A hand-raise PQL is a user who requests to speak to sales from within the product. +## Set up your development environment + +1. Set up GDK with a connection to your local CustomersDot instance. +1. Set up CustomersDot to talk to a staging instance of Platypus. + +1. Set up CustomersDot using the [normal install instructions](https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/staging/doc/setup/installation_steps.md). +1. Set the `CUSTOMER_PORTAL_URL` env var to your local (or ngrok) URL of your CustomersDot instance. +1. Place `export CUSTOMER_PORTAL_URL='https://XXX.ngrok.io/'` in your shell rc script (~/.zshrc or ~/.bash_profile or ~/.bashrc) and restart GDK. +1. Enter the credentials on CustomersDot development to Platypus in your `/config/secrets.yml` and restart. Credentials for the Platypus Staging are in the 1Password Growth vault. The URL for staging is `https://staging.ci.nexus.gitlabenvironment.cloud`. + +```yaml + platypus_url: "<%= ENV['PLATYPUS_URL'] %>" + platypus_client_id: "<%= ENV['PLATYPUS_CLIENT_ID'] %>" + platypus_client_secret: "<%= ENV['PLATYPUS_CLIENT_SECRET'] %>" +``` + +### Set up lead monitoring + +1. Set up access for Platypus Staging `https://staging.ci.nexus.gitlabenvironment.cloud` using the Platypus Staging credentials in the 1Password Growth vault. +1. Set up access for the Marketo sandbox, similar [to this example request](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/13162). + +### Manually test leads + +1. Register a new user with a unique email on your local GitLab instance. +1. Send the PQL lead by submitting your new form or creating a new trial or a new hand raise lead. +1. Use easily identifiable values that can be easily seen in Platypus staging. +1. Observe the entry in the staging instance of Platypus and paste in the merge request comment and mention. + +## Troubleshooting + +- Check the application and Sidekiq logs on `gitlab.com` and CustomersDot to monitor leads. +- Check the `leads` table in CustomersDot. +- Set up staging credentials for Platypus, and track the leads on the [Platypus Dashboard](https://staging.ci.nexus.gitlabenvironment.cloud/admin/queues/queue/new-lead-queue). +- Ask for access to the Marketo Sandbox and validate the leads there, [to this example request](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/13162). + ## Embed a hand-raise lead form [HandRaiseLeadButton](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/assets/javascripts/hand_raise_leads/hand_raise_lead/components/hand_raise_lead_button.vue) is a reusable component that adds a button and a hand-raise modal to any screen. @@ -92,7 +127,7 @@ The flow of a PQL lead is as follows: 1. Marketo does scoring and sends the form to Salesforce. 1. Our Sales team uses Salesforce to connect to the leads. -### Trial lead flow +### Trial lead flow #### Trial lead flow on GitLab.com @@ -131,7 +166,7 @@ sequenceDiagram HostedPlans|CreateTrialService->create_trial_history#: Creates a record in trial_histories table ``` -### Hand raise lead flow +### Hand raise lead flow #### Hand raise flow on GitLab.com @@ -161,11 +196,4 @@ sequenceDiagram Platypus->>Workato: [lead] Workato->>Marketo: [lead] Marketo->>Salesforce(SFDC): [lead] -``` - -## Monitor and manually test leads - -- Check the application and Sidekiq logs on `gitlab.com` and CustomersDot to monitor leads. -- Check the `leads` table in CustomersDot. -- Set up staging credentials for Platypus, and track the leads on the [Platypus Dashboard](https://staging.ci.nexus.gitlabenvironment.cloud/admin/queues/queue/new-lead-queue). -- Ask for access to the Marketo Sandbox and validate the leads there. +``` diff --git a/doc/development/project_templates.md b/doc/development/project_templates.md new file mode 100644 index 00000000000..74ded9c93fc --- /dev/null +++ b/doc/development/project_templates.md @@ -0,0 +1,157 @@ +--- +stage: Manage +group: Workspace +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +--- + +# Contribute to GitLab project templates + +Thanks for considering a contribution to the GitLab +[built-in project templates](../user/project/working_with_projects.md#create-a-project-from-a-built-in-template). + +## Prerequisites + +To add a new or update an existing template, you must have the following tools +installed: + +- `wget` +- `tar` +- `jq` + +## Create a new project + +To contribute a new built-in project template to be distributed with GitLab: + +1. Create a new public project with the project content you'd like to contribute + in a namespace of your choosing. You can [view a working example](https://gitlab.com/gitlab-org/project-templates/dotnetcore). + Projects should be as simple as possible and free of any unnecessary assets or dependencies. +1. When the project is ready for review, [create a new issue](https://gitlab.com/gitlab-org/gitlab/issues) with a link to your project. + In your issue, `@` mention the relevant Backend Engineering Manager and Product + Manager for the [Templates feature](https://about.gitlab.com/handbook/product/categories/#source-code-group). + +## Add the SVG icon to GitLab SVGs + +If the template you're adding has an SVG icon, you need to first add it to +<https://gitlab.com/gitlab-org/gitlab-svgs>: + +1. Follow the steps outlined in the + [GitLab SVGs project](https://gitlab.com/gitlab-org/gitlab-svgs/-/blob/main/README.md#adding-icons-or-illustrations) + and submit a merge request. +1. When the merge request is merged, `gitlab-bot` will pull the new changes in + the `gitlab-org/gitlab` project. +1. You can now continue on the vendoring process. + +## Vendoring process + +To make the project template available when creating a new project, the vendoring +process will have to be completed: + +1. [Export the project](../user/project/settings/import_export.md#export-a-project-and-its-data) + you created in the previous step and save the file as `<name>.tar.gz`, where + `<name>` is the short name of the project. +1. Edit the following files to include the project template. Two types of built-in + templates are available within GitLab: + - **Normal templates**: Available in GitLab Free and above (this is the most common type of built-in template). + See MR [!25318](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) for an example. + + To add a normal template: + + 1. Open `lib/gitlab/project_template.rb` and add details of the template + in the `localized_templates_table` method. In the following example, + the short name of the project is `hugo`: + + ```ruby + ProjectTemplate.new('hugo', 'Pages/Hugo', _('Everything you need to create a GitLab Pages site using Hugo'), 'https://gitlab.com/pages/hugo', 'illustrations/logos/hugo.svg'), + ``` + + If the vendored project doesn't have an SVG icon, omit `, 'illustrations/logos/hugo.svg'`. + + 1. Open `spec/lib/gitlab/project_template_spec.rb` and add the short name + of the template in the `.all` test. + 1. Open `app/assets/javascripts/projects/default_project_templates.js` and + add details of the template. For example: + + ```javascript + hugo: { + text: s__('ProjectTemplates|Pages/Hugo'), + icon: '.template-option .icon-hugo', + }, + ``` + + If the vendored project doesn't have an SVG icon, use `.icon-gitlab_logo` + instead. + + - **Enterprise templates**: Introduced in GitLab 12.10, that are available only in GitLab Premium and above. + See MR [!28187](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28187) for an example. + + To add an Enterprise template: + + 1. Open `ee/lib/ee/gitlab/project_template.rb` and add details of the template + in the `localized_ee_templates_table` method. For example: + + ```ruby + ::Gitlab::ProjectTemplate.new('hipaa_audit_protocol', 'HIPAA Audit Protocol', _('A project containing issues for each audit inquiry in the HIPAA Audit Protocol published by the U.S. Department of Health & Human Services'), 'https://gitlab.com/gitlab-org/project-templates/hipaa-audit-protocol', 'illustrations/logos/asklepian.svg') + ``` + + 1. Open `ee/spec/lib/gitlab/project_template_spec.rb` and add the short name + of the template in the `.all` test. + 1. Open `ee/app/assets/javascripts/projects/default_project_templates.js` and + add details of the template. For example: + + ```javascript + hipaa_audit_protocol: { + text: s__('ProjectTemplates|HIPAA Audit Protocol'), + icon: '.template-option .icon-hipaa_audit_protocol', + }, + ``` + +1. Run the `vendor_template` script. Make sure to pass the correct arguments: + + ```shell + scripts/vendor_template <git_repo_url> <name> <comment> + ``` + +1. Regenerate `gitlab.pot`: + + ```shell + bin/rake gettext:regenerate + ``` + +1. By now, there should be one new file under `vendor/project_templates/` and + 4 changed files. Commit all of them in a new branch and create a merge + request. + +## Test with GDK + +If you are using the GitLab Development Kit (GDK) you must disable `praefect` +and regenerate the Procfile, as the Rake task is not currently compatible with it: + +```yaml +# gitlab-development-kit/gdk.yml +praefect: + enabled: false +``` + +1. Follow the steps described in the [vendoring process](#vendoring-process). +1. Run the following Rake task where `<path>/<name>` is the + name you gave the template in `lib/gitlab/project_template.rb`: + + ```shell + bin/rake gitlab:update_project_templates[<path>/<name>] + ``` + +You can now test to create a new project by importing the new template in GDK. + +## Contribute an improvement to an existing template + +Existing templates are imported from the following groups: + +- [`project-templates`](https://gitlab.com/gitlab-org/project-templates) +- [`pages`](htps://gitlab.com/pages) + +To contribute a change, open a merge request in the relevant project +and mention `@gitlab-org/manage/import/backend` when you are ready for a review. + +Then, if your merge request gets accepted, either [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues) +to ask for it to get updated, or open a merge request updating +the [vendored template](#vendoring-process). diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index 5481da348e8..6751559b2ef 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -17,7 +17,11 @@ You can then connect to this session by using the [pry-shell](https://github.com You can watch [this video](https://www.youtube.com/watch?v=Lzs_PL_BySo), for more information about how to use the `pry-shell`. -## `byebug` vs `binding.pry` +WARNING: +`binding.pry` can occasionally experience autoloading issues and fail during name resolution. +If needed, `binding.irb` can be used instead with a more limited feature set. + +## `byebug` vs `binding.pry` vs `binding.irb` `byebug` has a very similar interface as `gdb`, but `byebug` does not use the powerful Pry REPL. @@ -41,6 +45,12 @@ this document, so for the full documentation head over to the [Pry wiki](https:/ Below are a few features definitely worth checking out, also run `help` in a pry session to see what else you can do. +## `binding.irb` + +As of Ruby 2.7, IRB ships with a simple interactive debugger. + +Check out [the docs](https://ruby-doc.org/stdlib-2.7.0/libdoc/irb/rdoc/Binding.html) for more. + ### State navigation With the [state navigation](https://github.com/pry/pry/wiki/State-navigation) diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index 424c089f88e..17f2fecc1bc 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/scalability.md b/doc/development/scalability.md index fe7063be0e8..4450df0399d 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -36,7 +36,7 @@ application starts, Rails queries the database schema, caching the tables and column types for the data requested. Because of this schema cache, dropping a column or table while the application is running can produce 500 errors to the user. This is why we have a [process for dropping columns and other -no-downtime changes](avoiding_downtime_in_migrations.md). +no-downtime changes](database/avoiding_downtime_in_migrations.md). #### Multi-tenancy diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 10f6c22e54a..8a86a46d1d3 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -203,7 +203,7 @@ Go's [`regexp`](https://pkg.go.dev/regexp) package uses `re2` and isn't vulnerab ### Description -A [Server-side Request Forgery (SSRF)](https://www.hackerone.com/blog-How-To-Server-Side-Request-Forgery-SSRF) is an attack in which an attacker +A [Server-side Request Forgery (SSRF)](https://www.hackerone.com/application-security/how-server-side-request-forgery-ssrf) is an attack in which an attacker is able coerce a application into making an outbound request to an unintended resource. This resource is usually internal. In GitLab, the connection most commonly uses HTTP, but an SSRF can be performed with any protocol, such as @@ -1165,7 +1165,7 @@ func printZipContents(src string) error { ## Time of check to time of use bugs Time of check to time of use, or TOCTOU, is a class of error which occur when the state of something changes unexpectedly partway during a process. -More specifically, it's when the property you checked and validated has changed when you finally get around to using that property. +More specifically, it's when the property you checked and validated has changed when you finally get around to using that property. These types of bugs are often seen in environments which allow multi-threading and concurrency, like filesystems and distributed web applications; these are a type of race condition. TOCTOU also occurs when state is checked and stored, then after a period of time that state is relied on without re-checking its accuracy and/or validity. @@ -1179,7 +1179,7 @@ GitLab-specific example can be found in [this issue](https://gitlab.com/gitlab-o **Example 3:** you need to fetch a remote file, and perform a `HEAD` request to get and validate the content length and content type. When you subsequently make a `GET` request, though, the file delivered is a different size or different file type. (This is stretching the definition of TOCTOU, but things _have_ changed between time of check and time of use). -**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly clicking upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database. +**Example 4:** you allow users to upvote a comment if they haven't already. The server is multi-threaded, and you aren't using transactions or an applicable database index. By repeatedly clicking upvote in quick succession a malicious user is able to add multiple upvotes: the requests arrive at the same time, the checks run in parallel and confirm that no upvote exists yet, and so each upvote is written to the database. Here's some pseudocode showing an example of a potential TOCTOU bug: diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 25e841e113b..ca4a0158051 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -760,7 +760,7 @@ To set up Service Ping locally, you must: 1. Clone and start [Versions Application](https://gitlab.com/gitlab-services/version-gitlab-com). Make sure you run `docker-compose up` to start a PostgreSQL and Redis instance. 1. Point GitLab to the Versions Application endpoint instead of the default endpoint: - 1. Open [service_ping/submit_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L5) in your local and modified `PRODUCTION_URL`. + 1. Open [service_ping/submit_service.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/service_ping/submit_service.rb#L5) locally and modify `STAGING_BASE_URL`. 1. Set it to the local Versions Application URL: `http://localhost:3000/usage_data`. ### Test local setup diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 6878fd1bf28..14bb90537e7 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -48,7 +48,7 @@ make better product decisions. There are several other benefits to enabling Service Ping: - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. -- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_reports.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. - You get better, more proactive support (assuming that our TAMs and support organization used the data to deliver more value). - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. @@ -76,7 +76,7 @@ tier. Users can continue to access the features in a paid tier without sharing u #### Features available in 14.1 and later -1. [Email from GitLab](../../tools/email.md). +1. [Email from GitLab](../../user/admin_area/email_from_gitlab.md). #### Features available in 14.4 and later @@ -582,7 +582,8 @@ ServicePing::SubmitService.new(skip_db_write: true).execute ## Manually upload Service Ping payload -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7388) in GitLab 14.8 with a flag named `admin_application_settings_service_usage_data_center`. Disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83265) in GitLab 14.10. Service Ping payload can be uploaded to GitLab even if your application instance doesn't have access to the internet, or you don't have Service Ping [cron job](#how-service-ping-works) enabled. @@ -598,6 +599,9 @@ To upload payload manually: 1. Select **Choose file** and choose the file from p5. 1. Select **Upload**. +The uploaded file is encrypted and sent using secure [HTTPS protocol](https://en.wikipedia.org/wiki/HTTPS). HTTPS creates a secure +communication channel between web browser and the server, and protects transmitted data against man-in-the-middle attacks. + ## Monitoring Service Ping reporting process state is monitored with [internal SiSense dashboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 6884844da3f..ab3d301908b 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Metrics Dictionary Guide [Service Ping](index.md) metrics are defined in individual YAML files definitions from which the -[Metrics Dictionary](https://metrics.gitlab.com/) is built. +[Metrics Dictionary](https://metrics.gitlab.com/) is built. Currently, the metrics dictionary is built automatically once a day. When a change to a metric is made in a YAML file, you can see the change in the dictionary within 24 hours. This guide describes the dictionary and how it's implemented. ## Metrics Definition and validation @@ -95,7 +95,7 @@ return to the instrumentation and update it. 1. Add the metric instrumentation class to `lib/gitlab/usage/metrics/instrumentations/`. 1. Add the metric logic in the instrumentation class. -1. Run the [metrics YAML generator](metrics_dictionary.md#metrics-definition-and-validation). +1. Run the [metrics YAML generator](metrics_dictionary.md#create-a-new-metric-definition). 1. Use the metric name suggestion to select a suitable metric name. 1. Update the metric's YAML definition with the correct `key_path`. diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index c684d9d12ef..3d56f3e777f 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -24,7 +24,9 @@ This guide describes how to develop Service Ping metrics using metrics instrumen A metric definition has the [`instrumentation_class`](metrics_dictionary.md) field, which can be set to a class. -The defined instrumentation class should have one of the existing metric classes: `DatabaseMetric`, `RedisMetric`, `RedisHLLMetric`, or `GenericMetric`. +The defined instrumentation class should inherit one of the existing metric classes: `DatabaseMetric`, `RedisMetric`, `RedisHLLMetric`, or `GenericMetric`. + +The current convention is that a single instrumentation class corresponds to a single metric. On a rare occasions, there are exceptions to that convention like [Redis metrics](#redis-metrics). To use a single instrumentation class for more than one metric, please reach out to one of the `@gitlab-org/growth/product-intelligence/engineers` members to consult about your case. Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire process of Service Ping generation. @@ -186,3 +188,30 @@ rails generate gitlab:usage_metric CountIssues --type database create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb ``` + +## Migrate Service Ping metrics to instrumentation classes + +This guide describes how to migrate a Service Ping metric from [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) or [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb) to instrumentation classes. + +1. Choose the metric type: + +- [Database metric](#database-metrics) +- [Redis HyperLogLog metrics](#redis-hyperloglog-metrics) +- [Redis metric](#redis-metrics) +- [Generic metric](#generic-metrics) + +1. Determine the location of instrumentation class: either under `ee` or outside `ee`. + +1. [Generate the instrumentation class file](#create-a-new-metric-instrumentation-class). + +1. Fill the instrumentation class body: + + - Add code logic for the metric. This might be similar to the metric implementation in `usage_data.rb`. + - Add tests for the individual metric [`spec/lib/gitlab/usage/metrics/instrumentations/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/usage/metrics/instrumentations). + - Add tests for Service Ping. + +1. [Generate the metric definition file](metrics_dictionary.md#create-a-new-metric-definition). + +1. Remove the code from [`lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb) or [`ee/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/usage_data.rb). + +1. Remove the tests from [`spec/lib/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/usage_data_spec.rb) or [`ee/spec/lib/ee/gitlab/usage_data.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/ee/gitlab/usage_data_spec.rb). diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index a7ecf15a493..844c989c640 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -60,6 +60,8 @@ The correct approach is to add a new metric for GitLab 12.6 release with updated and update existing business analysis artefacts to use `example_metric_without_archived` instead of `example_metric` +Currently, the [Metrics Dictionary](https://metrics.gitlab.com/) is built automatically once a day. When a change to a metric is made in a YAML file, you can see the change in the dictionary within 24 hours. + ## Remove a metric WARNING: @@ -95,6 +97,12 @@ To remove a metric: used to test the [`UsageDataController#create`](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/3760ef28/spec/controllers/usage_data_controller_spec.rb#L75) endpoint, and assure that test suite does not fail when metric that you wish to remove is not included into test payload. +1. Remove data from Redis + + For [Ordinary Redis](implement.md#ordinary-redis-counters) counters remove data stored in Redis. + + - Add a migration to remove the data from Redis for the related Redis keys. For more details, see [this MR example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82604/diffs). + 1. Create an issue in the [GitLab Data Team project](https://gitlab.com/gitlab-data/analytics/-/issues). Ask for confirmation that the metric is not referred to in any SiSense dashboards and diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 770b6650764..15bc01f1270 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -28,4 +28,4 @@ For results about an investigation conducted into an unexpected drop in Service ### Troubleshooting data warehouse layer -Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). +Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index 919f6935139..35f4b88351e 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -156,4 +156,4 @@ end You must rename the queue in a post-deployment migration not in a normal migration. Otherwise, it runs too early, before all the workers that -schedule these jobs have stopped running. See also [other examples](../post_deployment_migrations.md#use-cases). +schedule these jobs have stopped running. See also [other examples](../database/post_deployment_migrations.md#use-cases). diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md index eb406b02a91..0783721e628 100644 --- a/doc/development/single_table_inheritance.md +++ b/doc/development/single_table_inheritance.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -31,8 +31,8 @@ could result in loading unexpected code or associations which may cause unintend side effects or failures during upgrades. ```ruby -class SomeMigration < Gitlab::Database::Migration[1.0] - class Services < ActiveRecord::Base +class SomeMigration < Gitlab::Database::Migration[2.0] + class Services < MigrationRecord self.table_name = 'services' self.inheritance_column = :_type_disabled end diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 6061a1d4cd2..162b77772f9 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -21,8 +21,25 @@ For the recommended frontend tracking implementation, see [Usage recommendations Structured events and page views include the [`gitlab_standard`](schemas.md#gitlab_standard) context, using the `window.gl.snowplowStandardContext` object which includes [default data](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/views/layouts/_snowplow.html.haml) -as base. This object can be modified for any subsequent structured event fired, -although it's not recommended. +as base: + +| Property | Example | +| -------- | ------- | +| `context_generated_at` | `"2022-01-01T01:00:00.000Z"` | +| `environment` | `"production"` | +| `extra` | `{}` | +| `namespace_id` | `123` | +| `plan` | `"gold"` | +| `project_id` | `456` | +| `source` | `"gitlab-rails"` | +| `user_id` | `789`* | + +_\* Undergoes a pseudonymization process at the collector level._ + +These properties [are overriden](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/tracking/get_standard_context.js) +with frontend-specific values, like `source` (`gitlab-javascript`), `google_analytics_id` +and the custom `extra` object. You can modify this object for any subsequent +structured event that fires, although this is not recommended. Tracking implementations must have an `action` and a `category`. You can provide additional properties from the [structured event taxonomy](index.md#structured-event-taxonomy), in @@ -396,13 +413,13 @@ Use the following arguments: |------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| | `category` | String | | Area or aspect of the application. For example, `HealthCheckController` or `Lfs::FileTransformer`. | | `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | -| `label` | String | nil | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | -| `property` | String | nil | Any additional property of the element, or object being acted on. | -| `value` | Numeric | nil | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | -| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | -| `project` | Project | nil | The project associated with the event. | -| `user` | User | nil | The user associated with the event. | -| `namespace` | Namespace | nil | The namespace associated with the event. | +| `label` | String | `nil` | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | +| `property` | String | `nil` | Any additional property of the element, or object being acted on. | +| `value` | Numeric | `nil` | Describes a numeric value (decimal) directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| `context` | Array\[SelfDescribingJSON\] | `nil` | An array of custom contexts to send with this event. Most events should not have any custom contexts. | +| `project` | Project | `nil` | The project associated with the event. | +| `user` | User | `nil` | The user associated with the event. This value undergoes a pseudonymization process at the collector level. | +| `namespace` | Namespace | `nil` | The namespace associated with the event. | | `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | ### Unit testing diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index 29f4514a21e..9b684757fe1 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -150,6 +150,23 @@ ORDER BY page_view_start DESC LIMIT 100 ``` +#### Top 20 users who fired `reply_comment_button` in the last 30 days + +```sql +SELECT + count(*) as hits, + se_action, + se_category, + gsc_pseudonymized_user_id +FROM legacy.snowplow_gitlab_events_30 +WHERE + se_label = 'reply_comment_button' + AND gsc_pseudonymized_user_id IS NOT NULL +GROUP BY gsc_pseudonymized_user_id, se_category, se_action +ORDER BY count(*) DESC +LIMIT 20 +``` + #### Query JSON formatted data ```sql diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 63864c9329b..4066151600d 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -10,17 +10,18 @@ This page provides Snowplow schema reference for GitLab events. ## `gitlab_standard` -We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. +We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) for structured events and page views. The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) -class represents this schema in the application. Some properties are automatically populated for [frontend](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/views/layouts/_snowplow.html.haml) -events. +class represents this schema in the application. Some properties are +[automatically populated for frontend events](implementation.md#snowplow-javascript-frontend-tracking), +and can be [provided manually for backend events](implementation.md#implement-ruby-backend-tracking). | Field Name | Required | Default value | Type | Description | |----------------|:-------------------:|-----------------------|--|---------------------------------------------------------------------------------------------| | `project_id` | **{dotted-circle}** | Current project ID * | integer | | | `namespace_id` | **{dotted-circle}** | Current group/namespace ID * | integer | | -| `user_id` | **{dotted-circle}** | Current user ID * | integer | User database record ID attribute. This file undergoes a pseudonymization process at the collector level. | +| `user_id` | **{dotted-circle}** | Current user ID * | integer | User database record ID attribute. This value undergoes a pseudonymization process at the collector level. | | `context_generated_at` | **{dotted-circle}** | Current timestamp | string (date time format) | Timestamp indicating when context was generated. | | `environment` | **{check-circle}** | Current environment | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | | `source` | **{check-circle}** | Event source | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md index 75c8b306a67..47d775d89aa 100644 --- a/doc/development/snowplow/troubleshooting.md +++ b/doc/development/snowplow/troubleshooting.md @@ -28,7 +28,7 @@ While on CloudWatch dashboard set time range to last 4 weeks, to get better pict Drop occurring at application layer can be symptom of some issue, but it might be also a result of normal application lifecycle, intended changes done to product intelligence or experiments tracking or even a result of a public holiday in some regions of the world with a larger user-base. To verify if there is an underlying problem to solve, you can check following things: -1. Check `about.gitlab.com` website traffic on [Google Analytics](https://analytics.google.com/) to verify if some public holiday might impact overall use of GitLab system +1. Check `about.gitlab.com` website traffic on [Google Analytics](https://analytics.google.com/analytics/web/) to verify if some public holiday might impact overall use of GitLab system 1. You may require to open an access request for Google Analytics access first eg: [access request internal issue](https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/1772) 1. Plot `select date(dvce_created_tstamp) , event , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-10' group by 1 , 2 order by 1 , 2` in SiSense to see what type of events was responsible for drop 1. Plot `select date(dvce_created_tstamp) ,se_category , count(*) from legacy.snowplow_unnested_events_90 where dvce_created_tstamp > '2021-06-15' and dvce_created_tstamp < '2021-07-31' and event = 'struct' group by 1 , 2 order by 1, 2` what events recorded the biggest drops in suspected category @@ -47,4 +47,4 @@ Already conducted investigations: ### Troubleshooting data warehouse layer -Reach out to [Data team](https://about.gitlab.com/handbook/business-technology/data-team) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us) +Reach out to [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us) diff --git a/doc/development/spam_protection_and_captcha/graphql_api.md b/doc/development/spam_protection_and_captcha/graphql_api.md index b47e3f84320..e3f4e9069e5 100644 --- a/doc/development/spam_protection_and_captcha/graphql_api.md +++ b/doc/development/spam_protection_and_captcha/graphql_api.md @@ -13,28 +13,27 @@ related to changing a model's confidential/public flag. ## Add support to the GraphQL mutations -This implementation is very similar to the controller implementation. You create a `spam_params` -instance based on the request, and pass it to the relevant Service class constructor. +The main steps are: -The three main differences from the controller implementation are: +1. Use `include Mutations::SpamProtection` in your mutation. +1. Create a `spam_params` instance based on the request. Obtain the request from the context + via `context[:request]` when creating the `SpamParams` instance. +1. Pass `spam_params` to the relevant Service class constructor. +1. After you create or update the `Spammable` model instance, call `#check_spam_action_response!` + and pass it the model instance. This call: + 1. Performs the necessary spam checks on the model. + 1. If spam is detected: + - Raises a `GraphQL::ExecutionError` exception. + - Includes the relevant information added as error fields to the response via the `extensions:` parameter. + For more details on these fields, refer to the section in the GraphQL API documentation on + [Resolve mutations detected as spam](../../api/graphql/index.md#resolve-mutations-detected-as-spam). -1. Use `include Mutations::SpamProtection` instead of `...JsonFormatActionsSupport`. -1. Obtain the request from the context via `context[:request]` when creating the `SpamParams` - instance. -1. After you create or updated the `Spammable` model instance, call `#check_spam_action_response!` - and pass it the model instance. This call will: - 1. Perform the necessary spam checks on the model. - 1. If spam is detected: - - Raise a `GraphQL::ExecutionError` exception. - - Include the relevant information added as error fields to the response via the `extensions:` parameter. - For more details on these fields, refer to the section on - [Spam and CAPTCHA support in the GraphQL API](../../api/graphql/index.md#resolve-mutations-detected-as-spam). - - NOTE: - If you use the standard ApolloLink or Axios interceptor CAPTCHA support described - above, the field details are unimportant. They become important if you - attempt to use the GraphQL API directly to process a failed check for potential spam, and - resubmit the request with a solved CAPTCHA response. + NOTE: + If you use the standard ApolloLink or Axios interceptor CAPTCHA support described + above, you can ignore the field details, because they are handled + automatically. They become relevant if you attempt to use the GraphQL API directly to + process a failed check for potential spam, and resubmit the request with a solved + CAPTCHA response. For example: @@ -57,10 +56,13 @@ module Mutations widget = service_response.payload[:widget] check_spam_action_response!(widget) - # If possible spam wasdetected, an exception would have been thrown by + # If possible spam was detected, an exception would have been thrown by # `#check_spam_action_response!`, so the normal resolve return logic can follow below. end end end end ``` + +Refer to the [Exploratory Testing](exploratory_testing.md) section for instructions on how to test +CAPTCHA behavior in the GraphQL API. diff --git a/doc/development/spam_protection_and_captcha/index.md b/doc/development/spam_protection_and_captcha/index.md index 9b195df536d..dbe8c4aa4e9 100644 --- a/doc/development/spam_protection_and_captcha/index.md +++ b/doc/development/spam_protection_and_captcha/index.md @@ -16,7 +16,7 @@ To add this support, you must implement the following areas as applicable: 1. [Model and Services](model_and_services.md): The basic prerequisite changes to the backend code which are required to add spam or CAPTCHA API and UI support for a feature which does not yet have support. -1. REST API (Supported, documentation coming soon): The changes needed to add +1. [REST API](rest_api.md): The changes needed to add spam or CAPTCHA support to Grape REST API endpoints. Refer to the related [REST API documentation](../../api/index.md#resolve-requests-detected-as-spam). 1. [GraphQL API](graphql_api.md): The changes needed to add spam or CAPTCHA support to GraphQL diff --git a/doc/development/spam_protection_and_captcha/rest_api.md b/doc/development/spam_protection_and_captcha/rest_api.md new file mode 100644 index 00000000000..ad74977eb67 --- /dev/null +++ b/doc/development/spam_protection_and_captcha/rest_api.md @@ -0,0 +1,90 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# REST API spam protection and CAPTCHA support + +If the model can be modified via the REST API, you must also add support to all of the +relevant API endpoints which may modify spammable or spam-related attributes. This +definitely includes the `POST` and `PUT` mutations, but may also include others, such as those +related to changing a model's confidential/public flag. + +## Add support to the REST endpoints + +The main steps are: + +1. Add `helpers SpammableActions::CaptchaCheck::RestApiActionsSupport` in your `resource`. +1. Create a `spam_params` instance based on the request. +1. Pass `spam_params` to the relevant Service class constructor. +1. After you create or update the `Spammable` model instance, call `#check_spam_action_response!`, + save the created or updated instance in a variable. +1. Identify the error handling logic for the `failure` case of the request, + when create or update was not successful. These indicate possible spam detection, + which adds an error to the `Spammable` instance. + The error is usually similar to `render_api_error!` or `render_validation_error!`. +1. Wrap the existing error handling logic in a + `with_captcha_check_rest_api(spammable: my_spammable_instance)` call, passing the `Spammable` + model instance you saved in a variable as the `spammable:` named argument. This call will: + 1. Perform the necessary spam checks on the model. + 1. If spam is detected: + - Raise a Grape `#error!` exception with a descriptive spam-specific error message. + - Include the relevant information added as error fields to the response. + For more details on these fields, refer to the section in the REST API documentation on + [Resolve requests detected as spam](../../api/index.md#resolve-requests-detected-as-spam). + + NOTE: + If you use the standard ApolloLink or Axios interceptor CAPTCHA support described + above, you can ignore the field details, because they are handled + automatically. They become relevant if you attempt to use the GraphQL API directly to + process a failed check for potential spam, and resubmit the request with a solved + CAPTCHA response. + +Here is an example for the `post` and `put` actions on the `snippets` resource: + +```ruby +module API + class Snippets < ::API::Base + #... + resource :snippets do + # This helper provides `#with_captcha_check_rest_api` + helpers SpammableActions::CaptchaCheck::RestApiActionsSupport + + post do + #... + spam_params = ::Spam::SpamParams.new_from_request(request: request) + service_response = ::Snippets::CreateService.new(project: nil, current_user: current_user, params: attrs, spam_params: spam_params).execute + snippet = service_response.payload[:snippet] + + if service_response.success? + present snippet, with: Entities::PersonalSnippet, current_user: current_user + else + # Wrap the normal error response in a `with_captcha_check_rest_api(spammable: snippet)` block + with_captcha_check_rest_api(spammable: snippet) do + # If possible spam was detected, an exception would have been thrown by + # `#with_captcha_check_rest_api` for Grape to handle via `error!` + render_api_error!({ error: service_response.message }, service_response.http_status) + end + end + end + + put ':id' do + #... + spam_params = ::Spam::SpamParams.new_from_request(request: request) + service_response = ::Snippets::UpdateService.new(project: nil, current_user: current_user, params: attrs, spam_params: spam_params).execute(snippet) + + snippet = service_response.payload[:snippet] + + if service_response.success? + present snippet, with: Entities::PersonalSnippet, current_user: current_user + else + # Wrap the normal error response in a `with_captcha_check_rest_api(spammable: snippet)` block + with_captcha_check_rest_api(spammable: snippet) do + # If possible spam was detected, an exception would have been thrown by + # `#with_captcha_check_rest_api` for Grape to handle via `error!` + render_api_error!({ error: service_response.message }, service_response.http_status) + end + end + end +``` diff --git a/doc/development/spam_protection_and_captcha/web_ui.md b/doc/development/spam_protection_and_captcha/web_ui.md index 6aa01f401bd..9aeb9e96d44 100644 --- a/doc/development/spam_protection_and_captcha/web_ui.md +++ b/doc/development/spam_protection_and_captcha/web_ui.md @@ -37,7 +37,7 @@ additional fields being added to the models. Instead, communication is handled: The spam and CAPTCHA-related logic is also cleanly abstracted into reusable modules and helper methods which can wrap existing logic, and only alter the existing flow if potential spam is detected or a CAPTCHA display is needed. This approach allows the spam and CAPTCHA -support to be easily added to new areas of the application with minimal changes to +support to be added to new areas of the application with minimal changes to existing logic. In the case of the frontend, potentially **zero** changes are needed! On the frontend, this is handled abstractly and transparently using `ApolloLink` for Apollo, and an @@ -75,7 +75,7 @@ sequenceDiagram The backend is also cleanly abstracted via mixin modules and helper methods. The three main changes required to the relevant backend controller actions (normally just `create`/`update`) are: -1. Create a `SpamParams` parameter object instance based on the request, using the simple static +1. Create a `SpamParams` parameter object instance based on the request, using the static `#new_from_request` factory method. This method takes a request, and returns a `SpamParams` instance. 1. Pass the created `SpamParams` instance as the `spam_params` named argument to the Service class constructor, which you should have already added. If the spam check indicates diff --git a/doc/development/sql.md b/doc/development/sql.md index e2208caf35a..4b6153b7205 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -254,13 +254,13 @@ of records plucked. `MAX_PLUCK` defaults to `1_000` in `ApplicationRecord`. ## Inherit from ApplicationRecord -Most models in the GitLab codebase should inherit from `ApplicationRecord`, -rather than from `ActiveRecord::Base`. This allows helper methods to be easily -added. +Most models in the GitLab codebase should inherit from `ApplicationRecord` +or `Ci::ApplicationRecord` rather than from `ActiveRecord::Base`. This allows +helper methods to be easily added. An exception to this rule exists for models created in database migrations. As these should be isolated from application code, they should continue to subclass -from `ActiveRecord::Base`. +from `MigrationRecord` which is available only in migration context. ## Use UNIONs @@ -376,7 +376,7 @@ Explicit column list definition: ```ruby # Good, the SELECT columns are consistent -columns = User.cached_column_names # The helper returns fully qualified (table.column) column names (Arel) +columns = User.cached_column_list # The helper returns fully qualified (table.column) column names (Arel) scope1 = User.select(*columns).where(id: [1, 2, 3]) # selects the columns explicitly scope2 = User.select(*columns).where(id: [10, 11, 12]) # uses SELECT users.* diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 744d049f72d..8e3e6982430 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -1,273 +1,11 @@ --- -stage: Platforms -group: Scalability -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'stage_group_observability/index.md' +remove_date: '2022-06-15' --- -# Dashboards for stage groups +This document was moved to [another location](stage_group_observability/index.md). -## Introduction - -Observability is about bringing visibility into a system to see and understand the state of each component, with context, to support performance tuning and debugging. To run a SaaS platform at scale, a rich and detailed observability platform is a necessity. We have a set of monitoring dashboards designed for [each stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). - -These dashboards are designed to give an insight, to everyone working in a feature category, into how their code operates at GitLab.com scale. They are grouped per stage group to show the impact of feature/code changes, deployments, and feature-flag toggles. - -Each stage group has a dashboard consisting of metrics at the application level, such as Rails Web Requests, Rails API Requests, Sidekiq Jobs, and so on. The metrics in each dashboard are filtered and accumulated based on the [GitLab product categories](https://about.gitlab.com/handbook/product/categories/) and [feature categories](feature_categorization/index.md). - -The list of dashboards for each stage group is accessible at <https://dashboards.gitlab.net/dashboards/f/stage-groups/stage-groups> (GitLab team members only), or at [the public mirror](https://dashboards.gitlab.com/dashboards?tag=feature_category&tag=stage-groups) (accessible to everyone with a GitLab.com account, with some limitations). - -The dashboards for stage groups are at a very early stage. All contributions are welcome. If you have any questions or suggestions, please submit an issue in the [Scalability Team issues tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/new). - -## Dashboard content - -### Error budget - -Read more about how we are using error budgets overall in our -[handbook](https://about.gitlab.com/handbook/engineering/error-budgets/). - -By default, the first row of panels on the dashboard will show the [error -budget for the stage -group](https://about.gitlab.com/handbook/engineering/error-budgets/#budget-spend-by-stage-group). This -row shows how the features owned by -the group are contributing to our [overall -availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability). - -The budget is always aggregated over the 28 days before the [time -selected on the dashboard](#time-range-controls). - -We're currently displaying the information in 2 formats: - -1. Availability: This number can be compared to GitLab.com's overall - availability target of 99.95% uptime. -1. Budget Spent: This shows the time over the past 28 days that - features owned by the group have not been performing adequately. - -The budget is calculated based on indicators per component. Each -component can have 2 indicators: - -1. [Apdex](https://en.wikipedia.org/wiki/Apdex): The rate of - operations that performed adequately. - - The threshold for 'performed adequately' is stored in our [metrics - catalog](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog) - and depends on the service in question. For the Puma (Rails) - component of the - [API](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/api.jsonnet#L127), - [Git](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/git.jsonnet#L216), - and - [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) - services, that threshold is **5 seconds**. - - We're working on making this target configurable per endpoint in [this - project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). Learn - how to - [customize the request Apdex](application_slis/rails_request_apdex.md), this new Apdex - measurement is not yet part of the error budget. - - For Sidekiq job execution, the threshold depends on the - [job urgency](sidekiq/worker_attributes.md#job-urgency). It is - [currently](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L25-38) - **10 seconds** for high-urgency jobs and **5 minutes** for other - jobs. - - Some stage groups may have more services than these, and the - thresholds for those will be in the metrics catalog as well. - -1. Error rate: The rate of operations that had errors. - -The calculation to a ratio then happens as follows: - -```math -\frac {operations\_meeting\_apdex + (total\_operations - operations\_with\_errors)} {total\_apdex\_measurements + total\_operations} -``` - -### Check where budget is being spent - -The row below the error budget row is collapsed by default. Expanding -it shows which component and violation type had the most offending -operations in the past 28 days. - - - -The first panel on the left shows a table with the number of errors per -component. Digging into the first row in that table is going to have -the biggest impact on the budget spent. - -Commonly, the components spending most of the budget are Sidekiq or Puma. The panel in -the center explains what these violation types mean, and how to dig -deeper in the logs. - -The panel on the right provides links to Kibana that should reveal -which endpoints or Sidekiq jobs are causing the errors. - -To learn how to use these panels and logs for -determining which Rails endpoints are slow, -see the [Error Budget Attribution for Purchase group](https://youtu.be/M9u6unON7bU) video. - -Other components visible in the table come from -[service level indicators](https://sre.google/sre-book/service-level-objectives/) (SLIs) defined -in the [metrics -catalog](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/README.md). - -For those types of failures, you can follow the link to the service -dashboard linked from the `type` column. The service dashboard -contains a row specifically for the SLI that is causing the budget -spent, with useful links to the logs and a description of what the -component means. For example, see the `server` component of the -`web-pages` service: - - - -## Usage of the dashboard - -Inside a stage group dashboard, there are some notable components. Let's take the [Source Code group's dashboard](https://dashboards.gitlab.net/d/stage-groups-source_code/stage-groups-group-dashboard-create-source-code?orgId=1) as an example. - -### Time range controls - - - -- By default, all the times are in UTC time zone. [We use UTC when communicating in Engineering](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). -- All metrics recorded in the GitLab production system have [1-year retention](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/-/blob/31526b03fef823e2f9b3cda7c75dcd28a12418a3/attributes/prometheus.rb#L40). -- Alternatively, you can zoom in or filter the time range directly on a graph. See the [Grafana Time Range Controls](https://grafana.com/docs/grafana/latest/dashboards/time-range-controls/) documentation for more information. - -### Filters and annotations - -In each dashboard, there are two filters and some annotations switches on the top of the page. [Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/annotations/) mark some special events, which are meaningful to development and operational activities, directly on the graphs. - - - -| Name | Type | Description | -| ---- | ---- | ----------- | -| `PROMETHEUS_DS` | filter | Filter the selective [Prometheus data sources](https://about.gitlab.com/handbook/engineering/monitoring/#prometheus). The default value is `Global`, which aggregates the data from all available data sources. Most of the time, you don't need to care about this filter. | -| `environment` | filter | Filter the environment the metrics are fetched from. The default setting is production (`gprd`). Check [Production Environment mapping](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#environments) for other possibilities. | -| `deploy` | annotation | Mark a deployment event on the GitLab.com SaaS platform. | -| `canary-deploy` | annotation | Mark a [canary deployment](https://about.gitlab.com/handbook/engineering/#canary-testing) event on the GitLab.com SaaS platform. | -| `feature-flags` | annotation | Mark the time point where a feature flag is updated.| - -This is an example of a feature flag annotation displayed on a dashboard panel. - - - -### Metrics panels - - - -Although most of the metrics displayed in the panels are self-explanatory in their title and nearby description, note the following: - -- The events are counted, measured, accumulated, then collected, and stored as [time series](https://prometheus.io/docs/concepts/data_model/). The data are calculated using statistical methods to produce metrics. It means that metrics are approximately correct and meaningful over a time period. They help you have an overview of the stage of a system over time. They are not meant to give you precise numbers of a discrete event. If you need a higher level of accuracy, please look at another monitoring tool like [logs](https://about.gitlab.com/handbook/engineering/monitoring/#logs). Please read the following examples for more explanations. -- All the rate metrics' units are `requests per second`. The default aggregate time frame is 1 minute. For example, a panel shows the requests per second number at `2020-12-25 00:42:00` is `34.13`. It means at the minute 42 (from `2020-12-25 00:42:00` to `2020-12-25 00:42:59` ), there are approximately `34.13 * 60 = ~ 2047` requests processed by the web servers. -- You may encounter some gotchas related to decimal fraction and rounding up frequently, especially in low-traffic cases. For example, the error rate of `RepositoryUpdateMirrorWorker` at `2020-12-25 02:04:00` is `0.07`, equivalent to `4.2` jobs per minute. The raw result is `0.06666666667`, equivalent to 4 jobs per minute. -- All the rate metrics are more accurate when the data is big enough. The default floating-point precision is 2. In some extremely low panels, you would see `0.00` although there is still some real traffic. - -To inspect the raw data of the panel for further calculation, click on the Inspect button from the dropdown menu of a panel. Queries, raw data, and panel JSON structure are available. Read more at [Grafana panel inspection](https://grafana.com/docs/grafana/latest/panels/inspect-panel/). - -All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. Grafana consumes the data returned from queries to backend Prometheus data source, then presents them under different visualizations. The stage group dashboards are built to serve the most common use cases with a limited set of filters, and pre-built queries. Grafana provides a way to explore and visualize the metrics data with [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/). This would require some knowledge about [Prometheus PromQL query language](https://prometheus.io/docs/prometheus/latest/querying/basics/). - -## How to debug with the dashboards - -- A team member in the Code Review group has merged an MR which got deployed to production. -- To verify the deployment, we can check the [Code Review group's dashboard](https://dashboards.gitlab.net/d/stage-groups-code_review/stage-groups-group-dashboard-create-code-review?orgId=1). -- Sidekiq Error Rate panel shows an elevated error rate, specifically `UpdateMergeRequestsWorker`. - -  - -- If we click on `Kibana: Kibana Sidekiq failed request logs` link in the Extra links session, we can filter for `UpdateMergeRequestsWorker`, and read through the logs. - -  - -- [Sentry](https://sentry.gitlab.net/gitlab/gitlabcom/) gives us a way to find the exception where we can filter by transaction type and correlation_id from a Kibana's result item. - -  - -- A precise exception, including a stack trace, job arguments, and other information, should now appear. Happy debugging! - -## How to customize the dashboard - -All Grafana dashboards at GitLab are generated from the [Jsonnet files](https://github.com/grafana/grafonnet-lib) stored in [the runbook project](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards). Particularly, the stage group dashboards definitions are stored in [/dashboards/stage-groups](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards/stage-groups) subfolder in the Runbook. By convention, each group has a corresponding Jsonnet file. The dashboards are synced with GitLab [stage group data](https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/master/data/stages.yml) every month. Expansion and customization are one of the key principles used when we designed this system. To customize your group's dashboard, you need to edit the corresponding file and follow the [Runbook workflow](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards#dashboard-source). The dashboard is updated after the MR is merged. Looking at an autogenerated file, for example, [`product_planning.dashboard.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/stage-groups/product_planning.dashboard.jsonnet): - -```jsonnet -// This file is autogenerated using scripts/update_stage_groups_dashboards.rb -// Please feel free to customize this file. -local stageGroupDashboards = import './stage-group-dashboards.libsonnet'; - -stageGroupDashboards.dashboard('product_planning') -.stageGroupDashboardTrailer() -``` - -We provide basic customization to filter out the components essential to your group's activities. By default, only the `web`, `api`, and `sidekiq` components are available in the dashboard, while `git` is hidden. See [how to enable available components and optional graphs](#optional-graphs). - -You can also append further information or custom metrics to a dashboard. This is an example that adds some links and a total request rate on the top of the page: - -```jsonnet -local stageGroupDashboards = import './stage-group-dashboards.libsonnet'; -local grafana = import 'github.com/grafana/grafonnet-lib/grafonnet/grafana.libsonnet'; -local basic = import 'grafana/basic.libsonnet'; - -stageGroupDashboards.dashboard('source_code') -.addPanel( - grafana.text.new( - title='Group information', - mode='markdown', - content=||| - Useful link for the Source Code Management group dashboard: - - [Issue list](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name%5B%5D=repository) - - [Epic list](https://gitlab.com/groups/gitlab-org/-/epics?label_name[]=repository) - |||, - ), - gridPos={ x: 0, y: 0, w: 24, h: 4 } -) -.addPanel( - basic.timeseries( - title='Total Request Rate', - yAxisLabel='Requests per Second', - decimals=2, - query=||| - sum ( - rate(gitlab_transaction_duration_seconds_count{ - env='$environment', - environment='$environment', - feature_category=~'source_code_management', - }[$__interval]) - ) - ||| - ), - gridPos={ x: 0, y: 0, w: 24, h: 7 } -) -.stageGroupDashboardTrailer() -``` - - - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -If you want to see the workflow in action, we've recorded a pairing session on customizing a dashboard, -available on [GitLab Unfiltered](https://youtu.be/shEd_eiUjdI). - -For deeper customization and more complicated metrics, visit the [Grafonnet lib](https://github.com/grafana/grafonnet-lib) project and the [GitLab Prometheus Metrics](../administration/monitoring/prometheus/gitlab_metrics.md#gitlab-prometheus-metrics) documentation. - -### Optional Graphs - -Some Graphs aren't relevant for all groups, so they aren't added to -the dashboard by default. They can be added by customizing the -dashboard. - -By default, only the `web`, `api`, and `sidekiq` metrics are -shown. If you wish to see the metrics from the `git` fleet (or any -other component that might be added in the future), this could be -configured as follows: - -```jsonnet -stageGroupDashboards -.dashboard('source_code', components=stageGroupDashboards.supportedComponents) -.stageGroupDashboardTrailer() -``` - -If your group is interested in Sidekiq job durations and their -thresholds, these graphs can be added by calling the -`.addSidekiqJobDurationByUrgency` function: - -```jsonnet -stageGroupDashboards -.dashboard('access') -.addSidekiqJobDurationByUrgency() -.stageGroupDashboardTrailer() -``` +<!-- This redirect file can be deleted after <2022-06-15>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/stage_group_observability/dashboards/error_budget_detail.md b/doc/development/stage_group_observability/dashboards/error_budget_detail.md new file mode 100644 index 00000000000..19f98d404e7 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/error_budget_detail.md @@ -0,0 +1,127 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Error budget detail dashboard + +With error budget detailed dashboards you can explore the error budget +spent at specific moments in time. By default, the dashboard shows +the past 28 days. You can adjust it with the [time range controls](index.md#time-range-controls) +or by selecting a range on one of the graphs. + +This dashboard is the same kind of dashboard we use for service level +monitoring. For example, see the +[overview dashboard for the web service](https://dashboards.gitlab.net/d/web-main) (GitLab internal). + +## Error budget panels + +On top of each dashboard, there's the same panel with the [error budget](../index.md#error-budget). +Here, the time based targets adjust depending on the range. +For example, while the budget was 20 minutes per 28 days, it is only 1/4 of that for 7 days: + + + +Also, keep in mind that Grafana rounds the numbers. In this example the +total time spent is 5 minutes and 24 seconds, so 24 seconds over +budget. + +The attribution panels also show only failures that occurred +within the selected range. + +These two panels represent a view of the "official" error budget: they +take into account if an SLI was ignored. +The [attribution panels](../index.md#check-where-budget-is-being-spent) show which components +contributed the most over the selected period. + +The panels below take into account all SLIs that contribute to GitLab.com availability. +This includes SLIs that are ignored for the official error budget. + +## Time series for aggregations + +The time series panels for aggregations all contain three panels: + +- Apdex: the [Apdex score](https://en.wikipedia.org/wiki/Apdex) for one or more SLIs. Higher score is better. +- Error Ratio: the error ratio for one or more SLIs. Lower is better. +- Requests Per Second: the number of operations per second. Higher means a bigger impact on the error budget. + +The Apdex and error-ratio panels also contain two alerting thresholds: + +- The one-hour threshold: the fast burn rate. + + When this line is crossed, we've spent 2% of our monthly budget in the last hour. + +- The six-hour threshold: the slow burn rate. + + When this line is crossed, we've spent 2% of our budget in the last six hours. + +If there is no error-ratio or Apdex for a certain SLI, the panel is hidden. + +Read more about these alerting windows in +[Google SRE workbook](https://sre.google/workbook/alerting-on-slos/#recommended_time_windows_and_burn_rates_f). + +We don't have alerting on these metrics for stage groups. +This work is being discussed in [epic 615](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/615). +If this is something you would like for your group, let us know there. + +### Stage group aggregation + + + +The stage group aggregation shows a graph with the Apdex and errors +portion of the error budget over time. The lower a dip in the Apdex +graph or the higher a peak on the error ratio graph, the more budget +was spent at that moment. + +The third graph shows the sum of all the request rates for all +SLIs. Higher means there was more traffic. + +To zoom in on a particular moment where a lot of budget was spent, select the appropriate time in +the graph. + +### Service-level indicators + + + +This time series shows a breakdown of each SLI that could be contributing to the +error budget for a stage group. Similar to the stage group +aggregation, it contains an Apdex score, error ratio, and request +rate. + +Here we also display an explanation panel, describing the SLI and +linking to other monitoring tools. The links to logs (📖) or +visualizations (📈) in Kibana are scoped to the feature categories +for your stage group, and limited to the range selected. Keep in mind +that we only keep logs in Kibana for seven days. + +In the graphs, there is a single line per service. In the previous example image, +`rails_requests` is an SLI for the `web`, `api` and `git` services. + +Sidekiq is not included in this dashboard. We're tracking this in +[epic 700](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/700). + +### SLI detail + + + +The SLI details row shows a breakdown of a specific SLI based on the +labels present on the source metrics. + +For example, in the previous image, the `rails_requests` SLI has an `endpoint_id` label. +We can show how much a certain endpoint was requested (RPS), and how much it contributed to the error +budget spend. + +For Apdex we show the **Apdex Attribution** panel. The more prominent +color is the one that contributed most to the spend. To see the +top spending endpoint over the entire range, sort by the average. + +For error ratio we show an error rate. To see which label contributed most to the spend, sort by the +average. + +We don't have endpoint information available for Rails errors. This work is being planned in +[epic 663](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/663). + +The number of series to be loaded in the SLI details graphs is very +high when compared to the other aggregations. Because of this, it's not possible to +load more than a few days' worth of data. diff --git a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_7d_budget.png b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_7d_budget.png Binary files differnew file mode 100644 index 00000000000..1b2996d7d26 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_7d_budget.png diff --git a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli.png b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli.png Binary files differnew file mode 100644 index 00000000000..0472e35b0cb --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli.png diff --git a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png Binary files differnew file mode 100644 index 00000000000..99530886ae9 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_sli_detail.png diff --git a/doc/development/stage_group_observability/dashboards/img/error_budget_detail_stage_group_aggregation.png b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_stage_group_aggregation.png Binary files differnew file mode 100644 index 00000000000..d679637dcc4 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/img/error_budget_detail_stage_group_aggregation.png diff --git a/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_28d_budget.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_28d_budget.png Binary files differnew file mode 100644 index 00000000000..eb164dd3f68 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_28d_budget.png diff --git a/doc/development/img/stage_group_dashboards_annotation.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_annotation.png Binary files differindex 3776d87e5bb..3776d87e5bb 100644 --- a/doc/development/img/stage_group_dashboards_annotation.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_annotation.png diff --git a/doc/development/img/stage_group_dashboards_debug_1.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_1.png Binary files differindex 309fad89120..309fad89120 100644 --- a/doc/development/img/stage_group_dashboards_debug_1.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_1.png diff --git a/doc/development/img/stage_group_dashboards_debug_2.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_2.png Binary files differindex 2aad9ab5592..2aad9ab5592 100644 --- a/doc/development/img/stage_group_dashboards_debug_2.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_2.png diff --git a/doc/development/img/stage_group_dashboards_debug_3.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_3.png Binary files differindex 38647410ffd..38647410ffd 100644 --- a/doc/development/img/stage_group_dashboards_debug_3.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_debug_3.png diff --git a/doc/development/img/stage_group_dashboards_filters.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_filters.png Binary files differindex 27a836bc36d..27a836bc36d 100644 --- a/doc/development/img/stage_group_dashboards_filters.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_filters.png diff --git a/doc/development/img/stage_group_dashboards_metrics.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_metrics.png Binary files differindex 6b6faff6e3b..6b6faff6e3b 100644 --- a/doc/development/img/stage_group_dashboards_metrics.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_metrics.png diff --git a/doc/development/img/stage_group_dashboards_time_customization.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_time_customization.png Binary files differindex 49e61183b7c..49e61183b7c 100644 --- a/doc/development/img/stage_group_dashboards_time_customization.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_time_customization.png diff --git a/doc/development/img/stage_group_dashboards_time_filter.png b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_time_filter.png Binary files differindex 81a3dc789f1..81a3dc789f1 100644 --- a/doc/development/img/stage_group_dashboards_time_filter.png +++ b/doc/development/stage_group_observability/dashboards/img/stage_group_dashboards_time_filter.png diff --git a/doc/development/stage_group_observability/dashboards/index.md b/doc/development/stage_group_observability/dashboards/index.md new file mode 100644 index 00000000000..f4e646c8634 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/index.md @@ -0,0 +1,70 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Dashboards for stage groups + +We generate a lot of dashboards acting as windows to the metrics we +use to monitor GitLab.com. Most of our dashboards are generated from +Jsonnet in the +[runbooks repository](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards#dashboard-source). +Anyone can contribute to these, adding new dashboards or modifying +existing ones. + +When adding new dashboards for your stage groups, tagging them with +`stage_group:<group name>` cross-links the dashboard on other +dashboards with the same tag. You can create dashboards for stage groups +in the [`dashboards/stage-groups`](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards/stage-groups) +directory. Directories can't be nested more than one level deep. + +To see a list of all the dashboards for your stage group: + +1. In Grafana, go to the [Dashboard browser](https://dashboards.gitlab.net/dashboards?tag=stage-groups). +1. To see all of the dashboards for a specific group, filter for `stage_group:<group name>`. + +Some generated dashboards are already available: + +1. [Stage group dashboard](stage_group_dashboard.md): a customizable + dashboard with tailored metrics per group. +1. [Error budget detail dashboard](error_budget_detail.md): a + dashboard allowing to explore the error budget spend over time and + over multiple SLIs. + +## Time range controls + + + +By default, all the times are in UTC time zone. +[We use UTC when communicating in Engineering.](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) + +All metrics recorded in the GitLab production system have +[one-year retention](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/-/blob/31526b03fef823e2f9b3cda7c75dcd28a12418a3/attributes/prometheus.rb#L40). + +You can also zoom in and filter the time range directly on a graph. For more information, see the +[Grafana Time Range Controls](https://grafana.com/docs/grafana/latest/dashboards/time-range-controls/) +documentation. + +## Filters and annotations + +On each dashboard, there are two filters and some annotation switches on the top of the page. + +Some special events are meaningful to development and operational activities. +[Grafana annotations](https://grafana.com/docs/grafana/latest/dashboards/annotations/) mark them +directly on the graphs. + + + +| Name | Type | Description | +| --------------- | ---------- | ----------- | +| `PROMETHEUS_DS` | filter | Filter the selective [Prometheus data sources](https://about.gitlab.com/handbook/engineering/monitoring/#prometheus). The default value is `Global`, which aggregates the data from all available data sources. Most of the time, you don't need to care about this filter. | +| `environment` | filter | Filter the environment the metrics are fetched from. The default setting is production (`gprd`). For other options, see [Production Environment mapping](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#environments). | +| `stage` | filter | Filter metrics by stage: `main` or `cny` for canary. Default is `main` | +| `deploy` | annotation | Mark a deployment event on the GitLab.com SaaS platform. | +| `canary-deploy` | annotation | Mark a [canary deployment](https://about.gitlab.com/handbook/engineering/#canary-testing) event on the GitLab.com SaaS platform. | +| `feature-flags` | annotation | Mark the time point when a feature flag is updated. | + +Example of a feature flag annotation displayed on a dashboard panel: + + diff --git a/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md new file mode 100644 index 00000000000..c1831cfce69 --- /dev/null +++ b/doc/development/stage_group_observability/dashboards/stage_group_dashboard.md @@ -0,0 +1,200 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Stage group dashboard + +The stage group dashboard is generated dashboard that contains metrics +for common components used by most stage groups. The dashboard is +fully customizable and owned by the stage groups. + +This page explains what is on these dashboards, how to use their +contents, and how they can be customized. + +## Dashboard contents + +### Error budget panels + + + +The top panels display the [error budget](../index.md#error-budget). +These panels always show the 28 days before the end time selected in the +[time range controls](index.md#time-range-controls). This data doesn't +follow the selected range. It does respect the filters for environment +and stage. + +### Metrics panels + + + +Although most of the metrics displayed in the panels are self-explanatory in their title and nearby +description, note the following: + +- The events are counted, measured, accumulated, collected, and stored as + [time series](https://prometheus.io/docs/concepts/data_model/). The data is calculated using + statistical methods to produce metrics. It means that metrics are approximately correct and + meaningful over a time period. They help you get an overview of the stage of a system over time. + They are not meant to give you precise numbers of a discrete event. + + If you need a higher level of accuracy, use another monitoring tool, such as + [logs](https://about.gitlab.com/handbook/engineering/monitoring/#logs). + Read the following examples for more explanations. +- All the rate metrics' units are `requests per second`. The default aggregate time frame is 1 minute. + + For example, a panel shows the requests per second number at `2020-12-25 00:42:00` to be `34.13`. + It means at the minute 42 (from `2020-12-25 00:42:00` to `2020-12-25 00:42:59` ), there are + approximately `34.13 * 60 = ~ 2047` requests processed by the web servers. +- You might encounter some gotchas related to decimal fraction and rounding up frequently, especially + in low-traffic cases. For example, the error rate of `RepositoryUpdateMirrorWorker` at + `2020-12-25 02:04:00` is `0.07`, equivalent to `4.2` jobs per minute. The raw result is + `0.06666666667`, equivalent to 4 jobs per minute. +- All the rate metrics are more accurate when the data is big enough. The default floating-point + precision is 2. In some extremely low panels, you can see `0.00`, even though there is still some + real traffic. + +To inspect the raw data of the panel for further calculation, select **Inspect** from the dropdown +list of a panel. Queries, raw data, and panel JSON structure are available. +Read more at [Grafana panel inspection](https://grafana.com/docs/grafana/latest/panels/inspect-panel/). + +All the dashboards are powered by [Grafana](https://grafana.com/), a frontend for displaying metrics. +Grafana consumes the data returned from queries to backend Prometheus data source, then presents it +with visualizations. The stage group dashboards are built to serve the most common use cases with a +limited set of filters and pre-built queries. Grafana provides a way to explore and visualize the +metrics data with [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/). This requires +some knowledge of the [Prometheus PromQL query language](https://prometheus.io/docs/prometheus/latest/querying/basics/). + +## Example: Debugging with dashboards + +Example debugging workflow: + +1. A team member in the Code Review group has merged an MR which got deployed to production. +1. To verify the deployment, you can check the + [Code Review group's dashboard](https://dashboards.gitlab.net/d/stage-groups-code_review/stage-groups-group-dashboard-create-code-review?orgId=1). +1. Sidekiq Error Rate panel shows an elevated error rate, specifically `UpdateMergeRequestsWorker`. + +  + +1. If you select **Kibana: Kibana Sidekiq failed request logs** in the **Extra links** section, you can filter for `UpdateMergeRequestsWorker` and read through the logs. + +  + +1. With [Sentry](https://sentry.gitlab.net/gitlab/gitlabcom/) you can find the exception where you + can filter by transaction type and `correlation_id` from Kibana's result item. + +  + +1. A precise exception, including a stack trace, job arguments, and other information should now appear. + +Happy debugging! + +## Customizing the dashboard + +All Grafana dashboards at GitLab are generated from the [Jsonnet files](https://github.com/grafana/grafonnet-lib) +stored in [the runbooks project](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards). +Particularly, the stage group dashboards definitions are stored in +[`/dashboards/stage-groups`](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards/stage-groups). + +By convention, each group has a corresponding Jsonnet file. The dashboards are synced with GitLab +[stage group data](https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/master/data/stages.yml) every +month. + +Expansion and customization are one of the key principles used when we designed this system. +To customize your group's dashboard, edit the corresponding file and follow the +[Runbook workflow](https://gitlab.com/gitlab-com/runbooks/-/tree/master/dashboards#dashboard-source). +The dashboard is updated after the MR is merged. + +Looking at an autogenerated file, for example, +[`product_planning.dashboard.jsonnet`](https://gitlab.com/gitlab-com/runbooks/-/blob/master/dashboards/stage-groups/product_planning.dashboard.jsonnet): + +```jsonnet +// This file is autogenerated using scripts/update_stage_groups_dashboards.rb +// Please feel free to customize this file. +local stageGroupDashboards = import './stage-group-dashboards.libsonnet'; + +stageGroupDashboards.dashboard('product_planning') +.stageGroupDashboardTrailer() +``` + +We provide basic customization to filter out the components essential to your group's activities. +By default, only the `web`, `api`, and `sidekiq` components are available in the dashboard, while +`git` is hidden. See [how to enable available components and optional graphs](#optional-graphs). + +You can also append further information or custom metrics to a dashboard. The following example +adds some links and a total request rate to the top of the page: + +```jsonnet +local stageGroupDashboards = import './stage-group-dashboards.libsonnet'; +local grafana = import 'github.com/grafana/grafonnet-lib/grafonnet/grafana.libsonnet'; +local basic = import 'grafana/basic.libsonnet'; + +stageGroupDashboards.dashboard('source_code') +.addPanel( + grafana.text.new( + title='Group information', + mode='markdown', + content=||| + Useful link for the Source Code Management group dashboard: + - [Issue list](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name%5B%5D=repository) + - [Epic list](https://gitlab.com/groups/gitlab-org/-/epics?label_name[]=repository) + |||, + ), + gridPos={ x: 0, y: 0, w: 24, h: 4 } +) +.addPanel( + basic.timeseries( + title='Total Request Rate', + yAxisLabel='Requests per Second', + decimals=2, + query=||| + sum ( + rate(gitlab_transaction_duration_seconds_count{ + env='$environment', + environment='$environment', + feature_category=~'source_code_management', + }[$__interval]) + ) + ||| + ), + gridPos={ x: 0, y: 0, w: 24, h: 7 } +) +.stageGroupDashboardTrailer() +``` + + + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +If you want to see the workflow in action, we've recorded a pairing session on customizing a dashboard, +available on [GitLab Unfiltered](https://youtu.be/shEd_eiUjdI). + +For deeper customization and more complicated metrics, visit the +[Grafonnet lib](https://github.com/grafana/grafonnet-lib) project and the +[GitLab Prometheus Metrics](../../../administration/monitoring/prometheus/gitlab_metrics.md#gitlab-prometheus-metrics) +documentation. + +### Optional graphs + +Some graphs aren't relevant for all groups, so they aren't added to +the dashboard by default. They can be added by customizing the +dashboard. + +By default, only the `web`, `api`, and `sidekiq` metrics are +shown. If you wish to see the metrics from the `git` fleet (or any +other component that might be added in the future), you can configure it as follows: + +```jsonnet +stageGroupDashboards +.dashboard('source_code', components=stageGroupDashboards.supportedComponents) +.stageGroupDashboardTrailer() +``` + +If your group is interested in Sidekiq job durations and their +thresholds, you can add these graphs by calling the `.addSidekiqJobDurationByUrgency` function: + +```jsonnet +stageGroupDashboards +.dashboard('access') +.addSidekiqJobDurationByUrgency() +.stageGroupDashboardTrailer() +``` diff --git a/doc/development/img/stage_group_dashboards_error_attribution.png b/doc/development/stage_group_observability/img/stage_group_dashboards_error_attribution.png Binary files differindex f6ea7c004ac..f6ea7c004ac 100644 --- a/doc/development/img/stage_group_dashboards_error_attribution.png +++ b/doc/development/stage_group_observability/img/stage_group_dashboards_error_attribution.png diff --git a/doc/development/img/stage_group_dashboards_service_sli_detail.png b/doc/development/stage_group_observability/img/stage_group_dashboards_service_sli_detail.png Binary files differindex 5dc32063709..5dc32063709 100644 --- a/doc/development/img/stage_group_dashboards_service_sli_detail.png +++ b/doc/development/stage_group_observability/img/stage_group_dashboards_service_sli_detail.png diff --git a/doc/development/stage_group_observability/index.md b/doc/development/stage_group_observability/index.md new file mode 100644 index 00000000000..868e55735e8 --- /dev/null +++ b/doc/development/stage_group_observability/index.md @@ -0,0 +1,138 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Observability for stage groups + +Observability is about bringing visibility into a system to see and +understand the state of each component, with context, to support +performance tuning and debugging. To run a SaaS platform at scale, a +rich and detailed observability platform is needed. + +To make information available to [stage groups](https://about.gitlab.com/handbook/product/categories/#hierarchy), +we are aggregating metrics by feature category and then show +this information on [dashboards](dashboards/index.md) tailored to the groups. Only metrics +for the features built by the group are visible on their +dashboards. + +With a filtered view, groups can discover bugs and performance regressions that could otherwise +be missed when viewing aggregated data. + +For more specific information on dashboards, see: + +- [Dashboards](dashboards/index.md): a general overview of where to find dashboards + and how to use them. +- [Stage group dashboard](dashboards/stage_group_dashboard.md): how to use and customize the stage group dashboard. +- [Error budget detail](dashboards/error_budget_detail.md): how to explore error budget over time. + +## Error budget + +The error budget is calculated from the same [Service Level Indicators](https://en.wikipedia.org/wiki/Service_level_indicator) (SLIs) +that we use to monitor GitLab.com. The 28-day availability number for a +stage group is comparable to the +[monthly availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability) +we calculate for GitLab.com, except it's scoped to the features of a group. + +To learn more about how we use error budgets, see the +[Engineering Error Budgets](https://about.gitlab.com/handbook/engineering/error-budgets/) handbook page. + +By default, the first row of panels on both dashboards shows the +[error budget for the stage group](https://about.gitlab.com/handbook/engineering/error-budgets/#budget-spend-by-stage-group). +This row shows how features owned by the group contribute to our +[overall availability](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlabcom-availability). + +The official budget is aggregated over the 28 days. You can see it on the +[stage group dashboard](dashboards/stage_group_dashboard.md). +The [error budget detail dashboard](dashboards/error_budget_detail.md) +allows customizing the range. + +We show the information in two formats: + +- Availability: this number can be compared to GitLab.com overall + availability target of 99.95% uptime. +- Budget Spent: time over the past 28 days that features owned by the group have not been performing + adequately. + +The budget is calculated based on indicators per component. Each +component can have two indicators: + +- [Apdex](https://en.wikipedia.org/wiki/Apdex): the rate of operations that performed adequately. + + The threshold for "performing adequately" is stored in our + [metrics catalog](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog) + and depends on the service in question. For the Puma (Rails) component of the + [API](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/api.jsonnet#L127), + [Git](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/git.jsonnet#L216), + and + [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) + services, that threshold is **5 seconds** when not opted in to the + [`rails_requests` SLI](../application_slis/rails_request_apdex.md). + + We've made this target configurable in [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). + To learn how to customize the request Apdex, see + [Rails request Apdex SLI](../application_slis/rails_request_apdex.md). + This new Apdex measurement is not part of the error budget until you + [opt in](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1451). + + For Sidekiq job execution, the threshold depends on the + [job urgency](../sidekiq/worker_attributes.md#job-urgency). It is + [currently](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L25-38) + **10 seconds** for high-urgency jobs and **5 minutes** for other jobs. + + Some stage groups might have more services. The thresholds for them are also in the metrics catalog. + +- Error rate: The rate of operations that had errors. + +The calculation of the ratio happens as follows: + +```math +\frac {operations\_meeting\_apdex + (total\_operations - operations\_with\_errors)} {total\_apdex\_measurements + total\_operations} +``` + +## Check where budget is being spent + +Both the [stage group dashboard](dashboards/stage_group_dashboard.md) +and the [error budget detail dashboard](dashboards/error_budget_detail.md) +show panels to see where the error budget was spent. The stage group +dashboard always shows a fixed 28 days. The error budget detail +dashboard allows drilling down to the SLIs over time. + +The row below the error budget row is collapsed by default. Expanding +it shows which component and violation type had the most offending +operations in the past 28 days. + + + +The first panel on the left shows a table with the number of errors per +component. Digging into the first row in that table has +the biggest impact on the budget spent. + +Commonly, the components that spend most of the budget are Sidekiq or Puma. The panel in +the center explains what different violation types mean and how to dig +deeper in the logs. + +The panel on the right provides links to Kibana that should reveal +which endpoints or Sidekiq jobs are causing the errors. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +To learn how to use these panels and logs for +determining which Rails endpoints are slow, +see the [Error Budget Attribution for Purchase group](https://youtu.be/M9u6unON7bU) video. + +Other components visible in the table come from +[service-level indicators](https://sre.google/sre-book/service-level-objectives/) (SLIs) defined +in the [metrics catalog](https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/README.md). + +For those types of failures, you can follow the link to the service +dashboard linked from the `type` column. The service dashboard +contains a row specifically for the SLI that is causing the budget +spent, with links to logs and a description of what the +component means. + +For example, see the `server` component of the `web-pages` service: + + + +To add more SLIs tailored to specific features, you can use an [Application SLI](../application_slis/index.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index fe0c4c13ba2..7ae49d33e91 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -21,7 +21,7 @@ a level that is difficult to manage. Test heuristics can help solve this problem. They concisely address many of the common ways bugs manifest themselves in our code. When designing our tests, take time to review known test heuristics to inform our test design. We can find some helpful heuristics documented in the Handbook in the -[Test Engineering](https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-heuristics) section. +[Test Engineering](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/test-engineering/#test-heuristics) section. ## RSpec @@ -404,7 +404,7 @@ click_link _('UI testing docs') fill_in _('Search projects'), with: 'gitlab' # fill in text input with text -select _('Last updated'), from: 'Sort by' # select an option from a select input +select _('Updated date'), from: 'Sort by' # select an option from a select input check _('Checkbox label') uncheck _('Checkbox label') @@ -465,8 +465,8 @@ expect(page).to have_checked_field _('Checkbox label') expect(page).to have_unchecked_field _('Radio input label') expect(page).to have_select _('Sort by') -expect(page).to have_select _('Sort by'), selected: 'Last updated' # assert the option is selected -expect(page).to have_select _('Sort by'), options: ['Last updated', 'Created date', 'Due date'] # assert an exact list of options +expect(page).to have_select _('Sort by'), selected: 'Updated date' # assert the option is selected +expect(page).to have_select _('Sort by'), options: ['Updated date', 'Created date', 'Due date'] # assert an exact list of options expect(page).to have_select _('Sort by'), with_options: ['Created date', 'Due date'] # assert a partial list of options expect(page).to have_text _('Some paragraph text.') diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index e0f6cbe632d..bd9896934c7 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -279,6 +279,9 @@ When you add a new test that requires administrator access, apply the RSpec meta When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag. +NOTE: +If the _only_ action in the test that requires administrator access is to toggle a feature flag, please use the `feature_flag` tag instead. More details can be found in [testing with feature flags](feature_flags.md). + ## Prefer `Commit` resource over `ProjectPush` In line with [using the API](#prefer-api-over-ui), use a `Commit` resource whenever possible. diff --git a/doc/development/testing_guide/end_to_end/execution_context_selection.md b/doc/development/testing_guide/end_to_end/execution_context_selection.md index 0fdcf0c8c3b..0a4c5fcf451 100644 --- a/doc/development/testing_guide/end_to_end/execution_context_selection.md +++ b/doc/development/testing_guide/end_to_end/execution_context_selection.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Execution context selection -Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata. +Some tests are designed to be run against specific environments, or in specific [pipelines](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#scheduled-qa-test-pipelines) or jobs. We can specify the test execution context using the `only` and `except` metadata. ## Available switches @@ -118,7 +118,7 @@ To run a test tagged with `except` locally, you can either: Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }` -hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash. +hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#quarantining-tests) hash. For example, `quarantine: { only: { subdomain: :staging } }` only quarantines the test when run against `staging`. The quarantine feature can be explicitly disabled with the `DISABLE_QUARANTINE` environment variable. This can be useful when running tests locally. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index c3e3f117c2b..47ebef37a4d 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -14,19 +14,45 @@ automatically authenticates as an administrator as long as you provide an approp token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. -Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments -where administrator access is not available. +## `feature_flag` RSpec tag -WARNING: -You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), -or [feature group](../../feature_flags/index.md#feature-groups). This makes it possible to -test a feature in a shared environment without affecting other users. +Please be sure to include the `feature_flag` tag so that the test can be skipped on the appropriate environments. -For example, the code below would enable a feature flag named `:feature_flag_name` for the project +**Optional metadata:** + +`name` + +- Format: `feature_flag: { name: 'feature_flag_name' }` +- Used only for informational purposes at this time. It should be included to help quickly determine what +feature flag is under test. + +`scope` + +- Format: `feature_flag: { name: 'feature_flag_name', scope: :project }` +- When `scope` is set to `:global`, the test will be **skipped on all live .com environments**. This is to avoid issues with feature flag changes affecting other tests or users on that environment. +- When `scope` is set to any other value (such as `:project`, `:group` or `:user`), or if no `scope` is specified, the test will only be **skipped on canary and production**. +This is due to the fact that admin access is not available there. + +**WARNING:** You are strongly advised to first try and [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), +or [feature group](../../feature_flags/index.md#feature-groups). + +- If a global feature flag must be used, it is strongly recommended to apply `scope: :global` to the `feature_flag` metadata. This is, however, left up to the SET's discretion to determine the level of risk. + - For example, a test uses a global feature flag that only affects a small area of the application and is also needed to check for critical issues on live environments. + In such a scenario, it would be riskier to skip running the test. For cases like this, `scope` can be left out of the metadata so that it can still run in live environments + with admin access, such as staging. + +**Note on `requires_admin`:** This tag should still be applied if there are other actions within the test that require admin access that are unrelated to updating a +feature flag (ex: creating a user via the API). + +The code below would enable a feature flag named `:feature_flag_name` for the project created by the test: ```ruby -RSpec.describe "with feature flag enabled", :requires_admin do +RSpec.describe "with feature flag enabled", feature_flag: { + name: 'feature_flag_name', + scope: :project + } do + let(:project) { Resource::Project.fabricate_via_api! } before do @@ -162,7 +188,7 @@ for details. ## Confirming that end-to-end tests pass with a feature flag enabled -End-to-end tests should pass with a feature flag enabled before it is enabled on Staging or on GitLab.com. Tests that need to be updated should be identified as part of [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quad-planning/). The relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) is responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through quad-planning and a required test update is not made, test failures could block deployment. +End-to-end tests should pass with a feature flag enabled before it is enabled on Staging or on GitLab.com. Tests that need to be updated should be identified as part of [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/quad-planning/). The relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) is responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through quad-planning and a required test update is not made, test failures could block deployment. ### Automatic test execution when a feature flag definition changes diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index dc989acbdcc..1e7cba9d247 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -135,7 +135,7 @@ The [existing scenarios](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/doc that run in the downstream `gitlab-qa-mirror` pipeline include many tests, but there are times when you might want to run a test or a group of tests that are different than the groups in any of the existing scenarios. -For example, when we [dequarantine](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests) +For example, when we [dequarantine](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#dequarantining-tests) a flaky test we first want to make sure that it's no longer flaky. We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs. Both are manual jobs that you can configure using custom variables. @@ -281,6 +281,7 @@ Continued reading: - [Flows](flows.md) - [RSpec metadata/tags](rspec_metadata_tests.md) - [Execution context selection](execution_context_selection.md) +- [Troubleshooting](troubleshooting.md) ## Where can I ask for help? diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index f9b505a8271..45161404c73 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -11,41 +11,42 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec <!-- Please keep the tags in alphabetical order --> -| Tag | Description | -|-----|-------------| -| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | -| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | -| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | -| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | -| `:github` | The test requires a GitHub personal access token. | -| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | -| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | -| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | -| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. -| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | -| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | -| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | -| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | -| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | -| `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. | -| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | -| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | -| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | -| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | -| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). | -| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | -| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | -| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | -| `:requires_admin` | The test requires an administrator account. Tests with the tag are excluded when run against Canary and Production environments. | -| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | -| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | -| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | -| `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | -| `:skip_fips_env` | The test is excluded when run against an environment in FIPS mode. | -| `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | -| `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance.| -| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | -| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | -| `:transient` | The test tests transient bugs. It is excluded by default. | -| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keep track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding an optional numeric suffix like `issue_1`, `issue_2` etc. | +| Tag | Description | +|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `:elasticsearch` | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. | +| `:except` | The test is to be run in their typical execution contexts _except_ as specified. See [test execution context selection](execution_context_selection.md) for more information. | +| `:feature_flag` | The test uses a feature flag and therefore requires an administrator account to run. When `scope` is set to `:global`, the test will be skipped on all live .com environments. Otherwise, it will be skipped only on Canary and Production. See [testing with feature flags](../../../development/testing_guide/end_to_end/feature_flags.md) for more details. | +| `:geo` | The test requires two GitLab Geo instances - a primary and a secondary - to be spun up. | +| `:gitaly_cluster` | The test runs against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | +| `:github` | The test requires a GitHub personal access token. | +| `:group_saml` | The test requires a GitLab instance that has SAML SSO enabled at the group level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:instance_saml` | The test requires a GitLab instance that has SAML SSO enabled at the instance level. Interacts with an external SAML identity provider. Paired with the `:orchestrated` tag. | +| `:integrations` | This aims to test the available [integrations](../../../user/project/integrations/overview.md#integrations-listing). The test requires Docker to be installed in the run context. It will provision the containers and can be run against a local instance or using the `gitlab-qa` scenario `Test::Integration::Integrations` | +| `:service_ping_disabled` | The test interacts with the GitLab configuration service ping at the instance level to turn admin setting service ping checkbox on or off. This tag will have the test run only in the `service_ping_disabled` job and must be paired with the `:orchestrated` and `:requires_admin` tags. | +| `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) provisions the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | +| `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test also includes provisioning of at least one Kubernetes cluster to test against. _This tag is often be paired with `:orchestrated`._ | +| `:ldap_no_server` | The test requires a GitLab instance to be configured to use LDAP. To be used with the `:orchestrated` tag. It does not spin up an LDAP server at orchestration time. Instead, it creates the LDAP server at runtime. | +| `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | +| `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | +| `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. | +| `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | +| `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | +| `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | +| `:packages` | The test requires a GitLab instance that has the [Package Registry](../../../administration/packages/#gitlab-package-registry-administration) enabled. | +| `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#quarantining-tests), runs in a separate job that only includes quarantined tests, and is allowed to fail. The test is skipped in its regular job so that if it fails it doesn't hold up the pipeline. Note that you can also [quarantine a test only when it runs in a specific context](execution_context_selection.md#quarantine-a-test-for-a-specific-environment). | +| `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | +| `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | +| `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | +| `:requires_admin` | The test requires an administrator account. Tests with the tag are excluded when run against Canary and Production environments. | +| `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | +| `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | +| `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | +| `:skip_live_env` | The test is excluded when run against live deployed environments such as Staging, Canary, and Production. | +| `:skip_fips_env` | The test is excluded when run against an environment in FIPS mode. | +| `:skip_signup_disabled` | The test uses UI to sign up a new user and is skipped in any environment that does not allow new user registration via the UI. | +| `:smoke` | The test belongs to the test suite which verifies basic functionality of a GitLab instance. | +| `:smtp` | The test requires a GitLab instance to be configured to use an SMTP server. Tests SMTP notification email delivery from GitLab by using MailHog. | +| `:testcase` | The link to the test case issue in the [GitLab Project Test Cases](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases). | +| `:transient` | The test tests transient bugs. It is excluded by default. | +| `:issue`, `:issue_${num}` | Optional links to issues which might be related to the spec. Helps keep track of related issues and can also be used by tools that create test reports. Currently added automatically to `Allure` test report. Multiple tags can be used by adding an optional numeric suffix like `issue_1`, `issue_2` etc. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 49a9124253d..599e1104b72 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -299,7 +299,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you need a r #### Notes -- You can find the full image address from a pipeline by [following these instructions](https://about.gitlab.com/handbook/engineering/quality/guidelines/tips-and-tricks/#running-gitlab-qa-pipeline-against-a-specific-gitlab-release). You might be prompted to set the `GITLAB_QA_ACCESS_TOKEN` variable if you specify the full image address. +- You can find the full image address from a pipeline by [following these instructions](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/tips-and-tricks/#running-gitlab-qa-pipeline-against-a-specific-gitlab-release). You might be prompted to set the `GITLAB_QA_ACCESS_TOKEN` variable if you specify the full image address. - You can increase the wait time for replication by setting `GEO_MAX_FILE_REPLICATION_TIME` and `GEO_MAX_DB_REPLICATION_TIME`. The default is 120 seconds. - To save time during tests, create a Personal Access Token with API access on the Geo primary node, and pass that value in as `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN`. @@ -395,7 +395,7 @@ Tests that are tagged with `:mobile` can be run against specified mobile devices Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. -For tunnel installation instructions, read [Sauce Connect Proxy Installation](https://docs.saucelabs.com/secure-connections/sauce-connect/installation). To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. +For tunnel installation instructions, read [Sauce Connect Proxy Installation](https://docs.saucelabs.com/secure-connections/sauce-connect/installation/index.html). To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. NOTE: It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness. diff --git a/doc/development/testing_guide/end_to_end/troubleshooting.md b/doc/development/testing_guide/end_to_end/troubleshooting.md new file mode 100644 index 00000000000..951fb056a4c --- /dev/null +++ b/doc/development/testing_guide/end_to_end/troubleshooting.md @@ -0,0 +1,69 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Troubleshooting end-to-end tests + +## See what the browser is doing + +If end-to-end tests fail, it can be very helpful to see what is happening in your +browser when it fails. For example, if tests don't run at all, the test framework +might be trying to open a URL that isn't valid on your machine. This problem becomes +clearer if you see the page fail in the browser. + +To make the test framework show the browser as it runs the tests, +set `WEBDRIVER_HEADLESS=false`. For example: + +```shell +cd gitlab/qa +WEBDRIVER_HEADLESS=false bundle exec bin/qa Test::Instance::All http://localhost:3000 +``` + +## Enable logging + +Sometimes a test might fail and the failure stack trace doesn't provide enough +information to determine what went wrong. You can get more information by enabling +debug logs by setting `QA_DEBUG=true`, to see what the test framework is attempting. +For example: + +```shell +cd gitlab/qa +QA_DEBUG=true bundle exec bin/qa Test::Instance::All http://localhost:3000 +``` + +The test framework then outputs many logs showing the actions taken during +the tests: + +```plaintext +[date=2022-03-31 23:19:47 from=QA Tests] INFO -- Starting test: Create Merge request creation from fork can merge feature branch fork to mainline +[date=2022-03-31 23:19:49 from=QA Tests] DEBUG -- has_element? :login_page (wait: 0) returned: true +[date=2022-03-31 23:19:52 from=QA Tests] DEBUG -- filling :login_field with "root" +[date=2022-03-31 23:19:52 from=QA Tests] DEBUG -- filling :password_field with "*****" +[date=2022-03-31 23:19:52 from=QA Tests] DEBUG -- clicking :sign_in_button +``` + +## Tests don't run at all + +This section assumes you're running the tests locally (such as the GDK) and you're doing +so from the `gitlab/qa/` folder, not from `gitlab-qa`. For example, if you receive a +`Net::ReadTimeout` error, the browser might be unable to load the specified URL: + +```shell +cd gitlab/qa +bundle exec bin/qa Test::Instance::All http://localhost:3000 + +bundler: failed to load command: bin/qa (bin/qa) +Net::ReadTimeout: Net::ReadTimeout with #<TCPSocket:(closed)> +``` + +This error can happen if GitLab runs on an address that does not resolve from +`localhost`. For example, if you set GDK's `hostname` +[to a specific local IP address](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/run_qa_against_gdk.md#run-qa-tests-against-your-gdk-setup), +you must use that IP address instead of `localhost` in the command. +For example, if your IP is `192.168.0.12`: + +```shell +bundle exec bin/qa Test::Instance::All http://192.168.0.12:3000 +``` diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 27d5ae70ed7..f5483a4b79c 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -172,8 +172,6 @@ subgraph "CNG-mirror pipeline" them in its [registry](https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry). - We use the [`CNG-mirror`](https://gitlab.com/gitlab-org/build/CNG-mirror) project so that the `CNG`, (Cloud Native GitLab), project's registry is not overloaded with a lot of transient Docker images. - - Note that the official CNG images are built by the `cloud-native-image` - job, which runs only for tags, and triggers itself a [`CNG`](https://gitlab.com/gitlab-org/build/CNG) pipeline. 1. Once `review-build-cng` is done, the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job deploys the Review App using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/) to the [`review-apps`](https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps?project=gitlab-review-apps) @@ -224,14 +222,10 @@ If you need your Review App to stay up for a longer time, you can `review-deploy` job to update the "latest deployed at" time. The `review-cleanup` job that automatically runs in scheduled -pipelines (and is manual in merge request) stops stale Review Apps after 5 days, +pipelines stops stale Review Apps after 5 days, deletes their environment after 6 days, and cleans up any dangling Helm releases and Kubernetes resources after 7 days. -The `review-gcp-cleanup` job that automatically runs in scheduled pipelines -(and is manual in merge request) removes any dangling GCP network resources -that were not removed along with the Kubernetes resources. - ## Cluster configuration The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project. @@ -254,189 +248,7 @@ Leading indicators may be health check failures leading to restarts or majority The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. -### Database related errors in `review-deploy`, `review-qa-smoke`, or `review-qa-reliable` - -Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by -changes to migration files or schema, such as a migration being renamed or deleted. This typically manifests in migration errors such as: - -- migration job failing with a column that already exists -- migration job failing with a column that does not exist - -To recover from this, please attempt to [redeploy Review App from a clean slate](#redeploy-review-app-from-a-clean-slate) - -### Release failed with `ImagePullBackOff` - -**Potential cause:** - -If you see an `ImagePullBackoff` status, check for a missing Docker image. - -**Where to look for further debugging:** - -To check that the Docker images were created, run the following Docker command: - -```shell -`DOCKER_CLI_EXPERIMENTAL=enabled docker manifest repository:tag` -``` - -The output of this command indicates if the Docker image exists. For example: - -```shell -DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee:39467-allow-a-release-s-associated-milestones-to-be-edited-thro -``` - -If the Docker image does not exist: - -- Verify the `image.repository` and `image.tag` options in the `helm upgrade --install` command match the repository names used by CNG-mirror pipeline. -- Look further in the corresponding downstream CNG-mirror pipeline in `review-build-cng` job. - -### Node count is always increasing (never stabilizing or decreasing) - -**Potential cause:** - -That could be a sign that the `review-cleanup` job is -failing to cleanup stale Review Apps and Kubernetes resources. - -**Where to look for further debugging:** - -Look at the latest `review-cleanup` job log, and identify look for any -unexpected failure. - -### p99 CPU utilization is at 100% for most of the nodes and/or many components - -**Potential cause:** - -This could be a sign that Helm is failing to deploy Review Apps. When Helm has a -lot of `FAILED` releases, it seems that the CPU utilization is increasing, probably -due to Helm or Kubernetes trying to recreate the components. - -**Where to look for further debugging:** - -Look at a recent `review-deploy` job log. - -**Useful commands:** - -```shell -# Identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler -kubectl top nodes | sort --key 3 --numeric - -# Identify pods under heavy CPU load -kubectl top pods | sort --key 2 --numeric -``` - -### The `logging/user/events/FailedMount` chart is going up - -**Potential cause:** - -This could be a sign that there are too many stale secrets and/or configuration maps. - -**Where to look for further debugging:** - -Look at [the list of Configurations](https://console.cloud.google.com/kubernetes/config?project=gitlab-review-apps) -or `kubectl get secret,cm --sort-by='{.metadata.creationTimestamp}' | grep 'review-'`. - -Any secrets or configuration maps older than 5 days are suspect and should be deleted. - -**Useful commands:** - -```shell -# List secrets and config maps ordered by created date -kubectl get secret,cm --sort-by='{.metadata.creationTimestamp}' | grep 'review-' - -# Delete all secrets that are 5 to 9 days old -kubectl get secret --sort-by='{.metadata.creationTimestamp}' | grep '^review-' | grep '[5-9]d$' | cut -d' ' -f1 | xargs kubectl delete secret - -# Delete all secrets that are 10 to 99 days old -kubectl get secret --sort-by='{.metadata.creationTimestamp}' | grep '^review-' | grep '[1-9][0-9]d$' | cut -d' ' -f1 | xargs kubectl delete secret - -# Delete all config maps that are 5 to 9 days old -kubectl get cm --sort-by='{.metadata.creationTimestamp}' | grep 'review-' | grep -v 'dns-gitlab-review-app' | grep '[5-9]d$' | cut -d' ' -f1 | xargs kubectl delete cm - -# Delete all config maps that are 10 to 99 days old -kubectl get cm --sort-by='{.metadata.creationTimestamp}' | grep 'review-' | grep -v 'dns-gitlab-review-app' | grep '[1-9][0-9]d$' | cut -d' ' -f1 | xargs kubectl delete cm -``` - -### Using K9s - -[K9s](https://github.com/derailed/k9s) is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/review_apps/base-config.yaml). Kubernetes schedules pods to nodes based on resource requests and allow for CPU usage up to the limits. - -- In K9s you can sort or add filters by typing the `/` character - - `-lrelease=<review-app-slug>` - filters down to all pods for a release. This aids in determining what is having issues in a single deployment - - `-lapp=<app>` - filters down to all pods for a specific app. This aids in determining resource usage by app. -- You can scroll to a Kubernetes resource and hit `d`(describe), `s`(shell), `l`(logs) for a deeper inspection - - - -### Troubleshoot a pending `dns-gitlab-review-app-external-dns` Deployment - -#### Finding the problem - -[In the past](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62834), it happened -that the `dns-gitlab-review-app-external-dns` Deployment was in a pending state, -effectively preventing all the Review Apps from getting a DNS record assigned, -making them unreachable via domain name. - -This in turn prevented other components of the Review App to properly start -(for example, `gitlab-runner`). - -After some digging, we found that new mounts fail when performed -with transient scopes (for example, pods) of `systemd-mount`: - -```plaintext -MountVolume.SetUp failed for volume "dns-gitlab-review-app-external-dns-token-sj5jm" : mount failed: exit status 1 -Mounting command: systemd-run -Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm --scope -- mount -t tmpfs tmpfs /var/lib/kubelet/pods/06add1c3-87b4-11e9-80a9-42010a800107/volumes/kubernetes.io~secret/dns-gitlab-review-app-external-dns-token-sj5jm -Output: Failed to start transient scope unit: Connection timed out -``` - -This probably happened because the GitLab chart creates 67 resources, leading to -a lot of mount points being created on the underlying GCP node. - -The [underlying issue seems to be a `systemd` bug](https://github.com/kubernetes/kubernetes/issues/57345#issuecomment-359068048) -that was fixed in `systemd` `v237`. Unfortunately, our GCP nodes are currently -using `v232`. - -For the record, the debugging steps to find out this issue were: - -1. Switch kubectl context to `review-apps-ce` (we recommend using [`kubectx`](https://github.com/ahmetb/kubectx/)) -1. `kubectl get pods | grep dns` -1. `kubectl describe pod <pod name>` & confirm exact error message -1. Web search for exact error message, following rabbit hole to [a relevant Kubernetes bug report](https://github.com/kubernetes/kubernetes/issues/57345) -1. Access the node over SSH via the GCP console (**Computer Engine > VM - instances** then click the "SSH" button for the node where the `dns-gitlab-review-app-external-dns` pod runs) -1. In the node: `systemctl --version` => `systemd 232` -1. Gather some more information: - - `mount | grep kube | wc -l` (returns a count, for example, 290) - - `systemctl list-units --all | grep -i var-lib-kube | wc -l` (returns a count, for example, 142) -1. Check how many pods are in a bad state: - - Get all pods running a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME` - - Get all the `Running` pods on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep Running` - - Get all the pods in a bad state on a given node: `kubectl get pods --field-selector=spec.nodeName=NODE_NAME | grep -v 'Running' | grep -v 'Completed'` - -#### Solving the problem - -To resolve the problem, we needed to (forcibly) drain some nodes: - -1. Try a normal drain on the node where the `dns-gitlab-review-app-external-dns` - pod runs so that Kubernetes automatically move it to another node: `kubectl drain NODE_NAME` -1. If that doesn't work, you can also perform a forcible "drain" the node by removing all pods: `kubectl delete pods --field-selector=spec.nodeName=NODE_NAME` -1. In the node: - - Perform `systemctl daemon-reload` to remove the dead/inactive units - - If that doesn't solve the problem, perform a hard reboot: `sudo systemctl reboot` -1. Uncordon any cordoned nodes: `kubectl uncordon NODE_NAME` - -In parallel, since most Review Apps were in a broken state, we deleted them to -clean up the list of non-`Running` pods. -Following is a command to delete Review Apps based on their last deployment date -(current date was June 6th at the time) with - -```shell -helm ls -d | grep "Jun 4" | cut -f1 | xargs helm delete --purge -``` - -#### Mitigation steps taken to avoid this problem in the future - -We've created a new node pool with smaller machines to reduce the risk -that a machine reaches the "too many mount points" problem in the future. +See the [review apps page of the Engineering Productivity Runbook](https://gitlab.com/gitlab-org/quality/engineering-productivity/team/-/blob/main/runbook/review-apps.md) for troubleshooting review app releases. ## Frequently Asked Questions diff --git a/doc/development/workhorse/channel.md b/doc/development/workhorse/channel.md new file mode 100644 index 00000000000..33d7cc63f00 --- /dev/null +++ b/doc/development/workhorse/channel.md @@ -0,0 +1,201 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Websocket channel support for Workhorse + +In some cases, GitLab can provide the following through a WebSocket: + +- In-browser terminal access to an environment: a running server or container, + onto which a project has been deployed. +- Access to services running in CI. + +Workhorse manages the WebSocket upgrade and long-lived connection to the websocket +connection, which frees up GitLab to process other requests. This document outlines +the architecture of these connections. + +## Introduction to WebSockets + +Websockets are an "upgraded" `HTTP/1.1` request. They permit bidirectional +communication between a client and a server. **Websockets are not HTTP**. +Clients can send messages (known as frames) to the server at any time, and +vice versa. Client messages are not necessarily requests, and server messages are +not necessarily responses. WebSocket URLs have schemes like `ws://` (unencrypted) or +`wss://` (TLS-secured). + +When requesting an upgrade to WebSocket, the browser sends a `HTTP/1.1` +request like this: + +```plaintext +GET /path.ws HTTP/1.1 +Connection: upgrade +Upgrade: websocket +Sec-WebSocket-Protocol: terminal.gitlab.com +# More headers, including security measures +``` + +At this point, the connection is still HTTP, so this is a request. +The server can send a normal HTTP response, such as `404 Not Found` or +`500 Internal Server Error`. + +If the server decides to permit the upgrade, it sends a HTTP +`101 Switching Protocols` response. From this point, the connection is no longer +HTTP. It is now a WebSocket and frames, not HTTP requests, flow over it. The connection +persists until the client or server closes the connection. + +In addition to the sub-protocol, individual websocket frames may +also specify a message type, such as: + +- `BinaryMessage` +- `TextMessage` +- `Ping` +- `Pong` +- `Close` + +Only binary frames can contain arbitrary data. The frames are expected to be valid +UTF-8 strings, in addition to any sub-protocol expectations. + +## Browser to Workhorse + +Using the terminal as an example: + +1. GitLab serves a JavaScript terminal emulator to the browser on a URL like + `https://gitlab.com/group/project/-/environments/1/terminal`. +1. This URL opens a websocket connection to + `wss://gitlab.com/group/project/-/environments/1/terminal.ws`. + This endpoint exists only in Workhorse, and doesn't exist in GitLab. +1. When receiving the connection, Workhorse first performs a `preauthentication` + request to GitLab to confirm the client is authorized to access the requested terminal: + - If the client has the appropriate permissions and the terminal exists, GitLab + responds with a successful response that includes details of the terminal + the client should be connected to. + - Otherwise, Workhorse returns an appropriate HTTP error response. +1. If GitLab returns valid terminal details to Workhorse, it: + 1. Connects to the specified terminal. + 1. Upgrades the browser to a WebSocket. + 1. Proxies between the two connections for as long as the browser's credentials are valid. + 1. Send regular `PingMessage` control frames to the browser, to prevent intervening + proxies from terminating the connection while the browser is present. + +The browser must request an upgrade with a specific sub-protocol: + +- [`terminal.gitlab.com`](#terminalgitlabcom) +- [`base64.terminal.gitlab.com`](#base64terminalgitlabcom) + +### `terminal.gitlab.com` + +This sub-protocol considers `TextMessage` frames to be invalid. Control frames, +such as `PingMessage` or `CloseMessage`, have their usual meanings. + +- `BinaryMessage` frames sent from the browser to the server are + arbitrary text input. +- `BinaryMessage` frames sent from the server to the browser are + arbitrary text output. + +These frames are expected to contain ANSI text control codes +and may be in any encoding. + +### `base64.terminal.gitlab.com` + +This sub-protocol considers `BinaryMessage` frames to be invalid. +Control frames, such as `PingMessage` or `CloseMessage`, have +their usual meanings. + +- `TextMessage` frames sent from the browser to the server are + base64-encoded arbitrary text input. The server must + base64-decode them before inputting them. +- `TextMessage` frames sent from the server to the browser are + base64-encoded arbitrary text output. The browser must + base64-decode them before outputting them. + +In their base64-encoded form, these frames are expected to +contain ANSI terminal control codes, and may be in any encoding. + +## Workhorse to GitLab + +Using the terminal as an example, before upgrading the browser, +Workhorse sends a normal HTTP request to GitLab on a URL like +`https://gitlab.com/group/project/environments/1/terminal.ws/authorize`. +This returns a JSON response containing details of where the +terminal can be found, and how to connect it. In particular, +the following details are returned in case of success: + +- WebSocket URL to connect** to, such as `wss://example.com/terminals/1.ws?tty=1`. +- WebSocket sub-protocols to support, such as `["channel.k8s.io"]`. +- Headers to send, such as `Authorization: Token xxyyz`. +- Optional. Certificate authority to verify `wss` connections with. + +Workhorse periodically rechecks this endpoint. If it receives an error response, +or the details of the terminal change, it terminates the websocket session. + +## Workhorse to the WebSocket server + +In GitLab, environments or CI jobs may have a deployment service (like +`KubernetesService`) associated with them. This service knows +where the terminals or the service for an environment may be found, and GitLab +returns these details to Workhorse. + +These URLs are also WebSocket URLs. GitLab tells Workhorse which sub-protocols to +speak over the connection, along with any authentication details required by the +remote end. + +Before upgrading the browser's connection to a websocket, Workhorse: + +1. Opens a HTTP client connection, according to the details given to it by Workhorse. +1. Attempts to upgrade that connection to a websocket. + - If it fails, an error response is sent to the browser. + - If it succeeds, the browser is also upgraded. + +Workhorse now has two websocket connections, albeit with differing sub-protocols, +and then: + +- Decodes incoming frames from the browser, re-encodes them to the channel's + sub-protocol, and sends them to the channel. +- Decodes incoming frames from the channel, re-encodes them to the browser's + sub-protocol, and sends them to the browser. + +When either connection closes or enters an error state, Workhorse detects the error +and closes the other connection, terminating the channel session. If the browser +is the connection that has disconnected, Workhorse sends an ANSI `End of Transmission` +control code (the `0x04` byte) to the channel, encoded according to the appropriate +sub-protocol. To avoid being disconnected, Workhorse replies to any websocket ping +frame sent by the channel. + +Workhorse only supports the following sub-protocols: + +- [`channel.k8s.io`](#channelk8sio) +- [`base64.channel.k8s.io`](#base64channelk8sio) + +Supporting new deployment services requires new sub-protocols to be supported. + +### `channel.k8s.io` + +Used by Kubernetes, this sub-protocol defines a simple multiplexed channel. + +Control frames have their usual meanings. `TextMessage` frames are +invalid. `BinaryMessage` frames represent I/O to a specific file +descriptor. + +The first byte of each `BinaryMessage` frame represents the file +descriptor (`fd`) number, as a `uint8`. For example: + +- `0x00` corresponds to `fd 0`, `STDIN`. +- `0x01` corresponds to `fd 1`, `STDOUT`. + +The remaining bytes represent arbitrary data. For frames received +from the server, they are bytes that have been received from that +`fd`. For frames sent to the server, they are bytes that should be +written to that `fd`. + +### `base64.channel.k8s.io` + +Also used by Kubernetes, this sub-protocol defines a similar multiplexed +channel to `channel.k8s.io`. The main differences are: + +- `TextMessage` frames are valid, rather than `BinaryMessage` frames. +- The first byte of each `TextMessage` frame represents the file + descriptor as a numeric UTF-8 character, so the character `U+0030`, + or "0", is `fd 0`, `STDIN`. +- The remaining bytes represent base64-encoded arbitrary data. diff --git a/doc/development/workhorse/configuration.md b/doc/development/workhorse/configuration.md new file mode 100644 index 00000000000..7f9331e6f1e --- /dev/null +++ b/doc/development/workhorse/configuration.md @@ -0,0 +1,218 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Workhorse configuration + +For historical reasons, Workhorse uses: + +- Command line flags. +- A configuration file. +- Environment variables. + +Add any new Workhorse configuration options into the configuration file. + +## CLI options + +```plaintext + gitlab-workhorse [OPTIONS] + +Options: + -apiCiLongPollingDuration duration + Long polling duration for job requesting for runners (default 50ns) + -apiLimit uint + Number of API requests allowed at single time + -apiQueueDuration duration + Maximum queueing duration of requests (default 30s) + -apiQueueLimit uint + Number of API requests allowed to be queued + -authBackend string + Authentication/authorization backend (default "http://localhost:8080") + -authSocket string + Optional: Unix domain socket to dial authBackend at + -cableBackend string + Optional: ActionCable backend (default authBackend) + -cableSocket string + Optional: Unix domain socket to dial cableBackend at (default authSocket) + -config string + TOML file to load config from + -developmentMode + Allow the assets to be served from Rails app + -documentRoot string + Path to static files content (default "public") + -listenAddr string + Listen address for HTTP server (default "localhost:8181") + -listenNetwork string + Listen 'network' (tcp, tcp4, tcp6, unix) (default "tcp") + -listenUmask int + Umask for Unix socket + -logFile string + Log file location + -logFormat string + Log format to use defaults to text (text, json, structured, none) (default "text") + -pprofListenAddr string + pprof listening address, e.g. 'localhost:6060' + -prometheusListenAddr string + Prometheus listening address, e.g. 'localhost:9229' + -proxyHeadersTimeout duration + How long to wait for response headers when proxying the request (default 5m0s) + -secretPath string + File with secret key to authenticate with authBackend (default "./.gitlab_workhorse_secret") + -version + Print version and exit +``` + +The 'auth backend' refers to the GitLab Rails application. The name is +a holdover from when GitLab Workhorse only handled `git push` and `git pull` over +HTTP. + +GitLab Workhorse can listen on either a TCP or a Unix domain socket. It +can also open a second listening TCP listening socket with the Go +[`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/). + +GitLab Workhorse can listen on Redis build and runner registration events if you +pass a valid TOML configuration file through the `-config` flag. +A regular setup it only requires the following (replacing the string +with the actual socket) + +## Redis + +GitLab Workhorse integrates with Redis to do long polling for CI build +requests. To configure it: + +- Configure Redis settings in the TOML configuration file. +- Control polling behavior for CI build requests with the `-apiCiLongPollingDuration` + command-line flag. + +You can enable Redis in the configuration file while leaving CI polling +disabled. This configuration results in an idle Redis Pub/Sub connection. The +opposite is not possible: CI long polling requires a correct Redis configuration. + +For example, the `[redis]` section in the configuration file could contain: + +```plaintext +[redis] +URL = "unix:///var/run/gitlab/redis.sock" +Password = "my_awesome_password" +Sentinel = [ "tcp://sentinel1:23456", "tcp://sentinel2:23456" ] +SentinelMaster = "mymaster" +``` + +- `URL` - A string in the format `unix://path/to/redis.sock` or `tcp://host:port`. +- `Password` - Required only if your Redis instance is password-protected. +- `Sentinel` - Required if you use Sentinel. + +If both `Sentinel` and `URL` are given, only `Sentinel` is used. + +Optional fields: + +```plaintext +[redis] +DB = 0 +MaxIdle = 1 +MaxActive = 1 +``` + +- `DB` - The database to connect to. Defaults to `0`. +- `MaxIdle` - How many idle connections can be in the Redis pool at once. Defaults to `1`. +- `MaxActive` - How many connections the pool can keep. Defaults to `1`. + +## Relative URL support + +If you mount GitLab at a relative URL, like `example.com/gitlab`), use this +relative URL in the `authBackend` setting: + +```plaintext +gitlab-workhorse -authBackend http://localhost:8080/gitlab +``` + +## Interaction of authBackend and authSocket + +The interaction between `authBackend` and `authSocket` can be confusing. +If `authSocket` is set, it overrides the host portion of `authBackend`, but not +the relative path. + +In table form: + +| authBackend | authSocket | Workhorse connects to | Rails relative URL | +|--------------------------------|-------------------|-----------------------|--------------------| +| unset | unset | `localhost:8080` | `/` | +| `http://localhost:3000` | unset | `localhost:3000` | `/` | +| `http://localhost:3000/gitlab` | unset | `localhost:3000` | `/gitlab` | +| unset | `/path/to/socket` | `/path/to/socket` | `/` | +| `http://localhost:3000` | `/path/to/socket` | `/path/to/socket` | `/` | +| `http://localhost:3000/gitlab` | `/path/to/socket` | `/path/to/socket` | `/gitlab` | + +The same applies to `cableBackend` and `cableSocket`. + +## Error tracking + +GitLab-Workhorse supports remote error tracking with [Sentry](https://sentry.io). +To enable this feature, set the `GITLAB_WORKHORSE_SENTRY_DSN` environment variable. +You can also set the `GITLAB_WORKHORSE_SENTRY_ENVIRONMENT` environment variable to +use the Sentry environment feature to separate staging, production and +development. + +Omnibus GitLab (`/etc/gitlab/gitlab.rb`): + +```ruby +gitlab_workhorse['env'] = { + 'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar' + 'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production' +} +``` + +Source installations (`/etc/default/gitlab`): + +```plaintext +export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar' +export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production' +``` + +## Distributed tracing + +Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) +using [OpenTracing APIs](https://opentracing.io). + +By default, no tracing implementation is linked into the binary. You can link in +different OpenTracing providers with [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) +or build constraints by setting the `BUILD_TAGS` make variable. + +For more details of the supported providers, refer to LabKit. For an example of +Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"` like this: + +```shell +make BUILD_TAGS="tracer_static tracer_static_jaeger" +``` + +After you compile Workhorse with an OpenTracing provider, configure the tracing +configuration with the `GITLAB_TRACING` environment variable, like this: + +```shell +GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse +``` + +## Continuous profiling + +Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) +using [Stackdriver Profiler](https://cloud.google.com/profiler). By default, the +Stackdriver Profiler implementation is linked in the binary using +[build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not +required and can be skipped. For example: + +```shell +make BUILD_TAGS="" +``` + +After you compile Workhorse with continuous profiling, set the profiler configuration +with the `GITLAB_CONTINUOUS_PROFILING` environment variable. For example: + +```shell +GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse" +``` + +## Related topics + +- [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go). diff --git a/doc/development/workhorse/gitlab_features.md b/doc/development/workhorse/gitlab_features.md new file mode 100644 index 00000000000..2aa8d9d2399 --- /dev/null +++ b/doc/development/workhorse/gitlab_features.md @@ -0,0 +1,73 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Features that rely on Workhorse + +Workhorse itself is not a feature, but there are several features in +GitLab that would not work efficiently without Workhorse. + +To put the efficiency benefit in context, consider that in 2020Q3 on +GitLab.com [we see](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1) +Rails application threads using on average +about 200MB of RSS vs about 200KB for Workhorse goroutines. + +Examples of features that rely on Workhorse: + +## 1. `git clone` and `git push` over HTTP + +Git clone, pull and push are slow because they transfer large amounts +of data and because each is CPU intensive on the GitLab side. Without +Workhorse, HTTP access to Git repositories would compete with regular +web access to the application, requiring us to run way more Rails +application servers. + +## 2. CI runner long polling + +GitLab CI runners fetch new CI jobs by polling the GitLab server. +Workhorse acts as a kind of "waiting room" where CI runners can sit +and wait for new CI jobs. Because of Go's efficiency we can fit a lot +of runners in the waiting room at little cost. Without this waiting +room mechanism we would have to add a lot more Rails server capacity. + +## 3. File uploads and downloads + +File uploads and downloads may be slow either because the file is +large or because the user's connection is slow. Workhorse can handle +the slow part for Rails. This improves the efficiency of features such +as CI artifacts, package repositories, LFS objects, etc. + +## 4. Websocket proxying + +Features such as the web terminal require a long lived connection +between the user's web browser and a container inside GitLab that is +not directly accessible from the internet. Dedicating a Rails +application thread to proxying such a connection would cost much more +memory than it costs to have Workhorse look after it. + +## Quick facts (how does Workhorse work) + +- Workhorse can handle some requests without involving Rails at all: + for example, JavaScript files and CSS files are served straight + from disk. +- Workhorse can modify responses sent by Rails: for example if you use + `send_file` in Rails then GitLab Workhorse will open the file on + disk and send its contents as the response body to the client. +- Workhorse can take over requests after asking permission from Rails. + Example: handling `git clone`. +- Workhorse can modify requests before passing them to Rails. Example: + when handling a Git LFS upload Workhorse first asks permission from + Rails, then it stores the request body in a tempfile, then it sends + a modified request containing the tempfile path to Rails. +- Workhorse can manage long-lived WebSocket connections for Rails. + Example: handling the terminal websocket for environments. +- Workhorse does not connect to PostgreSQL, only to Rails and (optionally) Redis. +- We assume that all requests that reach Workhorse pass through an + upstream proxy such as NGINX or Apache first. +- Workhorse does not accept HTTPS connections. +- Workhorse does not clean up idle client connections. +- We assume that all requests to Rails pass through Workhorse. + +For more information see ['A brief history of GitLab Workhorse'](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). diff --git a/doc/development/workhorse/index.md b/doc/development/workhorse/index.md new file mode 100644 index 00000000000..f7ca16e0f31 --- /dev/null +++ b/doc/development/workhorse/index.md @@ -0,0 +1,84 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Workhorse + +GitLab Workhorse is a smart reverse proxy for GitLab. It handles +"large" HTTP requests such as file downloads, file uploads, Git +push/pull and Git archive downloads. + +Workhorse itself is not a feature, but there are [several features in +GitLab](gitlab_features.md) that would not work efficiently without Workhorse. + +The canonical source for Workhorse is +[`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse). +Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was +[`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master), +but that repository is no longer used for development. + +## Install Workhorse + +To install GitLab Workhorse you need [Go 1.15 or newer](https://golang.org/dl) and +[GNU Make](https://www.gnu.org/software/make/). + +To install into `/usr/local/bin` run `make install`. + +```plaintext +make install +``` + +To install into `/foo/bin` set the PREFIX variable. + +```plaintext +make install PREFIX=/foo +``` + +On some operating systems, such as FreeBSD, you may have to use +`gmake` instead of `make`. + +*NOTE*: Some features depends on build tags, make sure to check +[Workhorse configuration](configuration.md) to enable them. + +### Run time dependencies + +Workhorse uses [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for +removing EXIF data (which may contain sensitive information) from uploaded +images. If you installed GitLab: + +- Using the Omnibus package, you're all set. + *NOTE* that if you are using CentOS Minimal, you may need to install `perl` + package: `yum install perl` +- From source, make sure `exiftool` is installed: + + ```shell + # Debian/Ubuntu + sudo apt-get install libimage-exiftool-perl + + # RHEL/CentOS + sudo yum install perl-Image-ExifTool + ``` + +## Testing your code + +Run the tests with: + +```plaintext +make clean test +``` + +Each feature in GitLab Workhorse should have an integration test that +verifies that the feature 'kicks in' on the right requests and leaves +other requests unaffected. It is better to also have package-level tests +for specific behavior but the high-level integration tests should have +the first priority during development. + +It is OK if a feature is only covered by integration tests. + +<!-- +## License + +This code is distributed under the MIT license, see the [LICENSE](LICENSE) file. +--> diff --git a/doc/development/workhorse/new_features.md b/doc/development/workhorse/new_features.md new file mode 100644 index 00000000000..3ad15c1de16 --- /dev/null +++ b/doc/development/workhorse/new_features.md @@ -0,0 +1,78 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Adding new features to Workhorse + +GitLab Workhorse is a smart reverse proxy for GitLab. It handles +[long HTTP requests](#what-are-long-requests), such as: + +- File downloads. +- File uploads. +- Git pushes and pulls. +- Git archive downloads. + +Workhorse itself is not a feature, but [several features in GitLab](gitlab_features.md) +would not work efficiently without Workhorse. + +At a first glance, Workhorse appears to be just a pipeline for processing HTTP +streams to reduce the amount of logic in your Ruby on Rails controller. However, +don't treat it that way. Engineers trying to offload a feature to Workhorse often +find it takes more work than originally anticipated: + +- It's a new programming language, and only a few engineers at GitLab are Go developers. +- Workhorse has demanding requirements: + - It's stateless. + - Memory and disk usage must be kept under tight control. + - The request should not be slowed down in the process. + +## Avoid adding new features + +We suggest adding new features only if absolutely necessary and no other options exist. +Splitting a feature between the Rails codebase and Workhorse is a deliberate choice +to introduce technical debt. It adds complexity to the system, and coupling between +the two components: + +- Building features using Workhorse has a considerable complexity cost, so you should + prefer designs based on Rails requests and Sidekiq jobs. +- Even when using Rails-and-Sidekiq is more work than using Rails-and-Workhorse, + Rails-and-Sidekiq is easier to maintain in the long term. Workhorse is unique + to GitLab, while Rails-and-Sidekiq is an industry standard. +- For global behaviors around web requests, consider using a Rack middleware + instead of Workhorse. +- Generally speaking, use Rails-and-Workhorse only if the HTTP client expects + behavior reasonable to implement in Rails, like long requests. + +## What are long requests? + +One order of magnitude exists between Workhorse and Puma RAM usage. Having a connection +open for longer than milliseconds is problematic due to the amount of RAM +it monopolizes after it reaches the Ruby on Rails controller. We've identified two classes +of long requests: data transfers and HTTP long polling. Some examples: + +- `git push`. +- `git pull`. +- Uploading or downloading an artifact. +- A CI runner waiting for a new job. + +With the rise of cloud-native installations, Workhorse's feature set was extended +to add object storage direct-upload. This change removed the need for the shared +Network File System (NFS) drives. + +If you still think we should add a new feature to Workhorse, open an issue for the +Workhorse maintainers and explain: + +1. What you want to implement. +1. Why it can't be implemented in our Ruby codebase. + +The Workhorse maintainers can help you assess the situation. + +## Related topics + +- In 2020, `@nolith` presented the talk + ["Speed up the monolith. Building a smart reverse proxy in Go"](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/) + at FOSDEM. The talk includes more details on the history of Workhorse and the NFS removal. +- The [uploads development documentation](../uploads.md) contains the most common + use cases for adding a new type of upload. diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md index 176189298c8..d22ce12f9cd 100644 --- a/doc/gitlab-basics/create-branch.md +++ b/doc/gitlab-basics/create-branch.md @@ -1,19 +1,11 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: howto +redirect_to: '../tutorials/make_your_first_git_commit.md' +remove_date: '2022-06-26' --- -# How to create a branch **(FREE)** +This document was moved to [another location](../tutorials/make_your_first_git_commit.md). -A branch is an independent line of development in a [project](../user/project/index.md). - -When you create a branch (in your [terminal](start-using-git.md#create-a-branch) or with -[the web interface](../user/project/repository/web_editor.md#create-a-new-branch)), -you are creating a snapshot of a certain branch, usually the main branch, -at its current state. From there, you can start to make your own changes without -affecting the main codebase. The history of your changes is tracked in your branch. - -When your changes are ready, you then merge them into the rest of the codebase with a -[merge request](../user/project/merge_requests/creating_merge_requests.md). +<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/gitlab-basics/index.md b/doc/gitlab-basics/index.md index 5ba5366eafa..3cd5dfe7d18 100644 --- a/doc/gitlab-basics/index.md +++ b/doc/gitlab-basics/index.md @@ -1,49 +1,11 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -comments: false -type: index +redirect_to: '../user/index.md' +remove_date: '2022-07-08' --- -# GitLab basics guides **(FREE)** +This document was moved to [another location](../user/index.md). -This section provides resources to help you start working with GitLab and Git by focusing -on the basic features that you must use. - -This documentation is split into the following groups: - -- [GitLab-specific functionality](#gitlab-basics), for basic GitLab features. -- [General Git functionality](#working-with-git-from-the-command-line), for working - with Git in conjunction with GitLab. - -## GitLab basics - -The following are guides to basic GitLab functionality: - -- [Create and add your SSH public key](../ssh/index.md), for enabling Git over SSH. -- [Create a project](../user/project/working_with_projects.md#create-a-project), to start using GitLab. -- [Create a group](../user/group/index.md#create-a-group), to combine and administer - projects together. -- [Create a branch](create-branch.md), to make changes to files stored in a project's repository. -- [Feature branch workflow](feature_branch_workflow.md). -- [Fork a project](../user/project/repository/forking_workflow.md#creating-a-fork), to duplicate projects so they can be worked on in parallel. -- [Add a file](add-file.md), to add new files to a project's repository. -- [Create an issue](../user/project/issues/managing_issues.md#create-an-issue), - to start collaborating within a project. -- [Create a merge request](../user/project/merge_requests/creating_merge_requests.md), to request changes made in a branch - be merged into a project's repository. -- See how these features come together in the [GitLab Flow introduction video](https://youtu.be/InKNIvky2KE) - and [GitLab Flow page](../topics/gitlab_flow.md). - -## Working with Git from the command line - -If you're familiar with Git on the command line, you can interact with your GitLab -projects just as you would with any other Git repository. - -These resources can help you get further acclimated to working on the command line. - -- [Start using Git on the command line](start-using-git.md), for some simple Git commands. -- [Command line basics](command-line-commands.md), to create and edit files using the command line. - -More Git resources are available in the GitLab [Git documentation](../topics/git/index.md). +<!-- This redirect file can be deleted after 2022-07-08. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index dfd1f09e297..73a993c0fcf 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -6,7 +6,7 @@ type: howto, tutorial description: "Introduction to using Git through the command line." --- -# Start using Git on the command line **(FREE)** +# Git on the command line **(FREE)** [Git](https://git-scm.com/) is an open-source distributed version control system. GitLab is built on top of Git. @@ -14,6 +14,9 @@ on top of Git. You can do many Git operations directly in GitLab. However, the command line is required for advanced tasks, like fixing complex merge conflicts or rolling back commits. +If you're new to Git and want to learn by working in your own project, +[learn how to make your first commit](../tutorials/make_your_first_git_commit.md). + For a quick reference of Git commands, download a [Git Cheat Sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf). For more information about the advantages of working with Git and GitLab: @@ -24,75 +27,7 @@ For more information about the advantages of working with Git and GitLab: To help you visualize what you're doing locally, you can install a [Git GUI app](https://git-scm.com/download/gui/). -## Git terminology - -If you're familiar with Git terminology, you might want to skip this section and -go directly to [prerequisites](#prerequisites). - -### Repository - -In GitLab, files are stored in a **repository**. A repository is similar to how you -store files in a folder or directory on your computer. - -- A **remote repository** refers to the files in GitLab. -- A **local copy** refers to the files on your computer. - -<!-- vale gitlab.Spelling = NO --> -<!-- vale gitlab.SubstitutionWarning = NO --> -Often, the word "repository" is shortened to "repo". -<!-- vale gitlab.Spelling = YES --> -<!-- vale gitlab.SubstitutionWarning = YES --> - -In GitLab, a repository is contained in a **project**. - -### Fork - -When you want to contribute to someone else's repository, you make a copy of it. -This copy is called a [**fork**](../user/project/repository/forking_workflow.md#creating-a-fork). -The process is called "creating a fork." - -When you fork a repo, you create a copy of the project in your own -[namespace](../user/group/#namespaces). You then have write permissions to modify the project files -and settings. - -For example, you can fork this project, <https://gitlab.com/gitlab-tests/sample-project/>, into your namespace. -You now have your own copy of the repository. You can view the namespace in the URL, for example -`https://gitlab.com/your-namespace/sample-project/`. -Then you can clone the repository to your local machine, work on the files, and submit changes back to the -original repository. - -### Difference between download and clone - -To create a copy of a remote repository's files on your computer, you can either -**download** or **clone** the repository. If you download it, you cannot sync the repository with the -remote repository on GitLab. - -[Cloning](#clone-a-repository) a repository is the same as downloading, except it preserves the Git connection -with the remote repository. You can then modify the files locally and -upload the changes to the remote repository on GitLab. - -### Pull and push - -After you save a local copy of a repository and modify the files on your computer, you can upload the -changes to GitLab. This is referred to as **pushing** to the remote, because you use the command -[`git push`](#send-changes-to-gitlabcom). - -When the remote repository changes, your local copy is behind. You can update your local copy with the new -changes in the remote repository. -This is referred to as **pulling** from the remote, because you use the command -[`git pull`](#download-the-latest-changes-in-the-project). - -## Prerequisites - -To start using GitLab with Git, complete the following tasks: - -- Create and sign in to a GitLab account. -- [Open a terminal](#open-a-terminal). -- [Install Git](#install-git) on your computer. -- [Configure Git](#configure-git). -- [Choose a repository](#choose-a-repository). - -### Open a terminal +## Choose a terminal To execute Git commands on your computer, you must open a terminal (also known as command prompt, command shell, and command line). Here are some options: @@ -107,9 +42,9 @@ prompt, command shell, and command line). Here are some options: - For Linux users: - Built-in [Linux Terminal](https://ubuntu.com/tutorials/command-line-for-beginners#3-opening-a-terminal). -### Install Git +## Confirm Git is installed -Determine if Git is already installed on your computer by opening a terminal +You can determine if Git is already installed on your computer by opening a terminal and running this command: ```shell @@ -123,9 +58,8 @@ git version X.Y.Z ``` If your computer doesn't recognize `git` as a command, you must [install Git](../topics/git/how_to_install_git/index.md). -After you install Git, run `git --version` to confirm that it installed correctly. -### Configure Git +## Configure Git To start using Git from your computer, you must enter your credentials to identify yourself as the author of your work. The username and email address @@ -156,7 +90,7 @@ should match the ones you use in GitLab. You can read more on how Git manages configurations in the [Git configuration documentation](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration). -### Choose a repository +## Choose a repository Before you begin, choose the repository you want to work in. You can use any project you have permission to access on GitLab.com or any other GitLab instance. @@ -182,7 +116,7 @@ This connection requires you to add credentials. You can either use SSH or HTTPS Clone with SSH when you want to authenticate only one time. -1. Authenticate with GitLab by following the instructions in the [SSH documentation](../ssh/index.md). +1. Authenticate with GitLab by following the instructions in the [SSH documentation](../user/ssh.md). 1. Go to your project's landing page and select **Clone**. Copy the URL for **Clone with SSH**. 1. Open a terminal and go to the directory where you want to clone the files. Git automatically creates a folder with the repository name and downloads the files there. 1. Run this command: diff --git a/doc/index.md b/doc/index.md index 57eb9e195ff..2f3bfefe819 100644 --- a/doc/index.md +++ b/doc/index.md @@ -32,7 +32,7 @@ No matter how you use GitLab, we have documentation for you. | Essential documentation | Essential documentation | |:------------------------|:------------------------| | [**User documentation**](user/index.md)<br>Discover features and concepts for GitLab users. | [**Administrator documentation**](administration/index.md)<br/>Everything GitLab self-managed administrators need to know. | -| [**Contributing to GitLab**](#contributing-to-gitlab)<br/>At GitLab, everyone can contribute! | [**New to Git and GitLab?**](#new-to-git-and-gitlab)<br/>We have the resources to get you started. | +| [**Contributing to GitLab**](#contributing-to-gitlab)<br/>At GitLab, everyone can contribute! | [**New to Git and GitLab?**](tutorials/index.md)<br/>We have the resources to get you started. | | [**Build an integration with GitLab**](#build-an-integration-with-gitlab)<br/>Consult our integration documentation. | [**Coming to GitLab from another platform?**](#coming-to-gitlab-from-another-platform)<br/>Consult our guides. | | [**Install GitLab**](https://about.gitlab.com/install/)<br/>Installation options for different platforms. | [**Customers**](subscriptions/index.md)<br/>Information for new and existing customers. | | [**Update GitLab**](update/index.md)<br/>Update your GitLab self-managed instance to the latest version. | [**Reference Architectures**](administration/reference_architectures/index.md)<br/>GitLab reference architectures. | @@ -64,20 +64,6 @@ GitLab makes the software lifecycle faster and radically improves the speed of b GitLab provides solutions for [each of the stages of the DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/). -## New to Git and GitLab? - -Working with new systems can be daunting. - -We have the following documentation to rapidly uplift your GitLab knowledge: - -| Topic | Description | -|:--------------------------------------------------------------------------------------------------|:------------| -| [GitLab basics guides](gitlab-basics/index.md) | Start working on the command line and with GitLab. | -| [What is GitLab Flow?](https://about.gitlab.com/topics/version-control/what-is-gitlab-flow/) | Enhance your workflow with the best of GitLab Flow. | -| [Get started with GitLab CI/CD](ci/quick_start/index.md) | Quickly implement GitLab CI/CD. | -| [Auto DevOps](topics/autodevops/index.md) | Learn more about Auto DevOps in GitLab. | -| [GitLab Markdown](user/markdown.md) | Advanced formatting system (GitLab Flavored Markdown). | - ### User account Learn more about GitLab account management: @@ -89,16 +75,6 @@ Learn more about GitLab account management: | [User settings](user/profile/index.md#access-your-user-settings) | Manage your user settings, two factor authentication, and more. | | [User permissions](user/permissions.md) | Learn what each role in a project can do. | -### Git and GitLab - -Learn more about using Git, and using Git with GitLab: - -| Topic | Description | -|:-----------------------------------------------------------------------------|:------------| -| [Git](topics/git/index.md) | Getting started with Git, branching strategies, Git LFS, and advanced use. | -| [Git cheat sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) | Download a PDF describing the most used Git operations. | -| [GitLab Flow](topics/gitlab_flow.md) | Explore the best of Git with the GitLab Flow strategy. | - ## Coming to GitLab from another platform If you are coming to GitLab from another platform, the following information is useful: diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index b0e71cbc77e..8b827d05b57 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -299,8 +299,8 @@ The steps for doing this vary depending on which registrar you use and is beyond ## PostgreSQL with RDS -For our database server we will use Amazon RDS which offers Multi AZ -for redundancy. First we'll create a security group and subnet group, then we'll +For our database server we will use Amazon RDS for PostgreSQL which offers Multi AZ +for redundancy (Aurora is **not** supported). First we'll create a security group and subnet group, then we'll create the actual RDS instance. ### RDS Security Group diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 06518ff58de..780dfe17dac 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -71,7 +71,7 @@ The first items you need to configure are the basic settings of the underlying v the user Azure uses to connect to the VM through SSH. By default, the user has root access. 1. Determine if you want to provide your own SSH key or let Azure create one for you. - Read the [SSH documentation](../../ssh/index.md) to learn more about how to set up SSH + Read the [SSH documentation](../../user/ssh.md) to learn more about how to set up SSH public keys. Review your entered settings, and then proceed to the Disks tab. diff --git a/doc/install/cloud_native/index.md b/doc/install/cloud_native/index.md new file mode 100644 index 00000000000..45d484b045a --- /dev/null +++ b/doc/install/cloud_native/index.md @@ -0,0 +1,51 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +comments: false +description: Install GitLab in a cloud native environment +type: index +--- + +# Cloud Native GitLab **(FREE SELF)** + +[Cloud Native GitLab](https://gitlab.com/gitlab-org/build/CNG) provides cloud +native containers to deploy GitLab. These containers may be deployed and managed +via Helm using GitLab Charts or GitLab Operator on Kubernetes, OpenShift, +and Kubernetes compatible container platforms: + +- [Helm charts](https://docs.gitlab.com/charts/): The cloud native Helm chart + installs GitLab and all of its components on Kubernetes. Use this method if + your infrastructure is built on Kubernetes and you're familiar with how it + works. The methods for management, observability, and some concepts are + different than traditional deployments. +- [GitLab Operator](https://docs.gitlab.com/operator/): The GitLab Operator + provides an installation and management method for GitLab following the + [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). + You can also use the GitLab Operator to run GitLab in an + [OpenShift](../openshift_and_gitlab/index.md) environment. + +Here's an overview of how the containers are built: + +```mermaid +graph TD + subgraph Code + CNG --> HC + CNG --> GOP + HC --> GOP + end + + subgraph Deploy + GOP --> K8s + GOP --> OS + CNG --> DC + HC --> K8s + end + + CNG[Cloud Native GitLab containers] + HC[Helm Chart] + K8s(Kubernetes) + GOP[GitLab Operator] + OS(OpenShift) + DC(Docker Compose) +``` diff --git a/doc/install/index.md b/doc/install/index.md index b27d3683cd5..deb94031e44 100644 --- a/doc/install/index.md +++ b/doc/install/index.md @@ -29,10 +29,10 @@ install GitLab: |----------------------------------------------------------------|-------------|----------------| | [Linux package](https://docs.gitlab.com/omnibus/installation/) | The official deb/rpm packages (also known as Omnibus GitLab) that contains a bundle of GitLab and the components it depends on, including PostgreSQL, Redis, and Sidekiq. | This method is recommended for getting started. The Linux packages are mature, scalable, and are used today on GitLab.com. If you need additional flexibility and resilience, we recommend deploying GitLab as described in the [reference architecture documentation](../administration/reference_architectures/index.md). | | [Helm charts](https://docs.gitlab.com/charts/) | The cloud native Helm chart for installing GitLab and all of its components on Kubernetes. | When installing GitLab on Kubernetes, there are some trade-offs that you need to be aware of: <br/>- Administration and troubleshooting requires Kubernetes knowledge.<br/>- It can be more expensive for smaller installations. The default installation requires more resources than a single node Linux package deployment, as most services are deployed in a redundant fashion.<br/><br/> Use this method if your infrastructure is built on Kubernetes and you're familiar with how it works. The methods for management, observability, and some concepts are different than traditional deployments. | -| [Docker](https://docs.gitlab.com/omnibus/docker/) | The GitLab packages, Dockerized. | Use this method if you're familiar with Docker. | +| [Docker](docker.md) | The GitLab packages, Dockerized. | Use this method if you're familiar with Docker. | | [Source](installation.md) | Install GitLab and all of its components from scratch. | Use this method if none of the previous methods are available for your platform. Useful for unsupported systems like \*BSD.| -| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#documentation) | The GitLab Environment toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Customers are very welcome to trial and evaluate GET today, however be aware of [key limitations](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#missing-features-to-be-aware-of) of the current iteration. For production environments further manual setup will be required based on your specific requirements. | -| [GitLab Operator](https://docs.gitlab.com/charts/installation/operator.html) | The GitLab Operator provides an installation and management method for GitLab following the [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). | Use the GitLab Operator to run GitLab in an [OpenShift](openshift_and_gitlab/index.md) environment. | +| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#documentation) | The GitLab Environment Toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Customers are very welcome to trial and evaluate GET today, however be aware of [key limitations](https://gitlab.com/gitlab-org/gitlab-environment-toolkit#missing-features-to-be-aware-of) of the current iteration. For production environments further manual setup will be required based on your specific requirements. | +| [GitLab Operator](https://docs.gitlab.com/operator/) | The GitLab Operator provides an installation and management method for GitLab following the [Kubernetes Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/). | Use the GitLab Operator to run GitLab in an [OpenShift](openshift_and_gitlab/index.md) environment. | ## Install GitLab on cloud providers diff --git a/doc/install/installation.md b/doc/install/installation.md index 21d1ee84722..c03c600fe0b 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -51,7 +51,7 @@ If the highest number stable branch is unclear, check the [GitLab blog](https:// | [Ruby](#2-ruby) | `2.7` | From GitLab 13.6, Ruby 2.7 is required. Ruby 3.0 is not supported yet (see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/5149) for the current status). You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | | [Go](#3-go) | `1.16` | | | [Git](#git) | `2.33.x` | From GitLab 14.4, Git 2.33.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | -| [Node.js](#4-node) | `12.22.1` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 14.x is recommended, as it's faster. You can check which version you're running with `node -v`. You need to update it to a newer version if needed. | +| [Node.js](#4-node) | `14.15.0` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 16.x is recommended, as it's faster. You can check which version you're running with `node -v`. You need to update it to a newer version if needed. | ## GitLab directory structure @@ -263,7 +263,7 @@ GitLab requires the use of Node to compile JavaScript assets, and Yarn to manage JavaScript dependencies. The current minimum requirements for these are: -- `node` >= v12.22.1. (We recommend node 14.x as it is faster) +- `node` >= v14.15.0. (We recommend node 16.x as it is faster) - `yarn` = v1.22.x (Yarn 2 is not supported yet) In many distributions, @@ -271,8 +271,8 @@ the versions provided by the official package repositories are out of date, so we need to install through the following commands: ```shell -# install node v14.x -curl --location "https://deb.nodesource.com/setup_14.x" | sudo bash - +# install node v16.x +curl --location "https://deb.nodesource.com/setup_16.x" | sudo bash - sudo apt-get install -y nodejs npm install --global yarn @@ -1121,7 +1121,7 @@ host localhost # Give your setup a name (here: override localhost) hostname 127.0.0.1; # Your server name or IP ``` -You also need to change the corresponding options (for example, `ssh_user`, `ssh_host`, `admin_uri`) in the `config\gitlab.yml` file. +You also need to change the corresponding options (for example, `ssh_user`, `ssh_host`, `admin_uri`) in the `config/gitlab.yml` file. ### Additional Markup Styles diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md deleted file mode 100644 index 56dde411884..00000000000 --- a/doc/install/pivotal/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../index.md' -remove_date: '2022-03-08' ---- - -This document was removed. For information about installing GitLab, see [this page](../index.md). - -<!-- This redirect file can be deleted after <2022-03-08>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 11f623641c1..8006b886414 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -13,11 +13,11 @@ the minimum requirements needed to install and use GitLab. ### Supported Linux distributions -- Ubuntu (16.04/18.04/20.04) -- Debian (9/10) +- Ubuntu (18.04/20.04) +- Debian (9/10/11) - AlmaLinux (8) - CentOS (7) -- openSUSE Leap (15.2) +- openSUSE Leap (15.3) - SUSE Linux Enterprise Server (12 SP2/12 SP5) - Red Hat Enterprise Linux (use the AlmaLinux or CentOS instructions) - Scientific Linux (use the CentOS instructions) @@ -120,7 +120,7 @@ the following table) as these were used for development and testing: | GitLab version | Minimum PostgreSQL version | |----------------|----------------------------| | 13.0 | 11 | -| 14.0 | 12 | +| 14.0 | 12.10 | You must also ensure the following extensions are loaded into every GitLab database. [Read more about this requirement, and troubleshooting](postgresql_extensions.md). diff --git a/doc/integration/alicloud.md b/doc/integration/alicloud.md new file mode 100644 index 00000000000..85e1e2d4154 --- /dev/null +++ b/doc/integration/alicloud.md @@ -0,0 +1,91 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Use AliCloud as an OmniAuth authentication provider **(FREE)** + +You can enable the AliCloud OAuth 2.0 OmniAuth provider and sign in to +GitLab using your AliCloud account. + +## Create an AliCloud application + +Sign in to the AliCloud platform and create an application on it. AliCloud generates a client ID and secret key for you to use. + +1. Sign in to the [AliCloud platform](https://account.aliyun.com/login/login.htm). + +1. Go to the [OAuth application management page](https://ram.console.aliyun.com/applications). + +1. Select **Create Application**. + +1. Fill in the application details: + + - **Application Name**: This can be anything. + - **Display Name**: This can be anything. + - **Callback URL**: This URL should be formatted as `'GitLab instance URL' + '/users/auth/alicloud/callback'`. For example, `http://test.gitlab.com/users/auth/alicloud/callback`. + + Select **Save**. + +1. Add OAuth scopes in the application details page: + + 1. Under the **Application Name** column, select the name of the application you created. The application's details page opens. + 1. Under the **Application OAuth Scopes** tab, select **Add OAuth Scopes**. + 1. Select the **aliuid** and **profile** checkboxes. + 1. Select **OK**. + +  + +1. Create a secret in the application details page: + + 1. Under the **App Secrets** tab, select **Create Secret**. + 1. Copy the SecretValue generated. + +## Enable AliCloud OAuth in GitLab + +1. On your GitLab server, open the configuration file. + + - **For Omnibus installations** + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + - **For installations from source** + + ```shell + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. [Configure the initial settings](omniauth.md#configure-initial-settings). + +1. Add the provider configuration. Replace `YOUR_APP_ID` with the ID on the application details page + and `YOUR_APP_SECRET` with the **SecretValue** you got when you registered the AliCloud application. + + - **For Omnibus installations** + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: "alicloud", + app_id: "YOUR_APP_ID", + app_secret: "YOUR_APP_SECRET" + } + ] + ``` + + - **For installations from source** + + ```yaml + - { name: 'alicloud', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Save the configuration file. + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + if you installed using Omnibus, or [restart GitLab](../administration/restart_gitlab.md#installations-from-source) + if you installed from source. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 5265a24d299..5197a191b8e 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -982,7 +982,7 @@ There is also an easy way to check it automatically with `sudo gitlab-rake gitla This exception is seen when your Elasticsearch cluster is configured to reject requests above a certain size (10MiB in this case). This corresponds to the `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a larger size and restart your Elasticsearch cluster. -AWS has [fixed limits](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/aes-limits.html) for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of the underlying instance. +AWS has [fixed limits](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/limits.html#network-limits) for this setting ("Maximum size of HTTP request payloads"), based on the size of the underlying instance. ### My single node Elasticsearch cluster status never goes from `yellow` to `green` even though everything seems to be running properly @@ -1080,6 +1080,12 @@ If `ElasticCommitIndexerWorker` Sidekiq workers are failing with this error duri - To decrease the indexing throughput you can decrease `Bulk request concurrency` (see [Advanced Search settings](#advanced-search-configuration)). This is set to `10` by default, but you change it to as low as 1 to reduce the number of concurrent indexing operations. - If changing `Bulk request concurrency` didn't help, you can use the [queue selector](../administration/operations/extra_sidekiq_processes.md#queue-selector) option to [limit indexing jobs only to specific Sidekiq nodes](#index-large-instances-with-dedicated-sidekiq-nodes-or-processes), which should reduce the number of indexing requests. +### Indexing is very slow or fails with `rejected execution of coordinating operation` messages + +Bulk requests are getting rejected by the Elasticsearch node(s) likely due to load and lack of available memory. +Ensure that your Elasticsearch cluster meets the [system requirements](#system-requirements) and has enough resources +to perform bulk operations. See also the error ["429 (Too Many Requests)"](#indexing-fails-with-error-elastic-error-429-too-many-requests). + ### Access requirements for the self-managed AWS OpenSearch Service To use the self-managed AWS OpenSearch Service with GitLab, configure your instance's domain access policies diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index 977e794396e..f54542ff43f 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -28,7 +28,7 @@ To use the GitLab Gitpod integration, it must be enabled for your GitLab instanc 1. It's [enabled and configured by a GitLab administrator](#configure-a-self-managed-instance). 1. It's [enabled in their user settings](#enable-gitpod-in-your-user-settings). -To learn more about Gitpod, see their [features](https://www.gitpod.io/features) and +To learn more about Gitpod, see their [features](https://www.gitpod.io/) and [documentation](https://www.gitpod.io/docs/). ## Enable Gitpod in your user settings diff --git a/doc/integration/img/alicloud_scope.png b/doc/integration/img/alicloud_scope.png Binary files differnew file mode 100644 index 00000000000..7a52d97ecd5 --- /dev/null +++ b/doc/integration/img/alicloud_scope.png diff --git a/doc/integration/index.md b/doc/integration/index.md index 61d8547aaf7..b26c841f943 100644 --- a/doc/integration/index.md +++ b/doc/integration/index.md @@ -24,8 +24,8 @@ GitLab can be configured to authenticate access requests with the following auth - Integrate with [Kerberos](kerberos.md). - Enable sign in via [LDAP](../administration/auth/ldap/index.md). - Enable [OAuth2 provider](oauth_provider.md) application creation. -- Use [OmniAuth](omniauth.md) to enable sign in via Twitter, GitHub, GitLab.com, Google, - Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure, or Authentiq ID. +- Use [OmniAuth](omniauth.md) to enable sign in through Twitter, GitHub, GitLab.com, Google, + Bitbucket, Facebook, SAML, Crowd, Azure, or Authentiq ID. - Use GitLab as an [OpenID Connect](openid_connect_provider.md) identity provider. - Authenticate to [Vault](vault.md) through GitLab OpenID Connect. - Configure GitLab as a [SAML](saml.md) 2.0 Service Provider. @@ -89,7 +89,7 @@ at Super User also has relevant information. **Omnibus Trusted Chain** -[Install the self signed certificate or custom certificate authorities](https://docs.gitlab.com/omnibus/common_installation_problems/index.html#using-self-signed-certificate-or-custom-certificate-authorities) +[Install the self signed certificate or custom certificate authorities](https://docs.gitlab.com/omnibus/troubleshooting.html#using-self-signed-certificate-or-custom-certificate-authorities) in to Omnibus GitLab. It is enough to concatenate the certificate to the main trusted certificate diff --git a/doc/integration/jira/configure.md b/doc/integration/jira/configure.md index 2033ddbad6f..bfeac230f89 100644 --- a/doc/integration/jira/configure.md +++ b/doc/integration/jira/configure.md @@ -22,7 +22,8 @@ Prerequisites: To configure your project: -1. Go to your project and select [**Settings > Integrations**](../../user/project/integrations/overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Jira**. 1. Select **Enable integration**. 1. Select **Trigger** actions. Your choice determines whether a mention of Jira issue diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 59cdba93543..5c8f78a94b1 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -94,7 +94,7 @@ from outside the Marketplace, which allows you to install the application: 1. Place your Jira instance into [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode). 1. Sign in to your GitLab application as a user with administrator access. -1. Install the GitLab application from your self-managed GitLab instance, as +1. Install the GitLab application from your Jira instance, as described in the [Atlassian developer guides](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-3--install-and-test-your-app): 1. In your Jira instance, go to **Apps > Manage Apps** and select **Upload app**: diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md index 66810945d19..2f0ebea165a 100644 --- a/doc/integration/jira/development_panel.md +++ b/doc/integration/jira/development_panel.md @@ -4,25 +4,25 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Jira Development panel integration **(FREE)** +# GitLab Jira development panel integration **(FREE)** > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) from GitLab Premium to GitLab Free in 13.4. -With the Jira Development panel integration, you can reference Jira issues in GitLab. +With the Jira development panel integration, you can reference Jira issues in GitLab. When configured, activity (such as pipeline, deployment, and feature flags) displays in the Jira issue's -[Development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). -From the Development panel, you can open a detailed view and +[development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). +From the development panel, you can open a detailed view and [take various actions](#use-the-integration), including creating a new merge request from a branch:  -The information displayed in the Jira Development panel depends on where you mention the Jira issue ID: +The information displayed in the Jira development panel depends on where you mention the Jira issue ID: | Your mention of Jira issue ID in GitLab context | Automated effect in Jira issue | |---------------------------------------------------|--------------------------------------------------------------------------------------------------------| -| In a merge request title or description | Link to the MR is displayed in Development panel. | -| In a branch name | Link to the branch is displayed in Development panel. | -| In a commit message | Link to the commit is displayed in Development panel. | +| In a merge request title or description | Link to the MR is displayed in the development panel. | +| In a branch name | Link to the branch is displayed in the development panel. | +| In a commit message | Link to the commit is displayed in the development panel. | | In a commit message with Jira [Smart Commits](https://confluence.atlassian.com/fisheye/using-smart-commits-960155400.html) | Displays your custom comment or logged time spent and/or performs specified issue transition on merge. | This integration connects all GitLab projects to projects in the Jira instance in either: @@ -61,8 +61,8 @@ an issue transition, or add a custom comment, read the Atlassian page ## Configure the integration <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of how to configure Jira Development panel integration, see -[Agile Management - GitLab Jira Development panel integration](https://www.youtube.com/watch?v=VjVTOmMl85M). +For an overview of how to configure the Jira development panel integration, see +[Agile Management - GitLab Jira development panel integration](https://www.youtube.com/watch?v=VjVTOmMl85M). To simplify administration, we recommend that a GitLab group maintainer or group owner (or, if possible, instance administrator in the case of self-managed GitLab) set up the integration. @@ -89,7 +89,7 @@ This integration is not supported on GitLab instances under a [relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab). For example, `http://example.com/gitlab`. -## Troubleshoot the Development Panel +## Troubleshoot the development panel If you use Jira on your own server, go to the [Atlassian documentation](https://confluence.atlassian.com/jirakb/troubleshoot-the-development-panel-in-jira-server-574685212.html) for general troubleshooting information. diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index 3052d85b2cb..371f3a4ab8e 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -56,7 +56,7 @@ or the Jira DVCS (distributed version control system) connector, ## Authentication in Jira The authentication method in Jira depends on whether you host Jira on your own server or on -[Atlassian cloud](https://www.atlassian.com/cloud): +[Atlassian cloud](https://www.atlassian.com/migration/assess/why-cloud): - **Jira Server** supports basic authentication. When connecting, a **username and password** are required. Connecting to Jira Server using the Central Authentication Service (CAS) is not possible. For more information, read diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md index e24862242e1..28998851697 100644 --- a/doc/integration/jira/issues.md +++ b/doc/integration/jira/issues.md @@ -106,7 +106,7 @@ sorts by **Created date** by default, with the newest issues listed at the top:  -- To display the most recently updated issues first, select **Last updated**. +- To display the most recently updated issues first, select **Updated date**. - You can [search and filter](#search-and-filter-the-issues-list) the issues list. - In GitLab [versions 13.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/299832), you can select an issue from the list to view it in GitLab: diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 6ac4faa5a35..c8e2df1f88f 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -477,7 +477,7 @@ Mattermost System Console or manually. If a configuration setting is specified via both the `gitlab.rb` (as an environment variable) and `config.json` files, the environment variable gets precedence. -If you encounter any issues [visit the GitLab Mattermost troubleshooting forum](https://forum.mattermost.org/t/upgrading-to-gitlab-mattermost-in-gitlab-8-9/1735) and share any relevant portions of `mattermost.log` along with the step at which you encountered issues. +If you encounter any issues [visit the GitLab Mattermost troubleshooting forum](https://forum.mattermost.com/t/upgrading-to-gitlab-mattermost-in-gitlab-8-9/1735) and share any relevant portions of `mattermost.log` along with the step at which you encountered issues. ### Upgrading GitLab Mattermost outside of GitLab @@ -523,7 +523,7 @@ You can fix this by setting up a `mattermost-cli` [shell alias](#mattermost-comm For help and support around your GitLab Mattermost deployment please see: -- [Troubleshooting Forum](https://forum.mattermost.org/t/how-to-use-the-troubleshooting-forum/150) for configuration questions and issues. +- [Troubleshooting Forum](https://forum.mattermost.com/t/how-to-use-the-troubleshooting-forum/150) for configuration questions and issues. - [Troubleshooting FAQ](https://docs.mattermost.com/install/troubleshooting.html). - [Mattermost GitLab Issues Support Handbook](https://docs.mattermost.com/process/support.html?highlight=omnibus#gitlab-issues). - [GitLab Mattermost issue tracker](https://gitlab.com/gitlab-org/gitlab-mattermost/-/issues) for verified bugs with repro steps. diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 7a4bcba25e4..f6e41e808af 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -21,6 +21,7 @@ GitLab supports the following OmniAuth providers. | Provider documentation | OmniAuth provider name | |---------------------------------------------------------------------|----------------------------| +| [AliCloud](alicloud.md) | `alicloud` | | [Atlassian Crowd](../administration/auth/crowd.md) | `crowd` | | [Atlassian](../administration/auth/atlassian.md) | `atlassian_oauth2` | | [Auth0](auth0.md) | `auth0` | @@ -41,7 +42,6 @@ GitLab supports the following OmniAuth providers. | [OpenID Connect](../administration/auth/oidc.md) | `openid_connect` | | [Salesforce](salesforce.md) | `salesforce` | | [SAML](saml.md) | `saml` | -| [Shibboleth](saml.md) | `shibboleth` | | [Twitter](twitter.md) | `twitter` | ## Configure initial settings @@ -53,7 +53,7 @@ Setting | Description | Default value ---------------------------|-------------|-------------- `allow_single_sign_on` | Enables you to list the providers that automatically create a GitLab account. The provider names are available in the **OmniAuth provider name** column in the [supported providers table](#supported-providers). | The default is `false`. If `false`, users must be created manually, or they can't sign in using OmniAuth. `auto_link_ldap_user` | If enabled, creates an LDAP identity in GitLab for users that are created through an OmniAuth provider. You can enable this setting if you have the [LDAP (ActiveDirectory)](../administration/auth/ldap/index.md) integration enabled. Requires the `uid` of the user to be the same in both LDAP and the OmniAuth provider. | The default is `false`. -`block_auto_created_users` | If enabled, blocks users that are automatically created from signing in until they are approved by an administrator. | The default is `true`. If you set the value to `false`, make sure you only define providers for `allow_single_sign_on` that you can control, like SAML, Shibboleth, Crowd, or Google. Otherwise, any user on the internet can sign in to GitLab without an administrator's approval. +`block_auto_created_users` | If enabled, blocks users that are automatically created from signing in until they are approved by an administrator. | The default is `true`. If you set the value to `false`, make sure you only define providers for `allow_single_sign_on` that you can control, like SAML, Crowd, or Google. Otherwise, any user on the internet can sign in to GitLab without an administrator's approval. To change these settings: @@ -197,7 +197,7 @@ To enable automatic linking for SAML, see the [SAML setup instructions](saml.md# ## Create an external providers list You can define a list of external OmniAuth providers. -Users who create accounts or sign in to GitLab through the listed providers do not get access to [internal projects](../public_access/public_access.md#internal-projects-and-groups). +Users who create accounts or sign in to GitLab through the listed providers do not get access to [internal projects](../user/public_access.md#internal-projects-and-groups). To define the external providers list, use the full name of the provider, for example, `google_oauth2` for Google. For provider names, see the diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index ebd936424d3..8d4d8ff9f52 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -11,7 +11,7 @@ You can integrate your GitLab instance with [Salesforce](https://www.salesforce. ## Create a Salesforce Connected App To enable Salesforce OmniAuth provider, you must use Salesforce's credentials for your GitLab instance. -To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/s/articleView?id=connected_app_create.htm&type=5) on Salesforce. +To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/s/articleView?id=sf.connected_app_create.htm&type=5) on Salesforce. 1. Sign in to [Salesforce](https://login.salesforce.com/). diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 95bf835147d..c5383f9e34b 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -730,6 +730,11 @@ Refer to the documentation for your SAML Identity Provider for information on ho The [Generated passwords for users created through integrated authentication](../security/passwords_for_integrated_authentication_methods.md) guide provides an overview of how GitLab generates and sets passwords for users created via SAML. +## Link SAML identity for an existing user + +A user can manually link their SAML identity to an existing GitLab account by following the steps in +[Enable OmniAuth for an existing user](omniauth.md#enable-omniauth-for-an-existing-user). + ## Configuring Group SAML on a self-managed GitLab instance **(PREMIUM SELF)** For information on the GitLab.com implementation, please see the [SAML SSO for GitLab.com groups page](../user/group/saml_sso). @@ -785,7 +790,7 @@ documentation on how to use SAML to sign in to GitLab. Examples: - [ADFS (Active Directory Federation Services)](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) -- [Auth0](https://auth0.com/docs/configure/saml-configuration/configure-auth0-saml-identity-provider) +- [Auth0](https://auth0.com/docs/authenticate/protocols/saml/saml-sso-integrations/configure-auth0-saml-identity-provider) - [PingOne by Ping Identity](http://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html) GitLab provides the following setup notes for guidance only. diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index 0ad2b5ecf3b..5eb0624be31 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -86,26 +86,22 @@ The **Alert details** tab has two sections. The top section provides a short lis ### Metrics tab -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.2. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/340852) in GitLab 14.10. In GitLab 14.9 and earlier, this tab shows a metrics chart for alerts coming from Prometheus. -The **Metrics** tab displays a metrics chart for alerts coming from Prometheus. If the alert originated from any other tool, the **Metrics** tab is empty. -For externally-managed Prometheus instances, you must configure your alerting rules to display a chart in the alert. For information about how to configure -your alerting rules, see [Embedding metrics based on alerts in incident issues](../metrics/embed.md#embedding-metrics-based-on-alerts-in-incident-issues). See -[External Prometheus instances](../metrics/alerts.md#external-prometheus-instances) for information about setting up alerts for your self-managed Prometheus -instance. +In many cases, alerts are associated to metrics. You can upload screenshots of metric +charts in the **Metrics** tab. -Prerequisite: +To do so, either: -- You must have at least the Developer role. +- Select **upload** and then select an image from your file browser. +- Drag a file from your file browser and drop it in the drop zone. -To view the metrics for an alert: +When you upload an image, you can add text to the image and link it to the original graph. -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Monitor > Alerts**. -1. Select the alert you want to view. -1. Below the title of the alert, select the **Metrics** tab. + - +If you add a link, it is shown above the uploaded image. #### View an alert's logs diff --git a/doc/operations/incident_management/img/alert_detail_metrics_v13_2.png b/doc/operations/incident_management/img/alert_detail_metrics_v13_2.png Binary files differdeleted file mode 100644 index 84d83365ea8..00000000000 --- a/doc/operations/incident_management/img/alert_detail_metrics_v13_2.png +++ /dev/null diff --git a/doc/operations/incident_management/img/incident_list_v13_5.png b/doc/operations/incident_management/img/incident_list_v13_5.png Binary files differdeleted file mode 100644 index 88942a70e88..00000000000 --- a/doc/operations/incident_management/img/incident_list_v13_5.png +++ /dev/null diff --git a/doc/operations/incident_management/img/incident_metrics_tab_text_link_modal_v14_9.png b/doc/operations/incident_management/img/incident_metrics_tab_text_link_modal_v14_9.png Binary files differindex 1b045a13fc5..bae1e5ad1b4 100644 --- a/doc/operations/incident_management/img/incident_metrics_tab_text_link_modal_v14_9.png +++ b/doc/operations/incident_management/img/incident_metrics_tab_text_link_modal_v14_9.png diff --git a/doc/operations/incident_management/img/timeline_view_toggle_v13_5.png b/doc/operations/incident_management/img/timeline_view_toggle_v13_5.png Binary files differdeleted file mode 100644 index 542ca139f7e..00000000000 --- a/doc/operations/incident_management/img/timeline_view_toggle_v13_5.png +++ /dev/null diff --git a/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png b/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png Binary files differnew file mode 100644 index 00000000000..90f1d205fed --- /dev/null +++ b/doc/operations/incident_management/img/timeline_view_toggle_v14_10.png diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index c7ed386f505..66d04fce14c 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -169,7 +169,7 @@ Beneath the highlight bar, GitLab displays a summary that includes the following - Monitoring tool Comments are displayed in threads, but can be displayed chronologically -[in a timeline view](#timeline-view). +[by toggling on the recent updates view](#recent-updates-view). ### Metrics **(PREMIUM)** @@ -195,15 +195,15 @@ field populated.  -### Timeline view **(PREMIUM)** +### Recent updates view **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in GitLab 13.5. To quickly see the latest updates on an incident, click -**{comments}** **Turn timeline view on** in the comment bar to display comments +**{history}** **Turn recent updates view on** in the comment bar to display comments un-threaded and ordered chronologically, newest to oldest: - + ### Service Level Agreement countdown timer **(PREMIUM)** @@ -255,7 +255,13 @@ Add a to-do for incidents that you want to track in your to-do list. Click the ### Change incident status -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. + +FLAG: +This feature is available by default. To disable it per project or for your entire +instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) +named `incident_escalations`. For users with the Developer role or higher, select **Edit** in the **Status** section of the right-hand side bar of an incident, then select a status. **Triggered** is the default status for @@ -271,9 +277,15 @@ by changing the status. Setting the status to: For [incidents created from alerts](alerts.md#create-an-incident-from-an-alert), updating the incident status also updates the alert status. -## Change escalation policy **(PREMIUM)** +### Change escalation policy **(PREMIUM)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9. +FLAG: +This feature is available by default. To disable it per project or for your entire +instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) +named `incident_escalations`. For users with the Developer role or higher, select **Edit** in the **Escalation policy** section of the right-hand side bar of an incident, then select a policy. By default, new incidents do not have diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 27854a9e201..d0c14ab6e59 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -50,7 +50,13 @@ or stop alert escalations by [updating the alert's status](alerts.md#update-an-a ### Escalating an incident -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9 [with a flag](../../administration/feature_flags.md) named `incident_escalations`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/345769) in GitLab 14.10. + +FLAG: +This feature is available by default. To disable it per project or for your entire +instance, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) +named `incident_escalations`. For incidents, paging on-call responders is optional for each individual incident. To begin escalating the incident, [set the incident's escalation policy](incidents.md#change-escalation-policy). diff --git a/doc/operations/index.md b/doc/operations/index.md index 9b988ff561d..61d38c3d6a9 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -15,7 +15,7 @@ your applications. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 15.0. Metrics help you understand the health and performance of your infrastructure, applications, and systems by providing insights into your application's reliability, @@ -63,7 +63,7 @@ and the work required to fix them - all without leaving GitLab. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 15.0. Application tracing in GitLab is a way to measure an application's performance and health while it's running. After configuring your application to enable tracing, you @@ -83,7 +83,7 @@ microservices-based distributed systems - and displays results within GitLab. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 15.0. Developers need to troubleshoot application changes in development, and incident responders need aggregated, real-time logs when troubleshooting problems with diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index 702ff944fc5..0e446f04b9b 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -79,7 +79,7 @@ values extracted from the [`alerts` field in webhook payload](https://prometheus - `starts_at`: Alert start time from the payload's `startsAt` field - `full_query`: Alert query extracted from the payload's `generatorURL` field - Optional list of attached annotations extracted from `annotations/*` -- Alert [GFM](../../user/markdown.md): GitLab Flavored Markdown from the payload's `annotations/gitlab_incident_markdown` field. +- Alert [GLFM](../../user/markdown.md): GitLab Flavored Markdown from the payload's `annotations/gitlab_incident_markdown` field. - Alert Severity ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871) in GitLab version 13.9): Extracted from the alert payload field `labels/severity`. Maps case-insensitive value to [Alert's severity](../incident_management/alerts.md#alert-severity): diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md index 3e14917209a..5c7d6440a7e 100644 --- a/doc/operations/metrics/dashboards/default.md +++ b/doc/operations/metrics/dashboards/default.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab provides some dashboards out-of-the-box for any project with [Prometheus available](../../../user/project/integrations/prometheus.md). You can diff --git a/doc/operations/metrics/dashboards/develop.md b/doc/operations/metrics/dashboards/develop.md index fc7686c8f86..513e5532bee 100644 --- a/doc/operations/metrics/dashboards/develop.md +++ b/doc/operations/metrics/dashboards/develop.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab provides a template to make it easier for you to create templates for [custom dashboards](index.md). Templates provide helpful guidance and diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index a8ca23b7002..052f678f1b7 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. By default, all projects include a [GitLab-defined Prometheus dashboard](default.md), which includes a few key metrics, but you can also define your own custom dashboards. diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index 734b560bf13..a4cdfc5b9a0 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. The below panel types are supported in monitoring dashboards. diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md index 14da5cf4a04..aae5a623420 100644 --- a/doc/operations/metrics/dashboards/settings.md +++ b/doc/operations/metrics/dashboards/settings.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. You can configure your [Monitoring dashboard](../index.md) to display the time zone of your choice, and the links of your choice. diff --git a/doc/operations/metrics/dashboards/templating_variables.md b/doc/operations/metrics/dashboards/templating_variables.md index d751a96f379..7d1754f79f5 100644 --- a/doc/operations/metrics/dashboards/templating_variables.md +++ b/doc/operations/metrics/dashboards/templating_variables.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. Templating variables can be used to make your metrics dashboard more versatile. diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md index 369bcd1ddeb..b796cf4cef5 100644 --- a/doc/operations/metrics/dashboards/variables.md +++ b/doc/operations/metrics/dashboards/variables.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. ## Query variables diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index 81f1354d3c0..62389fa01ee 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. Dashboards have several components: diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md index 6a3712d8377..26ea7aa07eb 100644 --- a/doc/operations/metrics/embed.md +++ b/doc/operations/metrics/embed.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Embedding metric charts within GitLab Flavored Markdown **(FREE)** You can display metrics charts within -[GitLab Flavored Markdown](../../user/markdown.md#gitlab-flavored-markdown) +[GitLab Flavored Markdown (GLFM)](../../user/markdown.md) fields such as issue or merge request descriptions. The maximum number of embedded charts allowed in a GitLab Flavored Markdown field is 100. Embedding charts is useful when sharing an application incident or performance @@ -26,7 +26,7 @@ are still supported, and can be used to embed metric charts. To display metric charts, include a link of the form `https://<root_url>/<project>/-/metrics?environment=<environment_id>` in a field -that supports GitLab-flavored Markdown: +that supports GitLab Flavored Markdown: ```markdown ### Summary @@ -100,11 +100,11 @@ a chart corresponding to the query can be included if these requirements are met > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) from GitLab Ultimate to GitLab Free in 13.2. [Cluster Health Metrics](../../user/infrastructure/clusters/manage/clusters_health.md) -can also be embedded in [GitLab-flavored Markdown](../../user/markdown.md). +can also be embedded in [GitLab Flavored Markdown](../../user/markdown.md). To embed a metric chart, include a link to that chart in the form `https://<root_url>/<project>/-/cluster/<cluster_id>?<query_params>` anywhere that -GitLab-flavored Markdown is supported. To generate and copy a link to the chart, +GitLab Flavored Markdown is supported. To generate and copy a link to the chart, follow the instructions in the [Cluster Health Metric documentation](../../user/infrastructure/clusters/manage/clusters_health.md). diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index 044f6800e73..bbc05bb0f9a 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346540) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 15.0. Tracing provides insight into the performance and health of a deployed application, tracking each function or microservice that handles a given request. Tracing makes it easy to understand the diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index 8eee8fc9a6b..8b665972918 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -1,98 +1,11 @@ --- -stage: Manage -group: Workspace -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../user/public_access.md' +remove_date: '2022-06-18' --- -# Project and group visibility **(FREE)** +This document was moved to [another location](../user/public_access.md). -GitLab allows users with the Owner role to set a project's or group's visibility as: - -- **Public** -- **Internal** -- **Private** - -These visibility levels affect who can see the project in the public access directory (`/public` -for your GitLab instance). For example, <https://gitlab.com/public>. -You can control the visibility of individual features with -[project feature settings](../user/permissions.md#project-features). - -## Public projects and groups - -Public projects can be cloned **without any** authentication over HTTPS. - -They are listed in the public access directory (`/public`) for all users. - -**Any signed-in user** has the Guest role on the repository. - -NOTE: -By default, `/public` is visible to unauthenticated users. However, if the -[**Public** visibility level](../user/admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) -is restricted, `/public` is visible only to signed-in users. - -## Internal projects and groups - -Internal projects can be cloned by any signed-in user except -[external users](../user/permissions.md#external-users). - -They are also listed in the public access directory (`/public`), but only for signed-in users. - -Any signed-in users except [external users](../user/permissions.md#external-users) have the -Guest role on the repository. - -NOTE: -From July 2019, the `Internal` visibility setting is disabled for new projects, groups, -and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` -visibility setting keep this setting. You can read more about the change in the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12388). - -## Private projects and groups - -Private projects can only be cloned and viewed by project members (except for guests). - -They appear in the public access directory (`/public`) for project members only. - -## Change project visibility - -Prerequisite: - -- You must have the Owner role for a project. - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Settings > General**. -1. Expand **Visibility, project features, permissions**. -1. Change **Project visibility** to either **Private**, **Internal**, or **Public**. -1. Select **Save changes**. - -## Change group visibility - -Prerequisite: - -- You must have the Owner role for a group. - -1. On the top bar, select **Menu > Groups** and find your project. -1. On the left sidebar, select **Settings > General**. -1. Expand **Naming, visibility**. -1. Under **Visibility level** select either **Private**, **Internal**, or **Public**. -1. Select **Save changes**. - -## Restrict use of public or internal projects **(FREE SELF)** - -You can restrict the use of visibility levels for users when they create a project or a snippet. -This is useful to prevent users from publicly exposing their repositories by accident. The -restricted visibility settings do not apply to administrators. - -For details, see [Restricted visibility levels](../user/admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels). - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-06-18>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index cef52cc61d7..5458e9952fa 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -375,6 +375,30 @@ For example, for installations from source: sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 ``` +#### Incremental repository backups + +> - Introduced in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `incremental_repository_backup`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/355945) in GitLab 14.10. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `incremental_repository_backup`. +On GitLab.com, this feature is not available. + +Incremental backups can be faster than full backups because they only pack changes since the last backup into the backup +bundle for each repository. There must be an existing backup to create an incremental backup from and this backup will be overwritten. You can use the `BACKUP=timestamp_of_backup` option to choose which backup will be used. + +To create an incremental backup, run: + +```shell +sudo gitlab-backup create INCREMENTAL=yes +``` + +Incremental backups can also be created from [an untarred backup](#skipping-tar-creation) by using `SKIP=tar`: + +```shell +sudo gitlab-backup create INCREMENTAL=yes SKIP=tar +``` + #### Uploading backups to a remote (cloud) storage You can let the backup script upload (using the [Fog library](http://fog.io/)) @@ -700,6 +724,23 @@ sudo gitlab-backup create DIRECTORY=weekly Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. +#### Skip uploading backups to remote storage + +If you have configured GitLab to [upload backups in a remote storage](#uploading-backups-to-a-remote-cloud-storage), +you can use the `SKIP=remote` option to skip uploading your backups to the remote storage. + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup create SKIP=remote +``` + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=remote RAILS_ENV=production +``` + #### Uploading to locally mounted shares You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or @@ -1393,7 +1434,7 @@ To prepare the new server: 1. Flush the Redis database to disk, and stop GitLab other than the services needed for migration: ```shell - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket save && sudo gitlab-ctl stop && sudo gitlab-ctl start postgresql && sudo gitlab-ctl start gitaly ``` 1. Create a GitLab backup: @@ -1813,12 +1854,7 @@ If this happens, examine the following: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.2. > - [Deployed behind a feature flag](../user/feature_flags.md), enabled by default. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#disable-or-enable-gitaly-backup). - -There can be -[risks when disabling released features](../administration/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.10. [Feature flag `gitaly_backup`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83254) removed. The `gitaly-backup` binary is used by the backup Rake task to create and restore repository backups from Gitaly. `gitaly-backup` replaces the previous backup method that directly calls RPCs on Gitaly from GitLab. @@ -1835,41 +1871,3 @@ If you have a specific reason to change the path, it can be configured in Omnibu 1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect - -#### Disable or enable `gitaly-backup` - -`gitaly-backup` is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) -can opt to disable it. - -To disable it: - -```ruby -Feature.disable(:gitaly_backup) -``` - -To enable it: - -```ruby -Feature.enable(:gitaly_backup) -``` - -### Incremental repository backups - -> Introduced in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `incremental_repository_backup`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `incremental_repository_backup`. -On GitLab.com, this feature is not available. -This feature is not ready for production use. - -Incremental backups can be faster than full backups because they only pack changes since the last backup into the backup -bundle for each repository. Because incremental backups require access to the previous backup, you can't use incremental -backups with tar files. - -To create an incremental backup, run: - -```shell -sudo gitlab-backup create SKIP=tar INCREMENTAL=yes -``` diff --git a/doc/raketasks/migrate_snippets.md b/doc/raketasks/migrate_snippets.md index 0d2c4edb01f..c36009e69d0 100644 --- a/doc/raketasks/migrate_snippets.md +++ b/doc/raketasks/migrate_snippets.md @@ -18,7 +18,7 @@ For each snippet: - A file is created in the repository, using the snippet filename. - The snippet is committed to the repository. -GitLab performs this migration through a [Background Migration](../development/background_migrations.md) +GitLab performs this migration through a [Background Migration](../development/database/background_migrations.md) when the GitLab instance is upgraded to 13.0 or a higher version. However, if the migration fails for any of the snippets, they must be migrated individually. The following Rake tasks help with that process. diff --git a/doc/security/index.md b/doc/security/index.md index da3fa761f3f..73ac5028db5 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -26,6 +26,7 @@ type: index - [CI/CD variables](../ci/variables/index.md#cicd-variable-security) - [Token overview](token_overview.md) - [Project Import decompressed archive size limits](project_import_decompressed_archive_size_limits.md) +- [Responding to security incidents](responding_to_security_incidents.md) ## Securing your GitLab installation diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index 1940c5be73a..06934b187c1 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -74,11 +74,13 @@ If you know the username, user ID, or email address, you can use the Rails conso user = User.find_by(email: 'user@example.com') ``` -1. Reset the password: +1. Reset the password by setting a value for `user.password` and `user.password_confirmation`. For example, to set a new random + password: ```ruby - user.password = 'secret_pass' - user.password_confirmation = 'secret_pass' + new_password = ::User.random_password + user.password = new_password + user.password_confirmation = new_password ``` 1. Optional. Notify the user that an administrator changed their password: diff --git a/doc/security/responding_to_security_incidents.md b/doc/security/responding_to_security_incidents.md new file mode 100644 index 00000000000..b3bce785695 --- /dev/null +++ b/doc/security/responding_to_security_incidents.md @@ -0,0 +1,65 @@ +--- +stage: Manage +group: Authentication and Authorization +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, howto +--- + +# Responding to security incidents **(FREE SELF)** + +When a security incident occurs, you should follow the processes defined by your organization. However, you might consider some +additional steps. These suggestions are intended to supplement existing security incident response processes within your organization. + +## Suspected compromised user account + +If you suspect that a user account or bot account has been compromised, consider taking the following steps: + +- [Block the user](../user/admin_area/moderate_users.md#block-a-user) to mitigate any current risk. +- [Review the audit events](../administration/audit_events.md) available to you to identify any suspicious account behavior. For + example: + - Suspicious sign-in events. + - Creation or deletion of personal access tokens, project access tokens, and group access tokens. + - Creation or deletion of SSH or GPG keys. + - Creation, modification, or deletion of two-factor authentication. + - Changes to repositories. + - Changes to group or project configurations. + - Addition or modification of runners. + - Addition or modification of webhooks or Git hooks. +- Reset any credentials the user might have had access to. For example, users with at least the Maintainer role can view protected + [CI/CD variables](../ci/variables/index.md) and [runner registration tokens](token_overview.md#runner-registration-tokens). +- [Reset the user's password](reset_user_password.md). +- Get the user to [enable two factor authentication](../user/profile/account/two_factor_authentication.md) (2FA), and consider [enforcing 2FA at the instance or group level](two_factor_authentication.md) +- After completing an investigation and mitigating impacts, unblock the user. + +## Suspected compromised instance **(FREE SELF)** + +Self-managed GitLab customers and administrators are responsible for: + +- The security of their underlying hosts. +- Keeping GitLab itself up to date. + +It is important to [regularly update GitLab](../policy/maintenance.md), update your operating system and its software, and harden your +hosts in accordance with vendor guidance. + +If you suspect that your GitLab instance has been compromised, consider taking the following steps: + +- [Review the audit events](../administration/audit_events.md) available to you for suspicious account behavior. +- [Review all users](../user/admin_area/moderate_users.md) (including the Administrative root user), and follow the steps in [Suspected compromised user account](#suspected-compromised-user-account) if necessary. +- Review the [Credentials Inventory](../user/admin_area/credentials_inventory.md), if available to you. +- Change any sensitive credentials, variables, tokens, and secrets. For example, those located in instance configuration, database, + CI/CD pipelines, or elsewhere. +- Upgrade to the latest version of GitLab and adopt a plan to upgrade after every security patch release. + +In addition, the suggestions below are common steps taken in incident response plans when servers are compromised by malicious actors. + +WARNING: +Use these suggestions at your own risk. + +- Save any server state and logs to a write-once location, for later investigation. +- Look for unrecognized background processes. +- Check for open ports on the system. +- Rebuild the host from a known-good backup or from scratch, and apply all the latest security patches. +- Review network logs for uncommon traffic. +- Establish network monitoring and network-level controls. +- Restrict inbound and outbound network access to authorized users and servers only. +- Ensure all logs are routed to an independent write-only datastore. diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index 2e4a737f9aa..03bc0207cf5 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -39,7 +39,7 @@ Hovering over this icon tells you why the key is restricted. ## Default settings By default, the GitLab.com and self-managed settings for the -[supported key types](../ssh/index.md#supported-ssh-key-types) are: +[supported key types](../user/ssh.md#supported-ssh-key-types) are: - RSA SSH keys are allowed. - DSA SSH keys are forbidden ([since GitLab 11.0](https://about.gitlab.com/releases/2018/06/22/gitlab-11-0-released/#support-for-dsa-ssh-keys)). diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index cab9f6a957e..ae13881fe6f 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -114,7 +114,7 @@ FLAG: On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `two_factor_for_cli`. On GitLab.com, this feature is not available. The feature is not ready for production use. This feature flag also affects [session duration for Git Operations when 2FA is enabled](../user/admin_area/settings/account_and_limit_settings.md#customize-session-duration-for-git-operations-when-2fa-is-enabled). Two-factor authentication can be enforced for Git over SSH operations. However, we recommend using -[ED25519_SK](../ssh/index.md#ed25519_sk-ssh-keys) or [ECDSA_SK](../ssh/index.md#ecdsa_sk-ssh-keys) SSH keys instead. +[ED25519_SK](../user/ssh.md#ed25519_sk-ssh-keys) or [ECDSA_SK](../user/ssh.md#ecdsa_sk-ssh-keys) SSH keys instead. The one-time password (OTP) verification can be done using a command: diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index e8b0c08e240..dcdd18a9f0b 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -7,8 +7,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # User File Uploads **(FREE)** -> - In GitLab 14.8 and later, [authorization checks are enforced](https://gitlab.com/gitlab-org/gitlab/-/issues/26781) on media uploads. This change is being [rolled out incrementally](https://gitlab.com/gitlab-org/gitlab/-/issues/352291) on GitLab.com in 14.9. - Images that are attached to issues, merge requests, or comments do not require authentication to be viewed if they are accessed directly by URL. This direct URL contains a random 32-character ID that prevents unauthorized diff --git a/doc/ssh/index.md b/doc/ssh/index.md index 846e5c369bb..10184a63a7a 100644 --- a/doc/ssh/index.md +++ b/doc/ssh/index.md @@ -1,497 +1,11 @@ --- -stage: Manage -group: Authentication and Authorization -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: howto, reference +redirect_to: '../user/ssh.md' +remove_date: '2022-06-18' --- -# Use SSH keys to communicate with GitLab **(FREE)** +This document was moved to [another location](../user/ssh.md). -Git is a distributed version control system, which means you can work locally, -then share or *push* your changes to a server. In this case, the server you push to is GitLab. - -GitLab uses the SSH protocol to securely communicate with Git. -When you use SSH keys to authenticate to the GitLab remote server, -you don't need to supply your username and password each time. - -## Prerequisites - -To use SSH to communicate with GitLab, you need: - -- The OpenSSH client, which comes pre-installed on GNU/Linux, macOS, and Windows 10. -- SSH version 6.5 or later. Earlier versions used an MD5 signature, which is not secure. - -To view the version of SSH installed on your system, run `ssh -V`. - -## Supported SSH key types - -To communicate with GitLab, you can use the following SSH key types: - -- [ED25519](#ed25519-ssh-keys) -- [ED25519_SK](#ed25519_sk-ssh-keys) (Available in GitLab 14.8 and later.) -- [ECDSA_SK](#ecdsa_sk-ssh-keys) (Available in GitLab 14.8 and later.) -- [RSA](#rsa-ssh-keys) -- DSA ([Deprecated](https://about.gitlab.com/releases/2018/06/22/gitlab-11-0-released/#support-for-dsa-ssh-keys) in GitLab 11.0.) -- ECDSA (As noted in [Practical Cryptography With Go](https://leanpub.com/gocrypto/read#leanpub-auto-ecdsa), the security issues related to DSA also apply to ECDSA.) - -Administrators can [restrict which keys are permitted and their minimum lengths](../security/ssh_keys_restrictions.md). - -### ED25519 SSH keys - -The book [Practical Cryptography With Go](https://leanpub.com/gocrypto/read#leanpub-auto-chapter-5-digital-signatures) -suggests that [ED25519](https://ed25519.cr.yp.to/) keys are more secure and performant than RSA keys. - -OpenSSH 6.5 introduced ED25519 SSH keys in 2014 and they should be available on most -operating systems. - -### ED25519_SK SSH keys - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. - -To use ED25519_SK SSH keys on GitLab, your local client and GitLab server -must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. - -### ECDSA_SK SSH keys - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. - -To use ECDSA_SK SSH keys on GitLab, your local client and GitLab server -must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. - -### RSA SSH keys - -Available documentation suggests that ED25519 is more secure than RSA. - -If you use an RSA key, the US National Institute of Science and Technology in -[Publication 800-57 Part 3 (PDF)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-57Pt3r1.pdf) -recommends a key size of at least 2048 bits. The default key size depends on your version of `ssh-keygen`. -Review the `man` page for your installed `ssh-keygen` command for details. - -## See if you have an existing SSH key pair - -Before you create a key pair, see if a key pair already exists. - -1. On Windows, Linux, or macOS, go to your home directory. -1. Go to the `.ssh/` subdirectory. If the `.ssh/` subdirectory doesn't exist, - you are either not in the home directory, or you haven't used `ssh` before. - In the latter case, you need to [generate an SSH key pair](#generate-an-ssh-key-pair). -1. See if a file with one of the following formats exists: - - | Algorithm | Public key | Private key | - | --------- | ---------- | ----------- | - | ED25519 (preferred) | `id_ed25519.pub` | `id_ed25519` | - | ED25519_SK | `id_ed25519_sk.pub` | `id_ed25519_sk` | - | ECDSA_SK | `id_ecdsa_sk.pub` | `id_ecdsa_sk` | - | RSA (at least 2048-bit key size) | `id_rsa.pub` | `id_rsa` | - | DSA (deprecated) | `id_dsa.pub` | `id_dsa` | - | ECDSA | `id_ecdsa.pub` | `id_ecdsa` | - -## Generate an SSH key pair - -If you do not have an existing SSH key pair, generate a new one. - -1. Open a terminal. -1. Type `ssh-keygen -t` followed by the key type and an optional comment. - This comment is included in the `.pub` file that's created. - You may want to use an email address for the comment. - - For example, for ED25519: - - ```shell - ssh-keygen -t ed25519 -C "<comment>" - ``` - - For 2048-bit RSA: - - ```shell - ssh-keygen -t rsa -b 2048 -C "<comment>" - ``` - -1. Press Enter. Output similar to the following is displayed: - - ```plaintext - Generating public/private ed25519 key pair. - Enter file in which to save the key (/home/user/.ssh/id_ed25519): - ``` - -1. Accept the suggested filename and directory, unless you are generating a [deploy key](../user/project/deploy_keys/index.md) - or want to save in a specific directory where you store other keys. - - You can also dedicate the SSH key pair to a [specific host](#configure-ssh-to-point-to-a-different-directory). - -1. Specify a [passphrase](https://www.ssh.com/academy/ssh/passphrase): - - ```plaintext - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - ``` - -1. A confirmation is displayed, including information about where your files are stored. - -A public and private key are generated. -[Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account) and keep -the private key secure. - -### Configure SSH to point to a different directory - -If you did not save your SSH key pair in the default directory, -configure your SSH client to point to the directory where the private key is stored. - -1. Open a terminal and run this command: - - ```shell - eval $(ssh-agent -s) - ssh-add <directory to private SSH key> - ``` - -1. Save these settings in the `~/.ssh/config` file. For example: - - ```conf - # GitLab.com - Host gitlab.com - PreferredAuthentications publickey - IdentityFile ~/.ssh/gitlab_com_rsa - - # Private GitLab instance - Host gitlab.company.com - PreferredAuthentications publickey - IdentityFile ~/.ssh/example_com_rsa - ``` - - For more information on these settings, see the [`man ssh_config`](https://man.openbsd.org/ssh_config) page in the SSH configuration manual. - -Public SSH keys must be unique to GitLab because they bind to your account. -Your SSH key is the only identifier you have when you push code with SSH. -It must uniquely map to a single user. - -### Update your SSH key passphrase - -You can update the passphrase for your SSH key. - -1. Open a terminal and run this command: - - ```shell - ssh-keygen -p -f /path/to/ssh_key - ``` - -1. At the prompts, type the passphrase and press Enter. - -### Upgrade your RSA key pair to a more secure format - -If your version of OpenSSH is between 6.5 and 7.8, -you can save your private RSA SSH keys in a more secure -OpenSSH format. - -1. Open a terminal and run this command: - - ```shell - ssh-keygen -o -f ~/.ssh/id_rsa - ``` - - Alternatively, you can generate a new RSA key with the more secure encryption format with - the following command: - - ```shell - ssh-keygen -o -t rsa -b 4096 -C "<comment>" - ``` - -## Generate an SSH key pair for a FIDO/U2F hardware security key - -To generate ED25519_SK or ECDSA_SK SSH keys, you must use OpenSSH 8.2 or later. - -1. Insert a hardware security key into your computer. -1. Open a terminal. -1. Type `ssh-keygen -t` followed by the key type and an optional comment. - This comment is included in the `.pub` file that's created. - You may want to use an email address for the comment. - - For example, for ED25519_SK: - - ```shell - ssh-keygen -t ed25519-sk -C "<comment>" - ``` - - For ECDSA_SK: - - ```shell - ssh-keygen -t ecdsa-sk -C "<comment>" - ``` - - If your security key supports FIDO2 resident keys, you can enable this when - creating your SSH key: - - ```shell - ssh-keygen -t ed25519-sk -O resident -C "<comment>" - ``` - - `-O resident` indicates that the key should be stored on the FIDO authenticator itself. - Resident key is easier to import to a new computer because it can be loaded directly - from the security key by [`ssh-add -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-add.1#K) - or [`ssh-keygen -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-keygen#K). - -1. Select Enter. Output similar to the following is displayed: - - ```plaintext - Generating public/private ed25519-sk key pair. - You may need to touch your authenticator to authorize key generation. - ``` - -1. Touch the button on the hardware security key. - -1. Accept the suggested filename and directory: - - ```plaintext - Enter file in which to save the key (/home/user/.ssh/id_ed25519_sk): - ``` - -1. Specify a [passphrase](https://www.ssh.com/academy/ssh/passphrase): - - ```plaintext - Enter passphrase (empty for no passphrase): - Enter same passphrase again: - ``` - -1. A confirmation is displayed, including information about where your files are stored. - -A public and private key are generated. -[Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account). - -## Add an SSH key to your GitLab account - -To use SSH with GitLab, copy your public key to your GitLab account. - -1. Copy the contents of your public key file. You can do this manually or use a script. - For example, to copy an ED25519 key to the clipboard: - - **macOS:** - - ```shell - tr -d '\n' < ~/.ssh/id_ed25519.pub | pbcopy - ``` - - **Linux** (requires the `xclip` package): - - ```shell - xclip -sel clip < ~/.ssh/id_ed25519.pub - ``` - - **Git Bash on Windows:** - - ```shell - cat ~/.ssh/id_ed25519.pub | clip - ``` - - Replace `id_ed25519.pub` with your filename. For example, use `id_rsa.pub` for RSA. - -1. Sign in to GitLab. -1. On the top bar, in the top right corner, select your avatar. -1. Select **Preferences**. -1. On the left sidebar, select **SSH Keys**. -1. In the **Key** box, paste the contents of your public key. - If you manually copied the key, make sure you copy the entire key, - which starts with `ssh-rsa`, `ssh-dss`, `ecdsa-sha2-nistp256`, `ecdsa-sha2-nistp384`, `ecdsa-sha2-nistp521`, - `ssh-ed25519`, `sk-ecdsa-sha2-nistp256@openssh.com`, or `sk-ssh-ed25519@openssh.com`, and may end with a comment. -1. In the **Title** box, type a description, like `Work Laptop` or - `Home Workstation`. -1. Optional. In the **Expires at** box, select an expiration date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36243) in GitLab 12.9.) - In: - - GitLab 13.12 and earlier, the expiration date is informational only. It doesn't prevent - you from using the key. Administrators can view expiration dates and use them for - guidance when [deleting keys](../user/admin_area/credentials_inventory.md#delete-a-users-ssh-key). - - GitLab 14.0 and later, the expiration date is enforced. Administrators can - [allow expired keys to be used](../user/admin_area/settings/account_and_limit_settings.md#allow-expired-ssh-keys-to-be-used-deprecated). - - GitLab checks all SSH keys at 02:00 AM UTC every day. It emails an expiration notice for all SSH keys that expire on the current date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) - - GitLab checks all SSH keys at 01:00 AM UTC every day. It emails an expiration notice for all SSH keys that are scheduled to expire seven days from now. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) -1. Select **Add key**. - -## Verify that you can connect - -Verify that your SSH key was added correctly. - -1. For GitLab.com, to ensure you're connecting to the correct server, confirm the - [SSH host keys fingerprints](../user/gitlab_com/index.md#ssh-host-keys-fingerprints). -1. Open a terminal and run this command, replacing `gitlab.example.com` with your GitLab instance URL: - - ```shell - ssh -T git@gitlab.example.com - ``` - -1. If this is the first time you connect, you should verify the - authenticity of the GitLab host. If you see a message like: - - ```plaintext - The authenticity of host 'gitlab.example.com (35.231.145.151)' can't be established. - ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw. - Are you sure you want to continue connecting (yes/no)? yes - Warning: Permanently added 'gitlab.example.com' (ECDSA) to the list of known hosts. - ``` - - Type `yes` and press Enter. - -1. Run the `ssh -T git@gitlab.example.com` command again. You should receive a _Welcome to GitLab, `@username`!_ message. - -If the welcome message doesn't appear, you can troubleshoot by running `ssh` -in verbose mode: - -```shell -ssh -Tvvv git@gitlab.example.com -``` - -## Use different keys for different repositories - -You can use a different key for each repository. - -Open a terminal and run this command: - -```shell -git config core.sshCommand "ssh -o IdentitiesOnly=yes -i ~/.ssh/private-key-filename-for-this-repository -F /dev/null" -``` - -This command does not use the SSH Agent and requires Git 2.10 or later. For more information -on `ssh` command options, see the `man` pages for both `ssh` and `ssh_config`. - -## Use different accounts on a single GitLab instance - -You can use multiple accounts to connect to a single instance of GitLab. -You can do this by using the command in the [previous topic](#use-different-keys-for-different-repositories). -However, even if you set `IdentitiesOnly` to `yes`, you cannot sign in if an `IdentityFile` exists -outside of a `Host` block. - -Instead, you can assign aliases to hosts in the `~.ssh/config` file. - -- For the `Host`, use an alias like `user_1.gitlab.com` and - `user_2.gitlab.com`. Advanced configurations - are more difficult to maintain, and these strings are easier to - understand when you use tools like `git remote`. -- For the `IdentityFile`, use the path the private key. - -```conf -# User1 Account Identity -Host <user_1.gitlab.com> - Hostname gitlab.com - PreferredAuthentications publickey - IdentityFile ~/.ssh/<example_ssh_key1> - -# User2 Account Identity -Host <user_2.gitlab.com> - Hostname gitlab.com - PreferredAuthentications publickey - IdentityFile ~/.ssh/<example_ssh_key2> -``` - -Now, to clone a repository for `user_1`, use `user_1.gitlab.com` in the `git clone` command: - -```shell -git clone git@<user_1.gitlab.com>:gitlab-org/gitlab.git -``` - -To update a previously-cloned repository that is aliased as `origin`: - -```shell -git remote set-url origin git@<user_1.gitlab.com>:gitlab-org/gitlab.git -``` - -NOTE: -Private and public keys contain sensitive data. Ensure the permissions -on the files make them readable to you but not accessible to others. - -## Configure two-factor authentication (2FA) - -You can set up two-factor authentication (2FA) for -[Git over SSH](../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations). We recommend using -[ED25519_SK](#ed25519_sk-ssh-keys) or [ECDSA_SK](#ecdsa_sk-ssh-keys) SSH keys. - -## Use EGit on Eclipse - -If you are using [EGit](https://www.eclipse.org/egit/), you can [add your SSH key to Eclipse](https://wiki.eclipse.org/EGit/User_Guide#Eclipse_SSH_Configuration). - -## Use SSH on Microsoft Windows - -If you're running Windows 10, you can either use the [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install) -with [WSL 2](https://docs.microsoft.com/en-us/windows/wsl/install#update-to-wsl-2) which -has both `git` and `ssh` preinstalled, or install [Git for Windows](https://gitforwindows.org) to -use SSH through Powershell. - -The SSH key generated in WSL is not directly available for Git for Windows, and vice versa, -as both have a different home directory: - -- WSL: `/home/<user>` -- Git for Windows: `C:\Users\<user>` - -You can either copy over the `.ssh/` directory to use the same key, or generate a key in each environment. - -Alternative tools include: - -- [Cygwin](https://www.cygwin.com) -- [PuttyGen](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) - -## Overriding SSH settings on the GitLab server - -GitLab integrates with the system-installed SSH daemon and designates a user -(typically named `git`) through which all access requests are handled. Users -who connect to the GitLab server over SSH are identified by their SSH key instead -of their username. - -SSH *client* operations performed on the GitLab server are executed as this -user. You can modify this SSH configuration. For example, you can specify -a private SSH key for this user to use for authentication requests. However, this practice -is **not supported** and is strongly discouraged as it presents significant -security risks. - -GitLab checks for this condition, and directs you -to this section if your server is configured this way. For example: - -```shell -$ gitlab-rake gitlab:check - -Git user has default SSH configuration? ... no - Try fixing it: - mkdir ~/gitlab-check-backup-1504540051 - sudo mv /var/lib/git/.ssh/id_rsa ~/gitlab-check-backup-1504540051 - sudo mv /var/lib/git/.ssh/id_rsa.pub ~/gitlab-check-backup-1504540051 - For more information see: - [Overriding SSH settings on the GitLab server](#overriding-ssh-settings-on-the-gitlab-server) - Please fix the error above and rerun the checks. -``` - -Remove the custom configuration as soon as you can. These customizations -are **explicitly not supported** and may stop working at any time. - -## Troubleshooting - -### Password prompt with `git clone` - -When you run `git clone`, you may be prompted for a password, like `git@gitlab.example.com's password:`. -This indicates that something is wrong with your SSH setup. - -- Ensure that you generated your SSH key pair correctly and added the public SSH - key to your GitLab profile. -- Try to manually register your private SSH key by using `ssh-agent`. -- Try to debug the connection by running `ssh -Tv git@example.com`. - Replace `example.com` with your GitLab URL. - -### `Could not resolve hostname` error - -You may receive the following error when [verifying that you can connect](#verify-that-you-can-connect): - -```shell -ssh: Could not resolve hostname gitlab.example.com: nodename nor servname provided, or not known -``` - -If you receive this error, restart your terminal and try the command again. - -### `Key enrollment failed: invalid format` error - -You may receive the following error when [generating an SSH key pair for a FIDO/U2F hardware security key](#generate-an-ssh-key-pair-for-a-fidou2f-hardware-security-key): - -```shell -Key enrollment failed: invalid format -``` - -You can troubleshoot this by trying the following: - -- Run the `ssh-keygen` command using `sudo`. -- Verify your IDO/U2F hardware security key supports - the key type provided. -- Verify the version of OpenSSH is 8.2 or greater by - running `ssh -v`. +<!-- This redirect file can be deleted after <2022-06-18>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 7ca3a3ffade..17eafb7633e 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -21,7 +21,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Code owners](../user/project/code_owners.md) - Description templates: - [Setting a default template for merge requests and issues](../user/project/description_templates.md#set-a-default-template-for-merge-requests-and-issues) -- [Email from GitLab](../tools/email.md) +- [Email from GitLab](../user/admin_area/email_from_gitlab.md) - Groups: - [Creating group memberships via CN](../user/group/index.md#create-group-links-via-cn) - [Group push rules](../user/group/index.md#group-push-rules) diff --git a/doc/subscriptions/img/support-diagram.png b/doc/subscriptions/img/support-diagram.png Binary files differdeleted file mode 100644 index 1628a62f19a..00000000000 --- a/doc/subscriptions/img/support-diagram.png +++ /dev/null diff --git a/doc/subscriptions/img/support_diagram_c.png b/doc/subscriptions/img/support_diagram_c.png Binary files differnew file mode 100644 index 00000000000..a2fed80e912 --- /dev/null +++ b/doc/subscriptions/img/support_diagram_c.png diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index 6c5543edb58..824fe0bacde 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -34,8 +34,8 @@ GitLab SaaS or GitLab self-managed: your own GitLab instance. On a GitLab self-managed instance, a GitLab subscription provides the same set of -features for _all_ users. On GitLab SaaS, you can apply a subscription to either -a group or a personal namespace. +features for _all_ users. On GitLab SaaS, you can apply a subscription to a group +namespace. You cannot apply a subscription to a personal namespace. NOTE: Subscriptions cannot be transferred between GitLab SaaS and GitLab self-managed. @@ -256,12 +256,11 @@ Send all questions and requests related to the GitLab for Startups program to `s ### Support for Community Programs -Because these Community Programs are free of cost, regular Priority Support is not included. However, it can be purchased at a 95% discount in some cases. -If interested, email the relevant community program team: `education@gitlab.com`, `opensource@gitlab.com`, or `startups@gitlab.com`. +Because these Community Programs are free of cost, regular Priority Support is not included. As a community member, you can follow this diagram to find support: - + ## Contact Support diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index d38b56bb1f8..6765ca19518 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -126,31 +126,6 @@ Cloud licensing manages licenses for self-managed GitLab subscription plans. Clo - Activation: Unlock plan features and activate your self-managed instance by using an activation code. - License sync: Sync subscription data between your self-managed instance and GitLab. -### What cloud licensing includes - -#### Auto-renewals - -For renewals that occur on or after 2021-08-01, your subscriptions will auto-renew. -You have the option to manually cancel in the Customers Portal any time until thirty (30) days before renewal. - -#### Cloud licensing - -You can activate and manage your GitLab licenses by using the Customers Portal. -This feature was formerly known as Seat Link. - -#### Operational data - -Service data helps GitLab improve the product experience and provide proactive support. -Most data is categorized as optional and can be disabled. Data that is categorized as -operational, like number of issues, pipelines, merge requests, and version, is not configurable. - -Please see our [service usage privacy page](https://about.gitlab.com/handbook/legal/privacy/services-usage-data/) -for details on what information is collected. - -#### Quarterly subscription reconciliation - -See the [quarterly subscription reconciliation section](../quarterly_reconciliation.md) for more information. - ### How cloud licensing works #### Add your license @@ -173,6 +148,9 @@ This sync job runs daily around 3AM UTC. If the job fails, it is retried up to 1 The daily job provides **only** the following information to the Customers Portal: +- Company name +- Licensee name +- Licensee email - Date - Timestamp - License key @@ -268,22 +246,22 @@ instance, ensure you're purchasing enough seats to If you are an administrator, you can view the status of your subscription: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **License**. +1. On the left sidebar, select **Subscription**. -The **License** page includes the following details: +The **Subscription** page includes the following details: - Licensee - Plan - When it was uploaded, started, and when it expires -It also displays the following important statistics: +It also displays the following information: | Field | Description | |:-------------------|:------------| | Users in License | The number of users you've paid for in the current license loaded on the system. The number does not change unless you [add seats](#add-seats-to-a-subscription) during your current subscription period. | | Billable users | The daily count of billable users on your system. The count may change as you block or add users to your instance. | | Maximum users | The highest number of billable users on your system during the term of the loaded license. | -| Users over license | Calculated as `Maximum users` - `Users in License` for the current license term. This number incurs a retroactive charge that needs to be paid for at renewal. | +| Users over license | Calculated as `Maximum users` - `Users in License` for the current license term. This number incurs a retroactive charge that must be paid before renewal. | ## Export your license usage @@ -295,7 +273,7 @@ If you are an administrator, you can export your license usage into a CSV: 1. On the left sidebar, select **Subscription**. 1. In the top right, select **Export license usage file**. -This file contains all the information GitLab needs to manually process quarterly reconciliations or renewals. If your instance is firewalled or air-gapped, you can provide GitLab with this information. +This file contains the information GitLab uses to manually process quarterly reconciliations or renewals. If your instance is firewalled or air-gapped, you must provide GitLab with this information. The **License Usage** CSV includes the following details: diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 71d7e7f1426..c30e2703a29 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,771 +1,11 @@ --- -stage: Ecosystem -group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../administration/system_hooks.md' +remove_date: '2022-06-18' --- -# System hooks **(FREE SELF)** +This document was moved to [another location](../administration/system_hooks.md). -Your GitLab instance can perform HTTP POST requests on the following events: - -- `group_create` -- `group_destroy` -- `group_rename` -- `key_create` -- `key_destroy` -- `project_create` -- `project_destroy` -- `project_rename` -- `project_transfer` -- `project_update` -- `repository_update` -- `user_add_to_group` -- `user_add_to_team` -- `user_create` -- `user_destroy` -- `user_failed_login` -- `user_remove_from_group` -- `user_remove_from_team` -- `user_rename` -- `user_update_for_group` -- `user_update_for_team` - -The triggers for most of these are self-explanatory, but `project_update` and -`project_rename` deserve some clarification: `project_update` is fired any time -an attribute of a project is changed (including name, description, and tags) -_unless_ the `path` attribute is also changed. In that case, a `project_rename` -is triggered instead (so that, for instance, if all you care about is the -repository URL, you can just listen for `project_rename`). - -`user_failed_login` is sent whenever a _blocked_ user attempts to sign in and is -denied access. - -System hooks can be used, for example, for logging or changing information in an -LDAP server. - -In addition to these default events, you can enable triggers for other events, -such as push events, and disable the `repository_update` event -when you create a system hook. - -NOTE: -We follow the same structure and deprecations as [Webhooks](../user/project/integrations/webhooks.md) -for Push and Tag events, but we never display commits. - -## Create a system hook - -To create a system hook: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **System Hooks**. -1. Provide the **URL** and **Secret Token**. -1. Select the checkbox next to each optional **Trigger** you want to enable. -1. Select **Enable SSL verification**, if desired. -1. Click **Add system hook**. - -## Hooks request example - -**Request header**: - -```plaintext -X-Gitlab-Event: System Hook -``` - -**Project created:** - -```json -{ - "created_at": "2012-07-21T07:30:54Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "project_create", - "name": "StoreCloud", - "owner_email": "johnsmith@example.com", - "owner_name": "John Smith", - "owners": [{ - "name": "John", - "email": "user1@example.com" - }], - "path": "storecloud", - "path_with_namespace": "jsmith/storecloud", - "project_id": 74, - "project_visibility": "private" -} -``` - -**Project destroyed:** - -```json -{ - "created_at": "2012-07-21T07:30:58Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "project_destroy", - "name": "Underscore", - "owner_email": "johnsmith@example.com", - "owner_name": "John Smith", - "owners": [{ - "name": "John", - "email": "user1@example.com" - }], - "path": "underscore", - "path_with_namespace": "jsmith/underscore", - "project_id": 73, - "project_visibility": "internal" -} -``` - -**Project renamed:** - -```json -{ - "created_at": "2012-07-21T07:30:58Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "project_rename", - "name": "Underscore", - "path": "underscore", - "path_with_namespace": "jsmith/underscore", - "project_id": 73, - "owner_name": "John Smith", - "owner_email": "johnsmith@example.com", - "owners": [{ - "name": "John", - "email": "user1@example.com" - }], - "project_visibility": "internal", - "old_path_with_namespace": "jsmith/overscore" -} -``` - -Note that `project_rename` is not triggered if the namespace changes. -Please refer to `group_rename` and `user_rename` for that case. - -**Project transferred:** - -```json -{ - "created_at": "2012-07-21T07:30:58Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "project_transfer", - "name": "Underscore", - "path": "underscore", - "path_with_namespace": "scores/underscore", - "project_id": 73, - "owner_name": "John Smith", - "owner_email": "johnsmith@example.com", - "owners": [{ - "name": "John", - "email": "user1@example.com" - }], - "project_visibility": "internal", - "old_path_with_namespace": "jsmith/overscore" -} -``` - -**Project updated:** - -```json -{ - "created_at": "2012-07-21T07:30:54Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "project_update", - "name": "StoreCloud", - "owner_email": "johnsmith@example.com", - "owner_name": "John Smith", - "owners": [{ - "name": "John", - "email": "user1@example.com" - }], - "path": "storecloud", - "path_with_namespace": "jsmith/storecloud", - "project_id": 74, - "project_visibility": "private" -} -``` - -**New Team Member:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_add_to_team", - "access_level": "Maintainer", - "project_id": 74, - "project_name": "StoreCloud", - "project_path": "storecloud", - "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41, - "project_visibility": "private" -} -``` - -**Team Member Removed:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_remove_from_team", - "access_level": "Maintainer", - "project_id": 74, - "project_name": "StoreCloud", - "project_path": "storecloud", - "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41, - "project_visibility": "private" -} -``` - -**Team Member Updated:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_update_for_team", - "access_level": "Maintainer", - "project_id": 74, - "project_name": "StoreCloud", - "project_path": "storecloud", - "project_path_with_namespace": "jsmith/storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41, - "project_visibility": "private" -} -``` - -**User created:** - -```json -{ - "created_at": "2012-07-21T07:44:07Z", - "updated_at": "2012-07-21T07:38:22Z", - "email": "js@gitlabhq.com", - "event_name": "user_create", - "name": "John Smith", - "username": "js", - "user_id": 41 -} -``` - -**User removed:** - -```json -{ - "created_at": "2012-07-21T07:44:07Z", - "updated_at": "2012-07-21T07:38:22Z", - "email": "js@gitlabhq.com", - "event_name": "user_destroy", - "name": "John Smith", - "username": "js", - "user_id": 41 -} -``` - -**User failed login:** - -```json -{ - "event_name": "user_failed_login", - "created_at": "2017-10-03T06:08:48Z", - "updated_at": "2018-01-15T04:52:06Z", - "name": "John Smith", - "email": "user4@example.com", - "user_id": 26, - "username": "user4", - "state": "blocked" -} -``` - -If the user is blocked via LDAP, `state` is `ldap_blocked`. - -**User renamed:** - -```json -{ - "event_name": "user_rename", - "created_at": "2017-11-01T11:21:04Z", - "updated_at": "2017-11-01T14:04:47Z", - "name": "new-name", - "email": "best-email@example.tld", - "user_id": 58, - "username": "new-exciting-name", - "old_username": "old-boring-name" -} -``` - -**Key added** - -```json -{ - "event_name": "key_create", - "created_at": "2014-08-18 18:45:16 UTC", - "updated_at": "2012-07-21T07:38:22Z", - "username": "root", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", - "id": 4 -} -``` - -**Key removed** - -```json -{ - "event_name": "key_destroy", - "created_at": "2014-08-18 18:45:16 UTC", - "updated_at": "2012-07-21T07:38:22Z", - "username": "root", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", - "id": 4 -} -``` - -**Group created:** - -```json -{ - "created_at": "2012-07-21T07:30:54Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "group_create", - "name": "StoreCloud", - "path": "storecloud", - "group_id": 78 -} -``` - -**Group removed:** - -```json -{ - "created_at": "2012-07-21T07:30:54Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "group_destroy", - "name": "StoreCloud", - "path": "storecloud", - "group_id": 78 -} -``` - -**Group renamed:** - -```json -{ - "event_name": "group_rename", - "created_at": "2017-10-30T15:09:00Z", - "updated_at": "2017-11-01T10:23:52Z", - "name": "Better Name", - "path": "better-name", - "full_path": "parent-group/better-name", - "group_id": 64, - "old_path": "old-name", - "old_full_path": "parent-group/old-name" -} -``` - -**New Group Member:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_add_to_group", - "group_access": "Maintainer", - "group_id": 78, - "group_name": "StoreCloud", - "group_path": "storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41 -} -``` - -**Group Member Removed:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_remove_from_group", - "group_access": "Maintainer", - "group_id": 78, - "group_name": "StoreCloud", - "group_path": "storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41 -} -``` - -**Group Member Updated:** - -```json -{ - "created_at": "2012-07-21T07:30:56Z", - "updated_at": "2012-07-21T07:38:22Z", - "event_name": "user_update_for_group", - "group_access": "Maintainer", - "group_id": 78, - "group_name": "StoreCloud", - "group_path": "storecloud", - "user_email": "johnsmith@example.com", - "user_name": "John Smith", - "user_username": "johnsmith", - "user_id": 41 -} -``` - -## Push events - -Triggered when you push to the repository, except when pushing tags. -It generates one event per modified branch. - -**Request header**: - -```plaintext -X-Gitlab-Event: System Hook -``` - -**Request body:** - -```json -{ - "event_name": "push", - "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", - "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "ref": "refs/heads/master", - "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "user_id": 4, - "user_name": "John Smith", - "user_email": "john@example.com", - "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", - "project_id": 15, - "project":{ - "name":"Diaspora", - "description":"", - "web_url":"http://example.com/mike/diaspora", - "avatar_url":null, - "git_ssh_url":"git@example.com:mike/diaspora.git", - "git_http_url":"http://example.com/mike/diaspora.git", - "namespace":"Mike", - "visibility_level":0, - "path_with_namespace":"mike/diaspora", - "default_branch":"master", - "homepage":"http://example.com/mike/diaspora", - "url":"git@example.com:mike/diaspora.git", - "ssh_url":"git@example.com:mike/diaspora.git", - "http_url":"http://example.com/mike/diaspora.git" - }, - "repository":{ - "name": "Diaspora", - "url": "git@example.com:mike/diaspora.git", - "description": "", - "homepage": "http://example.com/mike/diaspora", - "git_http_url":"http://example.com/mike/diaspora.git", - "git_ssh_url":"git@example.com:mike/diaspora.git", - "visibility_level":0 - }, - "commits": [ - { - "id": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", - "message": "Add simple search to projects in public area", - "timestamp": "2013-05-13T18:18:08+00:00", - "url": "https://dev.gitlab.org/gitlab/gitlabhq/commit/c5feabde2d8cd023215af4d2ceeb7a64839fc428", - "author": { - "name": "Example User", - "email": "user@example.com" - } - } - ], - "total_commits_count": 1 -} -``` - -## Tag events - -Triggered when you create (or delete) tags to the repository. -It generates one event per modified tag. - -**Request header**: - -```plaintext -X-Gitlab-Event: System Hook -``` - -**Request body:** - -```json -{ - "event_name": "tag_push", - "before": "0000000000000000000000000000000000000000", - "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", - "ref": "refs/tags/v1.0.0", - "checkout_sha": "5937ac0a7beb003549fc5fd26fc247adbce4a52e", - "user_id": 1, - "user_name": "John Smith", - "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", - "project_id": 1, - "project":{ - "name":"Example", - "description":"", - "web_url":"http://example.com/jsmith/example", - "avatar_url":null, - "git_ssh_url":"git@example.com:jsmith/example.git", - "git_http_url":"http://example.com/jsmith/example.git", - "namespace":"Jsmith", - "visibility_level":0, - "path_with_namespace":"jsmith/example", - "default_branch":"master", - "homepage":"http://example.com/jsmith/example", - "url":"git@example.com:jsmith/example.git", - "ssh_url":"git@example.com:jsmith/example.git", - "http_url":"http://example.com/jsmith/example.git" - }, - "repository":{ - "name": "Example", - "url": "ssh://git@example.com/jsmith/example.git", - "description": "", - "homepage": "http://example.com/jsmith/example", - "git_http_url":"http://example.com/jsmith/example.git", - "git_ssh_url":"git@example.com:jsmith/example.git", - "visibility_level":0 - }, - "commits": [], - "total_commits_count": 0 -} -``` - -## Merge request events - -Triggered when a new merge request is created, an existing merge request was -updated/merged/closed or a commit is added in the source branch. - -**Request header**: - -```plaintext -X-Gitlab-Event: System Hook -``` - -```json -{ - "object_kind": "merge_request", - "event_type": "merge_request", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project": { - "id": 1, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlabhq/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", - "namespace":"GitlabHQ", - "visibility_level":20, - "path_with_namespace":"gitlabhq/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlabhq/gitlab-test", - "url":"http://example.com/gitlabhq/gitlab-test.git", - "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "http_url":"http://example.com/gitlabhq/gitlab-test.git" - }, - "repository": { - "name": "Gitlab Test", - "url": "http://example.com/gitlabhq/gitlab-test.git", - "description": "Aut reprehenderit ut est.", - "homepage": "http://example.com/gitlabhq/gitlab-test" - }, - "object_attributes": { - "id": 99, - "target_branch": "master", - "source_branch": "ms-viewport", - "source_project_id": 14, - "author_id": 51, - "assignee_id": 6, - "title": "MS-Viewport", - "created_at": "2013-12-03T17:23:34Z", - "updated_at": "2013-12-03T17:23:34Z", - "milestone_id": null, - "state": "opened", - "merge_status": "unchecked", - "target_project_id": 14, - "iid": 1, - "description": "", - "source": { - "name":"Awesome Project", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/awesome_space/awesome_project", - "avatar_url":null, - "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", - "git_http_url":"http://example.com/awesome_space/awesome_project.git", - "namespace":"Awesome Space", - "visibility_level":20, - "path_with_namespace":"awesome_space/awesome_project", - "default_branch":"master", - "homepage":"http://example.com/awesome_space/awesome_project", - "url":"http://example.com/awesome_space/awesome_project.git", - "ssh_url":"git@example.com:awesome_space/awesome_project.git", - "http_url":"http://example.com/awesome_space/awesome_project.git" - }, - "target": { - "name":"Awesome Project", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/awesome_space/awesome_project", - "avatar_url":null, - "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", - "git_http_url":"http://example.com/awesome_space/awesome_project.git", - "namespace":"Awesome Space", - "visibility_level":20, - "path_with_namespace":"awesome_space/awesome_project", - "default_branch":"master", - "homepage":"http://example.com/awesome_space/awesome_project", - "url":"http://example.com/awesome_space/awesome_project.git", - "ssh_url":"git@example.com:awesome_space/awesome_project.git", - "http_url":"http://example.com/awesome_space/awesome_project.git" - }, - "last_commit": { - "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "message": "fixed readme", - "timestamp": "2012-01-03T23:36:29+02:00", - "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "author": { - "name": "GitLab dev user", - "email": "gitlabdev@dv6700.(none)" - } - }, - "work_in_progress": false, - "url": "http://example.com/diaspora/merge_requests/1", - "action": "open", - "assignee": { - "name": "User1", - "username": "user1", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - } - }, - "labels": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "changes": { - "updated_by_id": { - "previous": null, - "current": 1 - }, - "updated_at": { - "previous": "2017-09-15 16:50:55 UTC", - "current":"2017-09-15 16:52:00 UTC" - }, - "labels": { - "previous": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "current": [{ - "id": 205, - "title": "Platform", - "color": "#123123", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "Platform related issues", - "type": "ProjectLabel", - "group_id": 41 - }] - } - } -} -``` - -## Repository Update events - -Triggered only once when you push to the repository (including tags). - -**Request header**: - -```plaintext -X-Gitlab-Event: System Hook -``` - -**Request body:** - -```json -{ - "event_name": "repository_update", - "user_id": 1, - "user_name": "John Smith", - "user_email": "admin@example.com", - "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", - "project_id": 1, - "project": { - "name":"Example", - "description":"", - "web_url":"http://example.com/jsmith/example", - "avatar_url":null, - "git_ssh_url":"git@example.com:jsmith/example.git", - "git_http_url":"http://example.com/jsmith/example.git", - "namespace":"Jsmith", - "visibility_level":0, - "path_with_namespace":"jsmith/example", - "default_branch":"master", - "homepage":"http://example.com/jsmith/example", - "url":"git@example.com:jsmith/example.git", - "ssh_url":"git@example.com:jsmith/example.git", - "http_url":"http://example.com/jsmith/example.git" - }, - "changes": [ - { - "before":"8205ea8d81ce0c6b90fbe8280d118cc9fdad6130", - "after":"4045ea7a3df38697b3730a20fb73c8bed8a3e69e", - "ref":"refs/heads/master" - } - ], - "refs":["refs/heads/master"] -} -``` - -## Local requests in system hooks - -[Requests to local network by system hooks](../security/webhooks.md) can be allowed -or blocked by an administrator. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-06-18>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/tools/email.md b/doc/tools/email.md index 0a3e37719a4..1d46a63bae4 100644 --- a/doc/tools/email.md +++ b/doc/tools/email.md @@ -1,60 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto, reference +redirect_to: '../user/admin_area/email_from_gitlab.md' +remove_date: '2022-06-18' --- -# Email from GitLab **(PREMIUM SELF)** +This document was moved to [another location](../user/admin_area/email_from_gitlab.md). -GitLab provides a tool to administrators for emailing all users, or users of -a chosen group or project, right from the Admin Area. Users receive the email -at their primary email address. - -For information about email notifications originating from GitLab, read -[GitLab notification emails](../user/profile/notifications.md). - -## Use-cases - -- Notify your users about a new project, a new feature, or a new product launch. -- Notify your users about a new deployment, or that downtime is expected - for a particular reason. - -## Sending emails to users from within GitLab - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Overview > Users**. -1. Select **Send email to users**. - -  - -1. Compose an email and choose where to send it (all users or users of a - chosen group or project). The email body only supports plain text messages. - HTML, Markdown, and other rich text formats are not supported, and is - sent as plain text to users. - -  - -NOTE: -[Starting with GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/31509), email notifications can be sent only once every 10 minutes. This helps minimize performance issues. - -## Unsubscribing from emails - -Users can choose to unsubscribe from receiving emails from GitLab by following -the unsubscribe link in the email. Unsubscribing is unauthenticated in order -to keep this feature simple. - -On unsubscribe, users receive an email notification that unsubscribe happened. -The endpoint that provides the unsubscribe option is rate-limited. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-06-18>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 9ab6235164a..ef7598248fe 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -10,7 +10,7 @@ This page gathers all the resources for the topic **Authentication** within GitL ## GitLab users -- [SSH](../../ssh/index.md) +- [SSH](../../user/ssh.md) - [Two-factor authentication](../../user/profile/account/two_factor_authentication.md) - [Why do I keep getting signed out?](../../user/profile/index.md#why-do-i-keep-getting-signed-out) - **Articles:** diff --git a/doc/topics/git/how_to_install_git/index.md b/doc/topics/git/how_to_install_git/index.md index 422919ea46c..1d1264fecf3 100644 --- a/doc/topics/git/how_to_install_git/index.md +++ b/doc/topics/git/how_to_install_git/index.md @@ -79,7 +79,7 @@ Go to the [Git website](https://git-scm.com/), and then download and install Git ## After you install Git -After you successfully install Git on your computer, read about [adding an SSH key to GitLab](../../../ssh/index.md). +After you successfully install Git on your computer, read about [adding an SSH key to GitLab](../../../user/ssh.md). <!-- ## Troubleshooting diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index e95d8121b66..54af1e99797 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -26,10 +26,11 @@ The following resources can help you get started with Git: - [Git-ing started with Git](https://www.youtube.com/watch?v=Ce5nz5n41z4), a video introduction to Git. +- [Make your first Git commit](../../tutorials/make_your_first_git_commit.md) - [Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) - [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) - [How to install Git](how_to_install_git/index.md) -- [Git terminology](../../gitlab-basics/start-using-git.md#git-terminology) +- [Git terminology](terminology.md) - [Start using Git on the command line](../../gitlab-basics/start-using-git.md) - [Edit files through the command line](../../gitlab-basics/command-line-commands.md) - [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index 977f51a7211..db63cee3523 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -10,7 +10,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs/index.html' Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. The general recommendation is to not have Git repositories -larger than 1GB to preserve performance. +larger than 1 GB to preserve performance.  @@ -19,7 +19,7 @@ blob or an LFS pointer. ## How it works -Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication +Git LFS client communicates with the GitLab server over HTTPS. It uses HTTP Basic Authentication to authorize client requests. After the request is authorized, Git LFS client receives instructions from where to fetch or where to push the large file. @@ -29,29 +29,23 @@ Documentation for GitLab instance administrators is under [LFS administration do ## Requirements -- Git LFS is supported in GitLab starting with version 8.2 -- Git LFS must be enabled under project settings -- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up must be installed +- Git LFS must be [enabled in project settings](../../../user/project/settings/index.md#sharing-and-permissions). +- [Git LFS client](https://git-lfs.github.com) version 1.0.1 or higher must be installed. ## Known limitations - Git LFS v1 original API is not supported, because it was deprecated early in LFS development. - When SSH is set as a remote, Git LFS objects still go through HTTPS. -- Any Git LFS request asks for HTTPS credentials to be provided so a good Git - credentials store is recommended. +- Any Git LFS request asks for HTTPS credentials, so we recommend a good Git + credentials store. - Git LFS always assumes HTTPS so if you have GitLab server on HTTP you must [add the URL to Git configuration manually](#troubleshooting). - [Group wikis](../../../user/project/wiki/group.md) do not support Git LFS. -NOTE: -With 8.12 GitLab added LFS support to SSH. The Git LFS communication -still goes over HTTP, but now the SSH client passes the correct credentials -to the Git LFS client. No action is required by the user. - ## Using Git LFS -Lets take a look at the workflow when you need to check large files into your Git +Let's take a look at the workflow when you need to check large files into your Git repository with Git LFS. For example, if you want to upload a very large file and check it into your Git repository: @@ -68,7 +62,7 @@ Git as usual without redoing the command to track a file with the same extension cp ~/tmp/debian.iso ./ # copy a large file into the current directory git add . # add the large file to the project git commit -am "Added Debian iso" # commit the file meta data -git push origin master # sync the git repo and large file to the GitLab server +git push origin main # sync the git repo and large file to the GitLab server ``` **Make sure** that `.gitattributes` is tracked by Git. Otherwise Git @@ -91,7 +85,7 @@ If you already cloned the repository and you want to get the latest LFS object that are on the remote repository, such as for a branch from origin: ```shell -git lfs fetch origin master +git lfs fetch origin main ``` Make sure your files aren't listed in `.gitignore`, otherwise, they are ignored by Git diff --git a/doc/topics/git/terminology.md b/doc/topics/git/terminology.md new file mode 100644 index 00000000000..35814543934 --- /dev/null +++ b/doc/topics/git/terminology.md @@ -0,0 +1,62 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Git terminology + +The following are commonly-used Git terms. + +## Repository + +In GitLab, files are stored in a **repository**. A repository is similar to how you +store files in a folder or directory on your computer. + +- A **remote repository** refers to the files in GitLab. +- A **local copy** refers to the files on your computer. + +<!-- vale gitlab.Spelling = NO --> +<!-- vale gitlab.SubstitutionWarning = NO --> +Often, the word "repository" is shortened to "repo". +<!-- vale gitlab.Spelling = YES --> +<!-- vale gitlab.SubstitutionWarning = YES --> + +In GitLab, a repository is contained in a **project**. + +## Fork + +When you want to contribute to someone else's repository, you make a copy of it. +This copy is called a [**fork**](../../user/project/repository/forking_workflow.md#creating-a-fork). +The process is called "creating a fork." + +When you fork a repo, you create a copy of the project in your own +[namespace](../../user/group/#namespaces). You then have write permissions to modify the project files +and settings. + +For example, you can fork this project, <https://gitlab.com/gitlab-tests/sample-project/>, into your namespace. +You now have your own copy of the repository. You can view the namespace in the URL, for example +`https://gitlab.com/your-namespace/sample-project/`. +Then you can clone the repository to your local machine, work on the files, and submit changes back to the +original repository. + +## Difference between download and clone + +To create a copy of a remote repository's files on your computer, you can either +**download** or **clone** the repository. If you download it, you cannot sync the repository with the +remote repository on GitLab. + +[Cloning](../../gitlab-basics/start-using-git.md#clone-a-repository) a repository is the same as downloading, except it preserves the Git connection +with the remote repository. You can then modify the files locally and +upload the changes to the remote repository on GitLab. + +## Pull and push + +After you save a local copy of a repository and modify the files on your computer, you can upload the +changes to GitLab. This is referred to as **pushing** to the remote, because you use the command +[`git push`](../../gitlab-basics/start-using-git.md#send-changes-to-gitlabcom). + +When the remote repository changes, your local copy is behind. You can update your local copy with the new +changes in the remote repository. +This is referred to as **pulling** from the remote, because you use the command +[`git pull`](../../gitlab-basics/start-using-git.md#download-the-latest-changes-in-the-project). diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index f881826e74a..0aadde7f7c2 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -45,7 +45,7 @@ set to 50MB. The default is 1MB. **If pushing over SSH**, first check your SSH configuration as 'Broken pipe' errors can sometimes be caused by underlying issues with SSH (such as authentication). Make sure that SSH is correctly configured by following the -instructions in the [SSH troubleshooting](../../ssh/index.md#password-prompt-with-git-clone) documentation. +instructions in the [SSH troubleshooting](../../user/ssh.md#password-prompt-with-git-clone) documentation. If you're a GitLab administrator with server access, you can also prevent session timeouts by configuring SSH `keep-alive` on the client or the server. @@ -232,3 +232,38 @@ too small, the error persists. Modifying the server is not always an option, and introduces more potential risk. Attempt local changes first. + +## Password expired error on Git fetch via SSH for LDAP user + +If `git fetch` returns this `HTTP 403 Forbidden` error on a self-managed instance of +GitLab, the password expiration date (`users.password_expires_at`) for this user in the +GitLab database is a date in the past: + +```plaintext +Your password expired. Please access GitLab from a web browser to update your password. +``` + +Requests made with a SSO account and where `password_expires_at` is not `null` +return this error: + +```plaintext +"403 Forbidden - Your password expired. Please access GitLab from a web browser to update your password." +``` + +To resolve this issue, you can update the password expiration by either: + +- Using the `gitlab-rails console`: + + ```ruby + gitlab-rails console + user.update!(password_expires_at: nil) + ``` + +- Using `gitlab-psql`: + + ```sql + # gitlab-psql + UPDATE users SET password_expires_at = null WHERE username='<USERNAME>'; + ``` + +The bug was reported [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332455). diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index d1be775268f..4426f955cb7 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -14,18 +14,18 @@ instance entirely offline. NOTE: This guide assumes the server is Ubuntu 20.04 using the [Omnibus installation method](https://docs.gitlab.com/omnibus/) and will be running GitLab [Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/). Instructions for other servers may vary. This guide also assumes the server host resolves as `my-host.internal`, which you should replace with your -server's FQDN, and that you have access to a different server with Internet access to download the required package files. +server's FQDN, and that you have access to a different server with Internet access to download the required package files. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video walkthrough of this process, see [Offline GitLab Installation: Downloading & Installing](https://www.youtube.com/watch?v=TJaq4ua2Prw). ### Download the GitLab package -You should [manually download the GitLab package](../../update/package/index.md#upgrade-using-a-manually-downloaded-package) and relevant dependencies using a server of the same operating system type that has access to the Internet. +You should [manually download the GitLab package](../../update/package/index.md#upgrade-using-a-manually-downloaded-package) and relevant dependencies using a server of the same operating system type that has access to the Internet. -If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive or writable DVD. +If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive or writable DVD. -In Ubuntu, this can be performed on a server with Internet access using the following commands: +In Ubuntu, this can be performed on a server with Internet access using the following commands: ```shell # Download the bash script to prepare the repository @@ -42,7 +42,7 @@ sudo cp /var/cache/apt/archives/*.deb /path/to/mount Prerequisites: -- Before installing the GitLab package on your offline environment, ensure that you have installed all required dependencies first. +- Before installing the GitLab package on your offline environment, ensure that you have installed all required dependencies first. If you are using Ubuntu, you can install the dependency `.deb` packages you copied across with `dpkg`. Do not install the GitLab package yet. @@ -202,7 +202,7 @@ done. ### Disable Version Check and Service Ping The Version Check and Service Ping services improve the GitLab user experience and ensure that -users are on the most up-to-date instances of GitLab. These two services can be turned off for air-gapped +users are on the most up-to-date instances of GitLab. These two services can be turned off for air-gapped environments so that they do not attempt and fail to reach out to GitLab services. Learn more about [disabling usage statistics](../../user/admin_area/settings/usage_statistics.md#enable-or-disable-usage-statistics). diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md index 7ed227adcac..c791b1f7185 100644 --- a/doc/topics/release_your_application.md +++ b/doc/topics/release_your_application.md @@ -64,3 +64,8 @@ Use GitLab [Releases](../user/project/releases/index.md) to plan, build, and del ### Feature flags Use [feature flags](../operations/feature_flags.md) to control and strategically rollout application deployments. + +## Deploy to Google Cloud + +GitLab [Cloud Seed](../cloud_seed/index.md) is an open-source Incubation Engineering program that +enables you to set up deployment credentials and deploy your application to Google Cloud Run with minimal friction. diff --git a/doc/topics/set_up_organization.md b/doc/topics/set_up_organization.md index 3f8a00f9981..877dbe95e70 100644 --- a/doc/topics/set_up_organization.md +++ b/doc/topics/set_up_organization.md @@ -13,5 +13,5 @@ and give everyone access to the projects they need. - [Workspace](../user/workspace/index.md) _(Coming soon)_ - [Groups](../user/group/index.md) - [User account options](../user/profile/index.md) -- [SSH keys](../ssh/index.md) +- [SSH keys](../user/ssh.md) - [GitLab.com settings](../user/gitlab_com/index.md) diff --git a/doc/topics/use_gitlab.md b/doc/topics/use_gitlab.md index 59a933b1441..6c6c5f71fc2 100644 --- a/doc/topics/use_gitlab.md +++ b/doc/topics/use_gitlab.md @@ -1,21 +1,11 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../user/index.md' +remove_date: '2022-06-17' --- -# Use GitLab **(FREE)** +This document was moved to [another location](../user/index.md). -Get to know the GitLab end-to-end workflow. Configure permissions, -organize your work, create and secure your application, and analyze its performance. Report on team productivity throughout the process. - -- [Set up your organization](set_up_organization.md) -- [Organize work with projects](../user/project/index.md) -- [Plan and track work](plan_and_track.md) -- [Build your application](build_your_application.md) -- [Secure your application](../user/application_security/index.md) -- [Deploy and release your application](release_your_application.md) -- [Monitor application performance](../operations/index.md) -- [Monitor runner performance](https://docs.gitlab.com/runner/monitoring/index.html) -- [Manage your infrastructure](../user/infrastructure/index.md) -- [Analyze GitLab usage](../user/analytics/index.md) +<!-- This redirect file can be deleted after <2022-06-17>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/tutorials/img/branches_dropdown_v14_10.png b/doc/tutorials/img/branches_dropdown_v14_10.png Binary files differnew file mode 100644 index 00000000000..73d94f9823a --- /dev/null +++ b/doc/tutorials/img/branches_dropdown_v14_10.png diff --git a/doc/tutorials/img/clone_project_v14_9.png b/doc/tutorials/img/clone_project_v14_9.png Binary files differnew file mode 100644 index 00000000000..98666c95ba3 --- /dev/null +++ b/doc/tutorials/img/clone_project_v14_9.png diff --git a/doc/tutorials/img/commit_message_v14_10.png b/doc/tutorials/img/commit_message_v14_10.png Binary files differnew file mode 100644 index 00000000000..5636a135b4e --- /dev/null +++ b/doc/tutorials/img/commit_message_v14_10.png diff --git a/doc/tutorials/index.md b/doc/tutorials/index.md index 91913813256..cf3c23a99a7 100644 --- a/doc/tutorials/index.md +++ b/doc/tutorials/index.md @@ -1,7 +1,7 @@ --- stage: none group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +info: For assistance with this tutorials page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # Learn GitLab with tutorials @@ -18,7 +18,7 @@ and running quickly. | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Introduction to GitLab](https://youtu.be/_4SmIyQ5eis?t=90) (59m 51s) | Walk through recommended processes and example workflows for using GitLab. | **{star}** | | [GitLab 101](https://gitlab.edcast.com/pathways/copy-of-gitlab-certification) | Learn the basics of GitLab in this certification course. | **{star}** | | <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Use GitLab for DevOps](https://www.youtube.com/watch?v=7q9Y1Cv-ib0) (12m 34s) | Use GitLab through the entire DevOps lifecycle, from planning to monitoring. | **{star}** | -| [Use Markdown at GitLab](../user/markdown.md) | GitLab Flavored Markdown (GFM) is used in many areas of GitLab, for example, in merge requests. | **{star}** | +| [Use Markdown at GitLab](../user/markdown.md) | GitLab Flavored Markdown (GLFM) is used in many areas of GitLab, for example, in merge requests. | **{star}** | | [GitLab 201](https://gitlab.edcast.com/pathways/ECL-44010cf6-7a9c-4b9b-b684-fa08508a3252) | Go beyond the basics to learn more about using GitLab for your work. | | | Learn GitLab project | You might already have the **Learn GitLab** project, which has tutorial-style issues to help you learn GitLab. If not, download [this export file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/project_templates/learn_gitlab_ultimate.tar.gz) and [import it to a new project](../user/project/settings/import_export.md#import-a-project-and-its-data). | | | [Productivity tips](https://about.gitlab.com/blog/2021/02/18/improve-your-gitlab-productivity-with-these-10-tips/) | Get tips to help make you a productive GitLab user. | | @@ -31,6 +31,7 @@ the most out of GitLab. | Topic | Description | Good for beginners | |-------|-------------|--------------------| +| [Make your first Git commit](make_your_first_git_commit.md) | Create a project, edit a file, and commit changes to a Git repository from the command line. | **{star}** | | [Start using Git on the command line](../gitlab-basics/start-using-git.md) | Learn how to set up Git, clone repositories, and work with branches. | **{star}** | | [Git cheat sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) | Download a PDF of common Git commands. | | diff --git a/doc/tutorials/make_your_first_git_commit.md b/doc/tutorials/make_your_first_git_commit.md new file mode 100644 index 00000000000..4b88b528be6 --- /dev/null +++ b/doc/tutorials/make_your_first_git_commit.md @@ -0,0 +1,273 @@ +--- +stage: none +group: unassigned +info: For assistance with this tutorial, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +--- + +# Make your first Git commit + +This tutorial is going to teach you a little bit about how Git works. It walks +you through the steps of creating your own project, editing a file, and +committing changes to a Git repository from the command line. + +When you're done, you'll have a project where you can practice using Git. + +## What you need + +Before you begin: + +- [Install Git on your local machine](../topics/git/how_to_install_git/index.md). +- Ensure you can sign in to an instance of GitLab. If your organization doesn't + have GitLab, create an account on GitLab.com. +- [Create SSH keys and add them to GitLab](../ssh/index.md). SSH keys are how you + securely communicate between your computer and GitLab. + +## What is Git? + +Before we jump into steps, let's go over some basic Git concepts. + +Git is a version control system. It's used to track changes to files. + +You store files, like code or documents, in a Git *repository*. When you want to edit the files, you +*clone* the repository to your computer, make the changes, and *push* your changes +back to the repository. In GitLab, a Git repository is located in +a *project*. + +Each time you push a change, Git records it as a unique *commit*. These commits make up +the history of when and how a file changed, and who changed it. + +```mermaid +graph LR + subgraph Repository commit history + A(Author: Alex<br>Date: 3 Jan at 1PM<br>Commit message: Added sales figures for January<br> Commit ID: 123abc12) ---> B + B(Author: Sam<br>Date: 4 Jan at 10AM<br>Commit message: Removed outdated marketing information<br> Commit ID: aabb1122) ---> C + C(Author: Zhang<br>Date: 5 Jan at 3PM<br>Commit message: Added a new 'Invoices' file<br> Commit ID: ddee4455) + end +``` + +When you work in a Git repository, you work in *branches*. By default, the contents +of a repository are in a default branch. To make changes, you: + +1. Create your own branch, which is a snapshot of the default branch at the time + you create it. +1. Make changes and push them to your branch. Each push creates a commit. +1. When you're ready, *merge* your branch into the default branch. + +```mermaid +flowchart LR + subgraph Default branch + A[Commit] --> B[Commit] --> C[Commit] --> D[Commit] + end + subgraph My branch + B --1. Create my branch--> E(Commit) + E --2. Add my commit--> F(Commit) + F --2. Add my commit--> G(Commit) + G --3. Merge my branch to default--> D + end +``` + +If this all feels a bit overwhelming, hang in there. You're about to see these concepts in action. + +## Steps + +Here's an overview of what we're going to do: + +1. [Create a sample project](#create-a-sample-project). +1. [Clone the repository](#clone-the-repository). +1. [Create a branch and make your changes](#create-a-branch-and-make-changes). +1. [Commit and push your changes](#commit-and-push-your-changes). +1. [Merge your changes](#merge-your-changes). +1. [View your changes in GitLab](#view-your-changes-in-gitlab). + +### Create a sample project + +To start, create a sample project in GitLab. + +1. In GitLab, on the top bar, select **Menu > Projects > Create new project**. +1. Select **Create blank project**. +1. For **Project name**, enter `My sample project`. The project slug is generated for you. + This slug is the URL you can use to access the project after it's created. +1. Ensure **Initialize repository with a README** is selected. + How you complete the other fields is up to you. +1. Select **Create project**. + +### Clone the repository + +Now you can clone the repository in your project. *Cloning* a repository means you're creating +a copy on your computer, or wherever you want to store and work with the files. + +1. On your project page, select **Clone**. Copy the URL for **Clone with SSH**. + +  + +1. Open a terminal on your computer and go to the directory + where you want to clone the files. + +1. Enter `git clone` and paste the URL: + + ```shell + git clone git@gitlab.com:gitlab-example/my-sample-project.git + ``` + +1. Go to the directory: + + ```shell + cd my-sample-project + ``` + +1. By default, you've cloned the default branch for the repository. Usually this + branch is `main`. To be sure, get the name of the default branch: + + ```shell + git branch + ``` + + The branch you're on is marked with an asterisk. + Press `Q` on your keyboard to return to the main terminal + window. + +### Create a branch and make changes + +Now that you have a copy of the repository, create your own branch so you can +work on your changes independently. + +1. Create a new branch called `example-tutorial-branch`. + + ```shell + git checkout -b example-tutorial-branch + ``` + +1. In a text editor like Visual Studio Code, Sublime, `vi`, or any other editor, + open the README.md file and add this text: + + ```plaintext + Hello world! I'm using Git! + ``` + +1. Save the file. + +1. Git keeps track of changed files. To confirm which files have changed, get + the status. + + ```shell + git status + ``` + + You should get output similar to the following: + + ```shell + On branch example-tutorial-branch + Changes not staged for commit: + (use "git add <file>..." to update what will be committed) + (use "git restore <file>..." to discard changes in working directory) + modified: README.md + + no changes added to commit (use "git add" and/or "git commit -a") + ``` + +### Commit and push your changes + +You've made changes to a file in your repository. Now it's time to record +those changes by making your first commit. + +1. Add the `README.md` file to the *staging* area. The staging area is where you + put files before you commit them. + + ```shell + git add README.md + ``` + +1. Confirm the file is staged: + + ```shell + git status + ``` + + You should get output similar to the following, and the filename should be in + green text. + + ```shell + On branch example-tutorial-branch + Changes to be committed: + (use "git restore --staged <file>..." to unstage) + modified: README.md + ``` + +1. Now commit the staged file, and include a message + that describes the change you made. Make sure you surround the message in double + quotes ("). + + ```shell + git commit -m "I added text to the README file" + ``` + +1. The change has been committed to your branch, but your branch and its commits + are still only available on your computer. No one else has access to them yet. + Push your branch to GitLab: + + ```shell + git push origin example-tutorial-branch + ``` + +Your branch is now available on GitLab and visible to other users in your project. + + + +### Merge your changes + +Now you're ready to merge the changes from your `example-tutorial-branch` branch +to the default branch (`main`). + +1. Check out the default branch for your repository. + + ```shell + git checkout main + ``` + +1. Merge your branch into the default branch. + + ```shell + git merge example-tutorial-branch + ``` + +1. Push the changes. + + ```shell + git push + ``` + +NOTE: +For this tutorial, you merge your branch directly to the default branch for your +repository. In GitLab, you typically use a [merge request](../user/project/merge_requests/) +to merge your branch. + +### View your changes in GitLab + +You did it! You updated the `README.md` file in your branch, and you merged those changes +into the `main` branch. + +Let's look in the UI and confirm your changes. Go to your project. + +- Scroll down and view the contents of the `README.md` file. + Your changes should be visible. +- Above the `README.md` file, view the text in the **Last commit** column. + Your commit message is displayed in this column: + +  + +- Above the file list, select **History** to view your commit details. + +Now you can return to the command line and change back to your personal branch +(`git checkout example-tutorial-branch`). You can continue updating files or +creating new ones. Type `git status` to view the status +of your changes and commit with abandon. + +Don't worry if you mess things up. Everything in Git can be reverted, and if you +find you can't recover, you can always create a new branch and start again. + +Nice work. + +## Find more Git learning resources + +- Get a complete introduction to Git in the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [Git for GitLab](https://www.youtube.com/watch?v=4lxvVj7wlZw) beginner's course (1h 33m). +- Find other tutorials about Git and GitLab on the [tutorials page](index.md). diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index 9453482cc67..7e8eaef2b4c 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -36,6 +36,68 @@ For deprecation reviewers (Technical Writers only): https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-doc --> +View deprecations by the product versions in which they were announced. + +Each deprecation has a **planned removal milestone** and indicates whether it is a breaking change. + +Most of the deprecations are **planned for removal in 15.0**, and many of them are **breaking changes**. + +## 14.10 + +### Manual iteration management + +WARNING: +This feature will be changed or removed in 16.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Manual iteration management is deprecated and only automatic iteration cadences will be supported in the future. + +Creating and deleting iterations will be fully removed in 16.0. Updating all iteration fields except for +`description` will also be removed. + +On the GraphQL API the following mutations will be removed: + + 1. `iterationCreate` + 1. `iterationDelete` + +The update `updateIteration` mutation will only allow updating the iteration's `description`. The following +arguments will be removed: + + 1. `title` + 1. `dueDate` + 1. `startDate` + +For more information about iteration cadences, you can refer to +[the documentation of the feature](https://docs.gitlab.com/ee/user/group/iterations/#iteration-cadences). + +**Planned removal milestone: 16.0 (2023-04-22)** + +### Outdated indices of Advanced Search migrations + +WARNING: +This feature will be changed or removed in 15.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +As Advanced Search migrations usually require support multiple code paths for a long period of time, it’s important to clean those up when we safely can. We use GitLab major version upgrades as a safe time to remove backward compatibility for indices that have not been fully migrated. See the [upgrade documentation](https://docs.gitlab.com/ee/update/index.html#upgrading-to-a-new-major-version) for details. + +**Planned removal milestone: 15.0 (2021-05-22)** + +### Toggle notes confidentiality on APIs + +WARNING: +This feature will be changed or removed in 16.0 +as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Before updating GitLab, review the details carefully to determine if you need to make any +changes to your code, settings, or workflow. + +Toggling notes confidentiality with REST and GraphQL APIs is being deprecated. Updating notes confidential attribute is no longer supported by any means. We are changing this to simplify the experience and prevent private information from being unintentionally exposed. + +**Planned removal milestone: 16.0 (2023-05-22)** + ## 14.9 ### Background upload for object storage @@ -59,7 +121,7 @@ GitLab will publish additional guidance to assist affected customers in migratin ### Deprecate support for Debian 9 -Long term service and support (LTSS) for [Debian 9 Stretch ends in July 2022](https://wiki.debian.org/LTS). Therefore, we will longer support the Debian 9 distribution for the GitLab package. Users can upgrade to Debian 10 or Debian 11. +Long term service and support (LTSS) for [Debian 9 Stretch ends in July 2022](https://wiki.debian.org/LTS). Therefore, we will no longer support the Debian 9 distribution for the GitLab package. Users can upgrade to Debian 10 or Debian 11. **Planned removal milestone: 15.1 (2022-06-22)** @@ -69,17 +131,17 @@ In 15.0, support for daemon mode for GitLab Pages will be removed. **Planned removal milestone: 15.0 (2022-05-22)** -### GitLab self-monitoring +### GitLab self-monitoring project WARNING: -This feature will be changed or removed in 15.0 +This feature will be changed or removed in 16.0 as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -GitLab self-monitoring gives administrators of self-hosted GitLab instances the tools to monitor the health of their instances. This feature is deprecated in GitLab 14.9, and is scheduled for removal in 15.0. +GitLab self-monitoring gives administrators of self-hosted GitLab instances the tools to monitor the health of their instances. This feature is deprecated in GitLab 14.9, and is scheduled for removal in 16.0. -**Planned removal milestone: 15.0 (2022-05-22)** +**Planned removal milestone: 16.0 (2023-05-22)** ### GraphQL permissions change for Package settings @@ -313,12 +375,12 @@ Specifically, the following are deprecated: - Requests that have the `status` field set to `approved`. Beginning in GitLab 15.0, status checks will only be updated to a passing state if the `status` field is both present -and set to `pass`. Requests that: +and set to `passed`. Requests that: - Do not contain the `status` field will be rejected with a `422` error. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338827). -- Contain any value other than `pass` will cause the status check to fail. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/339039). +- Contain any value other than `passed` will cause the status check to fail. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/339039). -To align with this change, API calls to list external status checks will also return the value of `pass` rather than +To align with this change, API calls to list external status checks will also return the value of `passed` rather than `approved` for status checks that have passed. **Planned removal milestone: 15.0 (2022-05-22)** @@ -528,29 +590,6 @@ existing runners. **Planned removal milestone: 16.0 (2023-04-22)** -### Reminder: support for NFS repository storage - -WARNING: -This feature will be changed or removed in 15.0 -as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). -Before updating GitLab, review the details carefully to determine if you need to make any -changes to your code, settings, or workflow. - -As [announced](https://about.gitlab.com/releases/2021/06/22/gitlab-14-0-released/#nfs-for-git-repository-storage-deprecated) at the -release of GitLab 14.0, technical support for NFS storage for Git repositories is being removed. Please see our official -[Statement of Support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for additional information. - -We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on -[migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/#migrating-to-gitaly-cluster). - -Gitaly Cluster offers tremendous benefits for our customers such as: - -- [Variable replication factors](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#configure-replication-factor) -- [Strong consistency](https://docs.gitlab.com/ee/administration/gitaly/#strong-consistency) -- [Distributed read capabilities](https://docs.gitlab.com/ee/administration/gitaly/#distributed-reads) - -**Planned removal milestone: 15.0 (2022-05-22)** - ### Request profiling WARNING: @@ -812,14 +851,29 @@ The new security approvals feature is similar to vulnerability check. For exampl ### `CI_BUILD_*` predefined variables WARNING: -This feature will be changed or removed in 15.0 +This feature will be changed or removed in 16.0 as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -The predefined CI/CD variables that start with `CI_BUILD_*` were deprecated in GitLab 9.0, and will be removed in GitLab 15.0. If you still use these variables, be sure to change to the current [`CI_JOB_*` predefined variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) which are identical (except for the updated name). +The predefined CI/CD variables that start with `CI_BUILD_*` were deprecated in GitLab 9.0, and will be removed in GitLab 16.0. If you still use these variables, be sure to change to the replacement [predefined variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) which are functionally identical: -**Planned removal milestone: 15.0 (2022-05-22)** +| Removed variable | Replacement variable | +| --------------------- |------------------------ | +| `CI_BUILD_BEFORE_SHA` | `CI_COMMIT_BEFORE_SHA` | +| `CI_BUILD_ID` | `CI_JOB_ID` | +| `CI_BUILD_MANUAL` | `CI_JOB_MANUAL` | +| `CI_BUILD_NAME` | `CI_JOB_NAME` | +| `CI_BUILD_REF` | `CI_COMMIT_SHA` | +| `CI_BUILD_REF_NAME` | `CI_COMMIT_REF_NAME` | +| `CI_BUILD_REF_SLUG` | `CI_COMMIT_REF_SLUG` | +| `CI_BUILD_REPO` | `CI_REPOSITORY_URL` | +| `CI_BUILD_STAGE` | `CI_JOB_STAGE` | +| `CI_BUILD_TAG` | `CI_COMMIT_TAG` | +| `CI_BUILD_TOKEN` | `CI_JOB_TOKEN` | +| `CI_BUILD_TRIGGERED` | `CI_PIPELINE_TRIGGERED` | + +**Planned removal milestone: 16.0 (2023-04-22)** ### `fixup!` commit messages setting draft status of associated Merge Request @@ -973,7 +1027,7 @@ The logging features in GitLab allow users to install the ELK stack (Elasticsear ### Monitor performance metrics through Prometheus WARNING: -This feature will be changed or removed in 15.0 +This feature will be changed or removed in 16.0 as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. @@ -981,7 +1035,7 @@ changes to your code, settings, or workflow. By displaying data stored in a Prometheus instance, GitLab allows users to view performance metrics. GitLab also displays visualizations of these metrics in dashboards. The user can connect to a previously-configured external Prometheus instance, or set up Prometheus as a GitLab Managed App. However, since certificate-based integration with Kubernetes clusters is deprecated in GitLab, the metrics functionality in GitLab that relies on Prometheus is also deprecated. This includes the metrics visualizations in dashboards. GitLab is working to develop a single user experience based on [Opstrace](https://about.gitlab.com/press/releases/2021-12-14-gitlab-acquires-opstrace-to-expand-its-devops-platform-with-open-source-observability-solution.html). An [issue exists](https://gitlab.com/groups/gitlab-org/-/epics/6976) for you to follow work on the Opstrace integration. -**Planned removal milestone: 15.0 (2022-05-22)** +**Planned removal milestone: 16.0 (2023-05-22)** ### Pseudonymizer @@ -1294,12 +1348,12 @@ In milestone 15.0, we will remove the `pipelines` attribute from the API respons ### REST and GraphQL API Runner status will not return `paused` WARNING: -This feature will be changed or removed in 15.0 +This feature will be changed or removed in 16.0 as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). Before updating GitLab, review the details carefully to determine if you need to make any changes to your code, settings, or workflow. -The GitLab Runner REST and GraphQL API endpoints will not return `paused` or `active` as a status in GitLab 15.0. +The GitLab Runner REST and GraphQL API endpoints will not return `paused` or `active` as a status in GitLab 16.0. A runner's status will only relate to runner contact status, such as: `online`, `offline`, or `not_connected`. Status `paused` or `active` will no longer appear. @@ -1307,7 +1361,7 @@ A runner's status will only relate to runner contact status, such as: When checking if a runner is `paused`, API users are advised to check the boolean attribute `paused` to be `true` instead. When checking if a runner is `active`, check if `paused` is `false`. -**Planned removal milestone: 15.0 (2022-05-22)** +**Planned removal milestone: 16.0 (2023-04-22)** ### Support for SLES 12 SP2 @@ -1522,13 +1576,7 @@ This will result in the rename of the sub-chart: `gitlab/task-runner` to `gitlab ### NFS for Git repository storage -WARNING: -This feature will be changed or removed in 15.0 -as a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). -Before updating GitLab, review the details carefully to determine if you need to make any -changes to your code, settings, or workflow. - -With the general availability of Gitaly Cluster ([introduced in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/)), we have deprecated development (bugfixes, performance improvements, etc) for NFS for Git repository storage in GitLab 14.0. We will continue to provide technical support for NFS for Git repositories throughout 14.x, but we will remove all support for NFS in GitLab 15.0. Please see our official [Statement of Support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for further information. +With the general availability of Gitaly Cluster ([introduced in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/)), we have deprecated development (bugfixes, performance improvements, etc) for NFS for Git repository storage in GitLab 14.0. We will continue to provide technical support for NFS for Git repositories throughout 14.x, but we will remove all support for NFS on November 22, 2022. This was originally planned for May 22, 2022, but in an effort to allow continued maturity of Gitaly Cluster, we have chosen to extend our deprecation of support date. Please see our official [Statement of Support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for further information. Gitaly Cluster offers tremendous benefits for our customers such as: @@ -1538,7 +1586,7 @@ Gitaly Cluster offers tremendous benefits for our customers such as: We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on [migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/index.html#migrate-to-gitaly-cluster). -**Planned removal milestone: 15.0 (2022-05-22)** +**Planned removal milestone: 15.6 (2022-11-22)** ### OAuth implicit grant diff --git a/doc/update/index.md b/doc/update/index.md index 5a00a728535..1e8badf59b4 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -192,6 +192,8 @@ pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_cla #### Background migrations stuck in 'pending' state GitLab 13.6 introduced an issue where a background migration named `BackfillJiraTrackerDeploymentType2` can be permanently stuck in a **pending** state across upgrades. To clean up this stuck migration, see the [13.6.0 version-specific instructions](#1360). +GitLab 14.4 introduced an issue where a background migration named `PopulateTopicsTotalProjectsCountCache` can be permanently stuck in a **pending** state across upgrades when the instance lacks records that match the migration's target. To clean up this stuck migration, see the [14.4.0 version-specific instructions](#1440). +GitLab 14.8 introduced an issue where a background migration named `PopulateTopicsNonPrivateProjectsCount` can be permanently stuck in a **pending** state across upgrades. To clean up this stuck migration, see the [14.8.0 version-specific instructions](#1480). For other background migrations stuck in pending, run the following check. If it returns non-zero and the count does not decrease over time, follow the rest of the steps in this section. @@ -398,6 +400,8 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap ### 14.8.0 +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, please review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.8.2 or later will reset runner registration tokens for your groups and projects. - The agent server for Kubernetes [is enabled by default](https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#the-agent-server-for-kubernetes-is-enabled-by-default) on Omnibus installations. If you run GitLab at scale, such as [the reference architectures](../administration/reference_architectures/index.md), @@ -417,15 +421,49 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap 1. Add `gitlab_kas['enable'] = false` to `gitlab.rb`. 1. If the server is already upgraded to 14.8, run `gitlab-ctl reconfigure`. +- GitLab 14.8.0 includes a +[background migration `PopulateTopicsNonPrivateProjectsCount`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79140) +that may remain stuck permanently in a **pending** state. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsNonPrivateProjectsCount").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsNonPrivateProjectsCountq", job.arguments) + end + ``` + +- If upgrading from a version earlier than 14.3.0, to avoid + [an issue with job retries](https://gitlab.com/gitlab-org/gitlab/-/issues/357822), first upgrade + to GitLab 14.7.x and make sure all batched migrations have finished. +- If upgrading from version 14.3.0 or later, you might notice a failed + [batched migration](../user/admin_area/monitoring/background_migrations.md) named + `BackfillNamespaceIdForNamespaceRoute`. You can [ignore](https://gitlab.com/gitlab-org/gitlab/-/issues/357822) + this. Retry it after you upgrade to version 14.9.x. ### 14.7.0 - See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472). +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, please review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.7.4 or later will reset runner registration tokens for your groups and projects. +- GitLab 14.7 introduced a change where Gitaly expects persistent files in the `/tmp` directory. + When using the `noatime` mount option on `/tmp` in a node running Gitaly, most Linux distributions + run into [an issue with Git server hooks getting deleted](https://gitlab.com/gitlab-org/gitaly/-/issues/4113). + These conditions are present in the default Amazon Linux configuration. + + If your Linux distribution manages files in `/tmp` with the `tmpfiles.d` service, you + can override the behavior of `tmpfiles.d` for the Gitaly files and avoid this issue: + + ```shell + sudo echo "x /tmp/gitaly-hooks-*" > /etc/tmpfiles.d/gitaly-workaround.conf + ``` ### 14.6.0 - See [LFS objects import and mirror issue in GitLab 14.6.0 to 14.7.2](#lfs-objects-import-and-mirror-issue-in-gitlab-1460-to-1472). - +- If upgrading from a version earlier than 14.6.5, 14.7.4, or 14.8.2, please review the [Critical Security Release: 14.8.2, 14.7.4, and 14.6.5](https://about.gitlab.com/releases/2022/02/25/critical-security-release-gitlab-14-8-2-released/) blog post. + Updating to 14.6.5 or later will reset runner registration tokens for your groups and projects. + ### 14.5.0 - When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you @@ -473,6 +511,17 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo as Sidekiq would continue using a bad connection. Geo and other features that rely on cron jobs running regularly do not work until Sidekiq is restarted. We recommend upgrading to GitLab 14.4.3 and later if this issue affects you. +- GitLab 14.4.0 includes a +[background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) +that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. + + To clean up this stuck job, run the following in the [GitLab Rails Console](../administration/operations/rails_console.md): + + ```ruby + Gitlab::Database::BackgroundMigrationJob.pending.where(class_name: "PopulateTopicsTotalProjectsCountCache").find_each do |job| + puts Gitlab::Database::BackgroundMigrationJob.mark_all_as_succeeded("PopulateTopicsTotalProjectsCountCache", job.arguments) + end + ``` ### 14.3.0 @@ -576,6 +625,8 @@ for how to proceed. - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. - See [Maintenance mode issue in GitLab 13.9 to 14.4](#maintenance-mode-issue-in-gitlab-139-to-144). +- See [Custom Rack Attack initializers](#custom-rack-attack-initializers) if you persist your own custom Rack Attack + initializers during upgrades. #### Upgrading to later 14.Y releases @@ -754,6 +805,14 @@ all servers must first be upgraded to 13.1.Z before upgrading to 13.2.0 or later 1. Only then, continue to upgrade to later versions of GitLab. +#### Custom Rack Attack initializers + +From GitLab 13.0.1, custom Rack Attack initializers (`config/initializers/rack_attack.rb`) are replaced with initializers +supplied with GitLab during upgrades. We recommend you use these GitLab-supplied initializers. + +If you persist your own Rack Attack initializers between upgrades, you might +[get `500` errors](https://gitlab.com/gitlab-org/gitlab/-/issues/334681) when [upgrading to GitLab 14.0 and later](#1400). + ### 12.2.0 In 12.2.0, we enabled Rails' authenticated cookie encryption. Old sessions are diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 22367435ae4..4537faaaead 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -199,6 +199,17 @@ cd /home/git/gitlab git diff origin/PREVIOUS_BRANCH:config/gitlab.yml.example origin/BRANCH:config/gitlab.yml.example ``` +#### New configuration options for `database.yml` + +There might be configuration options available for [`database.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/database.yml.postgresql). +View them with the command below and apply them manually to your current `database.yml`: + +```shell +cd /home/git/gitlab + +git diff origin/PREVIOUS_BRANCH:config/database.yml.postgresql origin/BRANCH:config/database.yml.postgresql +``` + #### NGINX configuration Ensure you're still up-to-date with the latest NGINX configuration changes: diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index d69d60fe73d..5aa80c12f11 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -13,16 +13,14 @@ there are the following requirements: - You can only upgrade one minor release at a time. So from 13.1 to 13.2, not to 13.3. If you skip releases, database modifications may be run in the wrong sequence [and leave the database schema in a broken state](https://gitlab.com/gitlab-org/gitlab/-/issues/321542). -- You have to use [post-deployment migrations](../development/post_deployment_migrations.md). +- You have to use [post-deployment migrations](../development/database/post_deployment_migrations.md). - You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported. -- Multi-node GitLab instance. Single-node instances may experience brief interruptions - [as services restart (Puma in particular)](#single-node-deployment). +- You have set up a multi-node GitLab instance. Single-node instances do not support zero-downtime upgrades. If you meet all the requirements above, follow these instructions in order. There are three sets of steps, depending on your deployment type: | Deployment type | Description | | --------------------------------------------------------------- | ------------------------------------------------ | -| [Single-node](#single-node-deployment) | GitLab CE/EE on a single node | | [Gitaly Cluster](#gitaly-cluster) | GitLab CE/EE using HA architecture for Gitaly Cluster | | [Multi-node / PostgreSQL HA](#use-postgresql-ha) | GitLab CE/EE using HA architecture for PostgreSQL | | [Multi-node / Redis HA](#use-redis-ha-using-sentinel) | GitLab CE/EE using HA architecture for Redis | @@ -87,80 +85,6 @@ migrations this could potentially lead to hours of downtime, depending on the size of your database. To work around this you must use PostgreSQL and meet the other online upgrade requirements mentioned above. -## Single-node deployment - -WARNING: -You can only upgrade one minor release at a time. - -Before following these instructions, note the following **important** information: - -- You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. - If you attempt more than one minor release, the upgrade may fail. -- On single-node Omnibus deployments, updates with no downtime are not possible when - using Puma because Puma always requires a complete restart. This is because the - [phased restart](https://github.com/puma/puma/blob/master/README.md#clustered-mode) - feature of Puma does not work with the way it is configured in GitLab all-in-one - packages (cluster-mode with app preloading). -- While it is possible to minimize downtime on a single-node instance by following - these instructions, **it is not possible to always achieve true zero downtime - updates**. Users may see some connections timeout or be refused for a few minutes, - depending on which services need to restart. -- On Omnibus deployments, the `/etc/gitlab/gitlab.rb` configuration file must **not** have - `gitlab_rails['auto_migrate'] = true`. - -1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab. - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - -1. Update the GitLab package: - - - For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/): - - ```shell - # Debian/Ubuntu - sudo apt-get update - sudo apt-get install gitlab-ee - - # Centos/RHEL - sudo yum install gitlab-ee - ``` - - - For GitLab Community Edition: - - ```shell - # Debian/Ubuntu - sudo apt-get update - sudo apt-get install gitlab-ce - - # Centos/RHEL - sudo yum install gitlab-ce - ``` - -1. To get the regular migrations and latest code in place, run - - ```shell - sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure - ``` - -1. Once the node is updated and `reconfigure` finished successfully, run post-deployment migrations with - - ```shell - sudo gitlab-rake db:migrate - ``` - -1. Hot reload `puma` and `sidekiq` services - - ```shell - sudo gitlab-ctl hup puma - sudo gitlab-ctl restart sidekiq - ``` - -If you do not want to run zero downtime upgrades in the future, make -sure you remove `/etc/gitlab/skip-auto-reconfigure` after -you've completed these steps. - ## Multi-node / HA deployment WARNING: diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index 2ad18d5f70e..077718863e7 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -1,73 +1,9 @@ --- -stage: Manage -group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'dev_ops_reports.md' +remove_date: '2022-06-16' --- -# DevOps Reports **(FREE SELF)** +This document was moved to [another location](dev_ops_reports.md). -DevOps Reports give you an overview of your entire instance's adoption of -[Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) -from planning to monitoring. - -To see DevOps Reports: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Analytics > DevOps Reports**. - -## DevOps Score - -> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. - -NOTE: -To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. - -You can use the DevOps score to compare your DevOps status to other organizations. - -The DevOps Score tab displays usage of major GitLab features on your instance over -the last 30 days, averaged over the number of billable users in that time period. -You can also see the Leader usage score, calculated from top-performing instances based on -[Service Ping data](../settings/usage_statistics.md#service-ping) that GitLab has collected. -Your score is compared to the lead score of each feature and then expressed -as a percentage at the bottom of said feature. Your overall **DevOps Score** is an average of your -feature scores. - -Service Ping data is aggregated on GitLab servers for analysis. Your usage -information is **not sent** to any other GitLab instances. -If you have just started using GitLab, it might take a few weeks for data to be collected before this -feature is available. - -## DevOps Adoption **(ULTIMATE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](../../../policy/alpha-beta-support.md#beta-features). -> - The Overview tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330401) in GitLab 14.1. -> - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. -> - Fuzz Testing metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/330398) in GitLab 14.2. -> - Dependency Scanning metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328034) in GitLab 14.2. -> - Multi-select [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. -> - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. - -DevOps Adoption shows feature adoption for development, security, and operations. - -| Category | Feature | -| --- | --- | -| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | -| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | -| Operations | Deployments<br>Pipelines<br>Runners | - -You can use Group DevOps Adoption to: - -- Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on -their DevOps journey. -- Find subgroups that have adopted certain features, and provide guidance to other subgroups on -how to use those features. -- Verify if you are getting the return on investment that you expected from GitLab. - -## Add or remove a group - -To add or remove a subgroup from the DevOps Adoption report: - -1. Select **Add or remove groups**. -1. Select the subgroup you want to add or remove and select **Save changes**. - - +<!-- This redirect file can be deleted after <2022-06-16>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/admin_area/analytics/dev_ops_reports.md b/doc/user/admin_area/analytics/dev_ops_reports.md new file mode 100644 index 00000000000..2ad18d5f70e --- /dev/null +++ b/doc/user/admin_area/analytics/dev_ops_reports.md @@ -0,0 +1,73 @@ +--- +stage: Manage +group: Optimize +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# DevOps Reports **(FREE SELF)** + +DevOps Reports give you an overview of your entire instance's adoption of +[Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) +from planning to monitoring. + +To see DevOps Reports: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Analytics > DevOps Reports**. + +## DevOps Score + +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. + +NOTE: +To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. + +You can use the DevOps score to compare your DevOps status to other organizations. + +The DevOps Score tab displays usage of major GitLab features on your instance over +the last 30 days, averaged over the number of billable users in that time period. +You can also see the Leader usage score, calculated from top-performing instances based on +[Service Ping data](../settings/usage_statistics.md#service-ping) that GitLab has collected. +Your score is compared to the lead score of each feature and then expressed +as a percentage at the bottom of said feature. Your overall **DevOps Score** is an average of your +feature scores. + +Service Ping data is aggregated on GitLab servers for analysis. Your usage +information is **not sent** to any other GitLab instances. +If you have just started using GitLab, it might take a few weeks for data to be collected before this +feature is available. + +## DevOps Adoption **(ULTIMATE SELF)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](../../../policy/alpha-beta-support.md#beta-features). +> - The Overview tab [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330401) in GitLab 14.1. +> - DAST and SAST metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328033) in GitLab 14.1. +> - Fuzz Testing metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/330398) in GitLab 14.2. +> - Dependency Scanning metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328034) in GitLab 14.2. +> - Multi-select [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. +> - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. + +DevOps Adoption shows feature adoption for development, security, and operations. + +| Category | Feature | +| --- | --- | +| Development | Approvals<br>Code owners<br>Issues<br>Merge requests | +| Security | DAST<br>Dependency Scanning<br>Fuzz Testing<br>SAST | +| Operations | Deployments<br>Pipelines<br>Runners | + +You can use Group DevOps Adoption to: + +- Identify specific subgroups that are lagging in their adoption of GitLab features, so you can guide them on +their DevOps journey. +- Find subgroups that have adopted certain features, and provide guidance to other subgroups on +how to use those features. +- Verify if you are getting the return on investment that you expected from GitLab. + +## Add or remove a group + +To add or remove a subgroup from the DevOps Adoption report: + +1. Select **Add or remove groups**. +1. Select the subgroup you want to add or remove and select **Save changes**. + + diff --git a/doc/user/admin_area/analytics/index.md b/doc/user/admin_area/analytics/index.md index cd505e154c6..9315b926acc 100644 --- a/doc/user/admin_area/analytics/index.md +++ b/doc/user/admin_area/analytics/index.md @@ -15,5 +15,5 @@ Administrators have access to instance-wide analytics: There are several kinds of statistics: -- [DevOps Reports](dev_ops_report.md): Provides an overview of your entire instance's feature usage. +- [DevOps Reports](dev_ops_reports.md): Provides an overview of your entire instance's feature usage. - [Usage Trends](usage_trends.md): Shows how much data your instance contains, and how that is changing. diff --git a/doc/user/admin_area/email_from_gitlab.md b/doc/user/admin_area/email_from_gitlab.md new file mode 100644 index 00000000000..52f27ed48e0 --- /dev/null +++ b/doc/user/admin_area/email_from_gitlab.md @@ -0,0 +1,60 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto, reference +--- + +# Email from GitLab **(PREMIUM SELF)** + +GitLab provides a tool to administrators for emailing all users, or users of +a chosen group or project, right from the Admin Area. Users receive the email +at their primary email address. + +For information about email notifications originating from GitLab, read +[GitLab notification emails](../profile/notifications.md). + +## Use-cases + +- Notify your users about a new project, a new feature, or a new product launch. +- Notify your users about a new deployment, or that downtime is expected + for a particular reason. + +## Sending emails to users from GitLab + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select **Send email to users**. + +  + +1. Compose an email and choose where to send it (all users or users of a + chosen group or project). The email body only supports plain text messages. + HTML, Markdown, and other rich text formats are not supported, and is + sent as plain text to users. + +  + +NOTE: +[Starting with GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/31509), email notifications can be sent only once every 10 minutes. This helps minimize performance issues. + +## Unsubscribing from emails + +Users can choose to unsubscribe from receiving emails from GitLab by following +the unsubscribe link in the email. Unsubscribing is unauthenticated in order +to keep this feature simple. + +On unsubscribe, users receive an email notification that unsubscribe happened. +The endpoint that provides the unsubscribe option is rate-limited. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/tools/email1.png b/doc/user/admin_area/img/email1.png Binary files differindex e79ccc3e9a9..e79ccc3e9a9 100644 --- a/doc/tools/email1.png +++ b/doc/user/admin_area/img/email1.png diff --git a/doc/tools/email2.png b/doc/user/admin_area/img/email2.png Binary files differindex d073c0e42da..d073c0e42da 100644 --- a/doc/tools/email2.png +++ b/doc/user/admin_area/img/email2.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 4a334c28496..f57672d3d36 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -27,7 +27,7 @@ The Admin Area is made up of the following sections: | **{overview}** [Overview](#overview-section) | View your GitLab [Dashboard](#admin-area-dashboard), and administer [projects](#administering-projects), [users](#administering-users), [groups](#administering-groups), [topics](#administering-topics), [jobs](#administering-jobs), [runners](#administering-runners), and [Gitaly servers](#administering-gitaly-servers). | | **{monitor}** Monitoring | View GitLab [system information](#system-information), and information on [background jobs](#background-jobs), [logs](#logs), [health checks](monitoring/health_check.md), [requests profiles](#requests-profiles), and [audit events](#audit-events). | | **{messages}** Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | -| **{hook}** System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | +| **{hook}** System Hooks | Configure [system hooks](../../administration/system_hooks.md) for many events. | | **{applications}** Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | | **{slight-frown}** Abuse Reports | Manage [abuse reports](review_abuse_reports.md) submitted by your users. | | **{license}** License | Add, display, and remove [licenses](license.md). | @@ -79,28 +79,29 @@ To access the Projects page: By default, all projects are listed, in reverse order of when they were last updated. For each project, the following information is listed: -- Name. -- Namespace. -- Description. -- Size, updated every 15 minutes at most. +- Name +- Namespace +- Description +- Size, updated every 15 minutes at most Projects can be edited or deleted. The list of projects can be sorted by: -- Name. -- Last created. -- Oldest created. -- Last updated. -- Oldest updated. -- Owner. +- Updated date +- Last created +- Name +- Most stars +- Oldest created +- Oldest updated +- Largest repository A user can choose to hide or show archived projects in the list. In the **Filter by name** field, type the project name you want to find, and GitLab filters them as you type. -Select from the **Namespace** dropdown to filter only projects in that namespace. +To filter only projects in that namespace, select from the **Namespace** dropdown list. You can combine the filter options. For example, to list only public projects with `score` in their name: diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index bee784e850b..773f91d3076 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -1,6 +1,6 @@ --- stage: Fulfillment -group: License +group: Provision info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -36,7 +36,7 @@ To activate your instance with an activation code: The subscription is activated. -If you have an offline or airgapped environment, +If you have an offline or air gapped environment, [activate GitLab EE with a license file or key](license_file.md) instead. If you have questions or need assistance activating your instance, @@ -51,7 +51,19 @@ To verify the edition, sign in to GitLab and select **Help** (**{question-o}**) > **Help**. The GitLab edition and version are listed at the top of the page. -If you are running GitLab Community Edition (CE), you can upgrade your installation to GitLab +If you are running GitLab Community Edition, you can upgrade your installation to GitLab EE. For more details, see [Upgrading between editions](../../update/index.md#upgrading-between-editions). -If you have questions or need assistance upgrading from GitLab CE to EE, +If you have questions or need assistance upgrading from GitLab Community Edition (CE) to EE, [contact GitLab Support](https://about.gitlab.com/support/#contact-support). + +## Troubleshooting + +### Cannot activate instance due to connectivity error + +This error occurs when you use an activation code to activate your instance, but your instance is unable to connect to the GitLab servers. + +You may have connectivity issues due to the following reasons: + +- **You have an offline or air gapped environment**: Configure your setup to allow connection to GitLab servers. If connection to GitLab servers is not possible, contact [GitLab support](https://about.gitlab.com/support/#contact-support) to request a license key. +- **Firewall settings**: Enable an encrypted HTTPS connection from your GitLab instance to `customers.gitlab.com` (with IP addresses 104.18.26.123 and 104.18.27.123) on port 443. +- **Customers Portal is not operational**: To check for performance or service disruptions, check the Customers Portal [status](https://status.gitlab.com/). diff --git a/doc/user/admin_area/license_file.md b/doc/user/admin_area/license_file.md index e0332e70681..5999e774d26 100644 --- a/doc/user/admin_area/license_file.md +++ b/doc/user/admin_area/license_file.md @@ -1,6 +1,6 @@ --- stage: Fulfillment -group: License +group: Provision info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -23,7 +23,7 @@ Otherwise, to add your license: 1. Select the **Terms of Service** checkbox. 1. Select **Add license**. -## Add your license during installation +## Add your license file during installation You can import a license file when you install GitLab. diff --git a/doc/user/admin_area/merge_requests_approvals.md b/doc/user/admin_area/merge_requests_approvals.md index 0ecf76902e1..ed7fdfe2111 100644 --- a/doc/user/admin_area/merge_requests_approvals.md +++ b/doc/user/admin_area/merge_requests_approvals.md @@ -38,4 +38,4 @@ Merge request approval settings that can be set at an instance level are: See also the following, which are affected by instance-level rules: - [Project merge request approval rules](../project/merge_requests/approvals/index.md). -- [Group merge request approval rules](../group/index.md#group-approval-rules) available in GitLab 13.9 and later. +- [Group merge request approval settings](../group/index.md#group-approval-settings) available in GitLab 13.9 and later. diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index 260a8515a1a..726827054da 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -185,3 +185,12 @@ The results from the query can be plugged into the command: ```shell sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]'] ``` + +### The `BackfillNamespaceIdForNamespaceRoute` batched migration job fails + +In GitLab 14.8, the `BackfillNamespaceIdForNamespaceRoute` batched background migration job +may fail to complete. When retried, a `500 Server Error` is returned. This issue was +[resolved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82387) in GitLab 14.9. + +To resolve this issue, [upgrade GitLab](../../../update/index.md) from 14.8 to 14.9. +You can ignore the failed batch migration until after you update to GitLab 14.9. diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 2f30298644a..e6d8107ed9b 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -199,7 +199,7 @@ To set a limit on how long these sessions are valid: > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.7. [Feature flag ff_limit_ssh_key_lifetime](https://gitlab.com/gitlab-org/gitlab/-/issues/347408) removed. Users can optionally specify a lifetime for -[SSH keys](../../../ssh/index.md). +[SSH keys](../../ssh.md). This lifetime is not a requirement, and can be set to any arbitrary number of days. SSH keys are user credentials to access GitLab. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 0330d89aedc..683c07460ee 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -28,7 +28,18 @@ From now on, every existing project and newly created ones that don't have a If you want to disable it for a specific project, you can do so in [its settings](../../../topics/autodevops/index.md#enable-or-disable-auto-devops). -## Shared runner details +## Enable shared runners for new projects + +You can set all new projects to have the instance's shared runners available by default. + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Select the **Enable shared runners for new projects** checkbox. + +Any time a new project is created, the shared runners are available. + +## Add a message for shared runners To display details about the instance's shared runners in all projects' runner settings: @@ -36,16 +47,17 @@ runner settings: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. -1. Enter your shared runner details in the **Shared runner details** field. +1. Enter text, including Markdown if you want, in the **Shared runner details** field. For example: + +  -You can use [Markdown](../../markdown.md) for improved formatting. To see the rendered -details: +To view the rendered details: 1. On the top bar, select **Menu > Project** and select any group or project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Runners**. - + ## Maximum artifacts size @@ -137,8 +149,8 @@ As an administrator you can set either a global or namespace-specific limit on t ## Archive jobs -Archiving jobs is useful for reducing the CI/CD footprint on the system by -removing some of the capabilities of the jobs (metadata needed to run the job), +Archiving jobs is useful for reducing the CI/CD footprint on the system by removing some +of the capabilities of the jobs (metadata stored in the database needed to run the job), but persisting the traces and artifacts for auditing purposes. To set the duration for which the jobs are considered as old and expired: @@ -149,7 +161,7 @@ To set the duration for which the jobs are considered as old and expired: 1. Set the value of **Archive jobs**. 1. Hit **Save changes** for the changes to take effect. -After that time passes, the jobs are archived and no longer able to be +After that time passes, the jobs are archived in the background and no longer able to be retried. Make it empty to never expire jobs. It has to be no less than 1 day, for example: <code>15 days</code>, <code>1 month</code>, <code>2 years</code>. @@ -178,6 +190,26 @@ of your GitLab instance (`.gitlab-ci.yml` if not set): It is also possible to specify a [custom CI/CD configuration file for a specific project](../../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file). +## Set CI/CD limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352175) in GitLab 14.10. + +You can configure some [CI/CD limits](../../../administration/instance_limits.md#cicd-limits) +from the Admin Area: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand the **Continuous Integration and Deployment** section. +1. In the **CI/CD limits** section, you can set the following limits: + - **Maximum number of jobs in a single pipeline** + - **Total number of jobs in currently active pipelines** + - **Maximum number of active pipelines per project** + - **Maximum number of pipeline subscriptions to and from a project** + - **Maximum number of pipeline schedules** + - **Maximum number of DAG dependencies that a job can have** + - **Maximum number of runners registered per group** + - **Maximum number of runners registered per project** + ## Enable or disable the pipeline suggestion banner By default, a banner displays in merge requests with no pipeline suggesting a @@ -196,7 +228,7 @@ To enable or disable the banner: WARNING: Required pipeline configurations is in its end-of-life process for Premium users. It's -[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) for use in GitLab 14.8, +[deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352316) in GitLab 14.8, and planned to be unavailable for Premium users in GitLab 15.0. This feature is planned to continue to be available for Ultimate users. Ultimate users are not impacted by this deprecation and removal. diff --git a/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_input_v14_10.png b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_input_v14_10.png Binary files differnew file mode 100644 index 00000000000..08451f36962 --- /dev/null +++ b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_input_v14_10.png diff --git a/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png Binary files differdeleted file mode 100644 index d8bc3deccd4..00000000000 --- a/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_10.png b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_10.png Binary files differnew file mode 100644 index 00000000000..64bd9cf6911 --- /dev/null +++ b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_10.png diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 2165dc54899..3eae0d0ff90 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -87,7 +87,7 @@ Alternatively, projects that are marked for removal can be deleted immediately. ## Configure project visibility defaults -To set the default [visibility levels for new projects](../../../public_access/public_access.md): +To set the default [visibility levels for new projects](../../public_access.md): 1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. @@ -112,7 +112,7 @@ To set the default visibility levels for new [snippets](../../snippets.md): 1. Select **Save changes**. For more details on snippet visibility, read -[Project visibility](../../../public_access/public_access.md). +[Project visibility](../../public_access.md). ## Configure group visibility defaults @@ -146,7 +146,7 @@ To restrict visibility levels for projects, snippets, and selected pages: 1. Select **Save changes**. For more details on project visibility, see -[Project visibility](../../../public_access/public_access.md). +[Project visibility](../../public_access.md). ## Configure allowed import sources @@ -162,7 +162,7 @@ You can specify from which hosting sites users can [import their projects](../.. ## Enable project export To enable the export of -[projects and their data](../../../user/project/settings/import_export.md#export-a-project-and-its-data): +[projects and their data](../../project/settings/import_export.md#export-a-project-and-its-data): 1. Sign in to GitLab as a user with Administrator access level. 1. On the top bar, select **Menu > Admin**. diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 8e231d18f41..f0de1e58891 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -34,34 +34,6 @@ To view CI/CD analytics: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Analytics > CI/CD Analytics**. -## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. -> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. - -The DevOps Research and Assessment ([DORA](https://cloud.google.com/blog/products/devops-sre/the-2019-accelerate-state-of-devops-elite-performance-productivity-and-scaling)) -team developed several key metrics that you can use as performance indicators for software development -teams: - -- Deployment frequency: How often an organization successfully releases to production. -- Lead time for changes: The amount of time it takes for code to reach production. -- Change failure rate: The percentage of deployments that cause a failure in production. -- Time to restore service: How long it takes for an organization to recover from a failure in - production. - -### Supported metrics in GitLab - -The following table shows the supported metrics, at which level they are supported, and which GitLab version (API and UI) they were introduced: - -| Metric | Level | API version | Chart (UI) version | Comments | -|---------------------------|---------------------|--------------------------------------|---------------------------------------|-----------| -| `deployment_frequency` | Project-level | [13.7+](../../api/dora/metrics.md) | [13.8+](#view-deployment-frequency-chart) | The [old API endpoint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | -| `deployment_frequency` | Group-level | [13.10+](../../api/dora/metrics.md) | [13.12+](#view-deployment-frequency-chart) | | -| `lead_time_for_changes` | Project-level | [13.10+](../../api/dora/metrics.md) | [13.11+](#view-lead-time-for-changes-chart) | Unit in seconds. Aggregation method is median. | -| `lead_time_for_changes` | Group-level | [13.10+](../../api/dora/metrics.md) | [14.0+](#view-lead-time-for-changes-chart) | Unit in seconds. Aggregation method is median. | -| `change_failure_rate` | Project/Group-level | To be supported | To be supported | | -| `time_to_restore_service` | Project/Group-level | To be supported | To be supported | | - ## View deployment frequency chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.8. diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 01ee9857060..704476cdc90 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -54,52 +54,81 @@ The following analytics features are available for users to create personalized Be sure to review the documentation page for this feature for GitLab tier requirements. +## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. +> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. + +The [DevOps Research and Assessment (DORA)](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) +team developed several key metrics that you can use as performance indicators for software development +teams. + +### Deployment frequency + +Deployment frequency is the frequency of successful deployments to production (hourly, daily, weekly, monthly, or yearly). +This measures how often you deliver value to end users. A higher deployment frequency means you can +get feedback sooner and iterate faster to deliver improvements and features. GitLab measures this as the number of +deployments to a production environment in the given time period. + +Deployment frequency displays in several charts: + +- [Group-level value stream analytics](../group/value_stream_analytics/index.md) +- [Project-level value stream analytics](value_stream_analytics.md) +- [CI/CD analytics](ci_cd_analytics.md) + +### Lead time for changes + +Lead time for changes measures the time to deliver a feature once it has been developed, +as described in [Measuring DevOps Performance](https://devops.com/measuring-devops-performance/). + +Lead time for changes displays in several charts: + +- [Group-level value stream analytics](../group/value_stream_analytics/index.md) +- [Project-level value stream analytics](value_stream_analytics.md) +- [CI/CD analytics](ci_cd_analytics.md) + +### Time to restore service + +Time to restore service measures how long it takes an organization to recover from a failure in production. +GitLab measures this as the average time required to close the incidents +in the given time period. This assumes: + +- All incidents are related to a production environment. +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only +one production deployment, and any production deployment is related to no more than one incident). + +To retrieve metrics for time to restore service, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +### Change failure rate + +Change failure rate measures the percentage of deployments that cause a failure in production. GitLab measures this as the number +of incidents divided by the number of deployments to a +production environment in the given time period. This assumes: + +- All incidents are related to a production environment. +- Incidents and deployments have a strictly one-to-one relationship. An incident is related to only +one production deployment, and any production deployment is related to no +more than one incident. + +To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + +### Supported DORA metrics in GitLab + +| Metric | Level | API | UI chart | Comments | +|---------------------------|-------------------------|-------------------------------------|---------------------------------------|-------------------------------| +| `deployment_frequency` | Project | [GitLab 13.7 and later](../../api/dora/metrics.md) | GitLab 14.8 and later | The [previous API endpoint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | +| `deployment_frequency` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.12 and later | | +| `lead_time_for_changes` | Project | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.11 and later | Unit in seconds. Aggregation method is median. | +| `lead_time_for_changes` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 14.0 and later | Unit in seconds. Aggregation method is median. | +| `time_to_restore_service` | Project and group | [GitLab 14.9 and later](../../api/dora/metrics.md) | Not supported | | +| `change_failure_rate` | Project and group | [GitLab 14.10 and later](../../api/dora/metrics.md) | Not supported | | + ## Definitions We use the following terms to describe GitLab analytics: - **Cycle time:** The duration of only the execution work. Cycle time is often displayed in combination with the lead time, which is longer than the cycle time. GitLab measures cycle time from the earliest commit of a [linked issue's merge request](../project/issues/crosslinking_issues.md) to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation is always later than commit time. GitLab displays cycle time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](../analytics/value_stream_analytics.md). - **Deploys:** The total number of successful deployments to production in the given time frame (across all applicable projects). GitLab displays deploys in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). -- **DORA (DevOps Research and Assessment)** ["Four Keys"](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance): - - **Speed/Velocity** - - - **Deployment frequency:** The relative frequency of successful deployments to production - (hourly, daily, weekly, monthly, or yearly). - This measures how often you are delivering value to end users. A higher deployment - frequency means you are able to get feedback and iterate faster to deliver - improvements and features. GitLab measures this as the number of deployments to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in - the given time period. - GitLab displays deployment frequency in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). - - **Lead Time for Changes:** The time it takes for a commit to get into production. GitLab - measures this as the median duration between merge request merge and deployment to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) for - all MRs deployed in the given time period. This measure under estimates lead time because - merge time is always later than commit time. The - [standard definition](https://github.com/GoogleCloudPlatform/fourkeys/blob/main/METRICS.md#lead-time-for-changes) uses median commit time. - [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/328459) to start - measuring from "issue first commit" as a better proxy, although still imperfect. - - - **Stability** - - **Change Failure Rate:** The percentage of deployments causing a failure in production. - GitLab measures this as the number of [incidents](../../operations/incident_management/incidents.md) - divided by the number of deployments to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in - the given time period. This assumes: - - - All incidents are related to a production environment. - - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is - related to only one production deployment, and any production deployment is related to no - more than one incident). - - - **Time to Restore Service:** How long it takes an organization to recover from a failure in - production. GitLab measures this as the average time required to close the - [incidents](../../operations/incident_management/incidents.md) in the given time period. - This assumes: - - - All incidents are related to a [production environment](../../ci/environments/index.md#deployment-tier-of-environments). - - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is related to only one production deployment, and any production deployment is related to no more than one incident). - - **Lead time:** The duration of your value stream, from start to finish. Different to [Lead time for changes](#lead-time-for-changes). Often displayed in combination with "cycle time," which is shorter. GitLab measures lead time from issue creation to issue close. GitLab displays lead @@ -121,8 +150,3 @@ with "plan" and ends with "monitor". GitLab helps you track your value stream us - **Velocity:** The total issue burden completed in some period of time. The burden is usually measured in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab measures velocity as the total points or weight of issues closed in a given period of time. - -## Lead time for changes - -"Lead Time for Changes" differs from "Lead Time" because it "focuses on measuring only the time to -deliver a feature once it has been developed", as described in ([Measuring DevOps Performance](https://devops.com/measuring-devops-performance/)). diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 5413c28912a..ed94686b7a3 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -91,20 +91,25 @@ The API fuzzing configuration form helps you create or modify your project's API configuration. The form lets you choose values for the most common API fuzzing options and builds a YAML snippet that you can paste in your GitLab CI/CD configuration. -#### Configure Web API fuzzing with the configuration form +#### Configure Web API fuzzing in the UI To generate an API Fuzzing configuration snippet: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. -1. In the **API Fuzzing** row, select **Configure**. -1. Complete the form as needed. Read below for more information on available configuration options. +1. In the **API Fuzzing** row, select **Enable API Fuzzing**. +1. Complete the fields. For details see [Available CI/CD variables](#available-cicd-variables). 1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the options you've selected in the form. -1. Choose one of the following actions: - 1. To copy the snippet to your clipboard and be redirected to your project's `.gitlab-ci.yml` file, - where you can paste the YAML configuration, select **Copy code and open `.gitlab-ci.yml` file**. - 1. To copy the snippet to your clipboard and close the modal, select **Copy code only**. +1. Do one of the following: + 1. To copy the snippet to your clipboard, select **Copy code only**. + 1. To add the snippet to your project's `.gitlab-ci.yml` file, select + **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. + 1. Paste the snippet into the `.gitlab-ci.yml` file. + 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. + 1. Select the **Edit** tab, then select **Commit changes**. + +When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include an API Fuzzing job. ### OpenAPI Specification @@ -112,6 +117,7 @@ To generate an API Fuzzing configuration snippet: > - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. > - Support for OpenAPI Specification v3.1 was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.2. > - Support to generate media type `application/xml` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.8. +> - Support to select media types was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. @@ -125,6 +131,25 @@ the body generation is limited to these body types: - `application/json` - `application/xml` +### OpenAPI and media types + +A media type (formerly known as MIME type) is an identifier for file formats and format contents transmitted. A OpenAPI document lets you specify that a given operation can accept different media types, hence a given request can send data using different file content. As for example, a `PUT /user` operation to update user data could accept data in either XML (media type `application/xml`) or JSON (media type `application/json`) format. +OpenAPI 2.x lets you specify the accepted media types globally or per operation, and OpenAPI 3.x lets you specify the accepted media types per operation. API Fuzzing checks the listed media types and tries to produce sample data for each supported media type. + +- In [GitLab 14.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/333304), the default behavior is to select one of the supported media types to use. The first supported media type is chosen from the list. This behavior is configurable. +- In GitLab 14.9 and earlier, the default behavior is to perform testing using all supported media types. This means if two media types are listed (for example, `application/json` and `application/xml`), tests are performed using JSON, and then the same tests using XML. + +Testing the same operation (for example, `POST /user`) using different media types (for example, `application/json` and `application/xml`) is not always desirable. +For example, if the target application executes the same code regardless of the request content type, it will take longer to finish the test session, and it may report duplicate vulnerabilities related to the request body depending on the target app. + +The environment variable `FUZZAPI_OPENAPI_ALL_MEDIA_TYPES` lets you specify whether or not to use all supported media types instead of one when generating requests for a given operation. When the environmental variable `FUZZAPI_OPENAPI_ALL_MEDIA_TYPES` is set to any value, API Fuzzing will try to generate requests for all supported media types instead of one in a given operation. This will cause testing to take longer as testing is repeated for each provided media type. + +Alternatively, the variable `FUZZAPI_OPENAPI_MEDIA_TYPES` is used to provide a list of media types that will each be tested. Providing more than one media type causes testing to take longer, as testing is performed for each media type selected. When the environment variable `FUZZAPI_OPENAPI_MEDIA_TYPES` is set to a list of media types, only the listed media types are included when creating requests. + +Multiple media types in `FUZZAPI_OPENAPI_MEDIA_TYPES` must separated by a colon (`:`). For example, to limit request generation to the media types `application/x-www-form-urlencoded` and `multipart/form-data`, set the environment variable `FUZZAPI_OPENAPI_MEDIA_TYPES` to `application/x-www-form-urlencoded:multipart/form-data`. Only supported media types in this list are included when creating requests, though unsupported media types are always skipped. A media type text may contain different sections. For example, `application/vnd.api+json; charset=UTF-8` is a compound of `type "/" [tree "."] subtype ["+" suffix]* [";" parameter]`. Parameters are not taken into account when filtering media types on request generation. + +The environment variables `FUZZAPI_OPENAPI_ALL_MEDIA_TYPES` and `FUZZAPI_OPENAPI_MEDIA_TYPES` allow you to decide how to handle media types. These settings are mutually exclusive. If both are enabled, API Fuzzing reports an error. + #### Configure Web API fuzzing with an OpenAPI Specification To configure API fuzzing in GitLab with an OpenAPI Specification: @@ -345,8 +370,8 @@ By default, the API fuzzer uses the Postman file to resolve Postman variable val is set in a GitLab CI/CD variable `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`, then the JSON file takes precedence to get Postman variable values. -Although Postman can export environment variables into a JSON file, the format is not compatible -with the JSON expected by `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`. +WARNING: +Although Postman can export environment variables into a JSON file, the format is not compatible with the JSON expected by `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`. Here is an example of using `FUZZAPI_POSTMAN_COLLECTION_VARIABLES`: @@ -561,13 +586,19 @@ profile increases as the number of tests increases. | CI/CD variable | Description | |-------------------------------------------------------------|-------------| | `SECURE_ANALYZERS_PREFIX` | Specify the Docker registry base address from which to download the analyzer. | -| `FUZZAPI_VERSION` | Specify API Fuzzing container version. Defaults to `latest`. | +| `FUZZAPI_VERSION` | Specify API Fuzzing container version. Defaults to `1`. | +| `FUZZAPI_IMAGE_SUFFIX` | Specify a container image suffix. Defaults to none. | | `FUZZAPI_TARGET_URL` | Base URL of API testing target. | | `FUZZAPI_CONFIG` | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/276395) in GitLab 13.12, replaced with default `.gitlab/gitlab-api-fuzzing-config.yml`. API Fuzzing configuration file. | |[`FUZZAPI_PROFILE`](#api-fuzzing-profiles) | Configuration profile to use during testing. Defaults to `Quick-10`. | |[`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | +|[`FUZZAPI_EXCLUDE_URLS`](#exclude-urls) | Exclude API URL from testing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357195) in GitLab 14.10. | +|[`FUZZAPI_EXCLUDE_PARAMETER_ENV`](#exclude-parameters) | JSON string containing excluded parameters. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292196) in GitLab 14.10. | +|[`FUZZAPI_EXCLUDE_PARAMETER_FILE`](#exclude-parameters) | Path to a JSON file containing excluded parameters. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292196) in GitLab 14.10. | |[`FUZZAPI_OPENAPI`](#openapi-specification) | OpenAPI Specification file or URL. | |[`FUZZAPI_OPENAPI_RELAXED_VALIDATION`](#openapi-specification) | Relax document validation. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345950) in GitLab 14.7. | +|[`FUZZAPI_OPENAPI_ALL_MEDIA_TYPES`](#openapi-specification) | Use all supported media types instead of one when generating requests. Causes test duration to be longer. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. | +|[`FUZZAPI_OPENAPI_MEDIA_TYPES`](#openapi-specification) | Colon (`:`) separated media types accepted for testing. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. | |[`FUZZAPI_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | |[`FUZZAPI_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | |[`FUZZAPI_POSTMAN_COLLECTION_VARIABLES`](#postman-variables) | Path to a JSON file to extract Postman variable values. | @@ -891,7 +922,7 @@ def get_auth_response(): # In our example, access token is retrieved from a given endpoint try: - # Performs a http request, response sample: + # Performs a http request, response sample: # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" } response = get_auth_response() @@ -921,7 +952,7 @@ except Exception as e: logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') raise -# computes object that holds overrides file content. +# computes object that holds overrides file content. # It uses data fetched from request overrides_data = { "headers": { @@ -1036,6 +1067,294 @@ variables: FUZZAPI_EXCLUDE_PATHS=/auth*;/v1/* ``` +### Exclude parameters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292196) in GitLab 14.10. + +While testing an API you may might want to exclude a parameter (query string, header, or body element) from testing. This may be needed because a parameter always causes a failure, slows down testing, or for other reasons. To exclude parameters you can use one of the following variables: `FUZZAPI_EXCLUDE_PARAMETER_ENV` or `FUZZAPI_EXCLUDE_PARAMETER_FILE`. + +The `FUZZAPI_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `FUZZAPI_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime from a pre script using `FUZZAPI_PRE_SCRIPT`. + +#### Exclude parameters using a JSON document + +The JSON document contains a JSON object which uses specific properties to identify which parameter should be excluded. +You can provide the following properties to exclude specific parameters during the scanning process: + +- `headers`: Use this property to exclude specific headers. The property's value is an array of header names to be excluded. Names are case-insensitive. +- `cookies`: Use this property's value to exclude specific cookies. The property's value is an array of cookie names to be excluded. Names are case-sensitive. +- `query`: Use this property to exclude specific fields from the query string. The property's value is an array of field names from the query string to be excluded. Names are case-sensitive. +- `body-form`: Use this property to exclude specific fields from a request that uses the media type `application/x-www-form-urlencoded`. The property's value is an array of the field names from the body to be excluded. Names are case-sensitive. +- `body-json`: Use this property to exclude specific JSON nodes from a request that uses the media type `application/json`. The property's value is an array, each entry of the array is a [JSON Path](https://goessner.net/articles/JsonPath/) expression. +- `body-xml`: Use this property to exclude specific XML nodes from a request that uses media type `application/xml`. The property's value is an array, each entry of the array is a [XPath v2](https://www.w3.org/TR/xpath20/) expression. + +The following JSON document is an example of the expected structure to exclude parameters. + +```json +{ + "headers": [ + "header1", + "header2" + ], + "cookies": [ + "cookie1", + "cookie2" + ], + "query": [ + "query-string1", + "query-string2" + ], + "body-form": [ + "form-param1", + "form-param2" + ], + "body-json": [ + "json-path-expression-1", + "json-path-expression-2" + ], + "body-xml" : [ + "xpath-expression-1", + "xpath-expression-2" + ] +} +``` + +#### Examples + +##### Excluding a single header + +To exclude the header `Upgrade-Insecure-Requests`, set the `header` property's value to an array with the header name: `[ "Upgrade-Insecure-Requests" ]`. For instance, the JSON document looks like this: + +```json +{ + "headers": [ "Upgrade-Insecure-Requests" ] +} +``` + +Header names are case-insensitive, thus the header name `UPGRADE-INSECURE-REQUESTS` is equivalent to `Upgrade-Insecure-Requests`. + +##### Excluding both a header and two cookies + +To exclude the header `Authorization` and the cookies `PHPSESSID` and `csrftoken`, set the `headers` property's value to an array with header name `[ "Authorization" ]` and the `cookies` property's value to an array with the cookies' names `[ "PHPSESSID", "csrftoken" ]`. For instance, the JSON document looks like this: + +```json +{ + "headers": [ "Authorization" ], + "cookies": [ "PHPSESSID", "csrftoken" ] +} +``` + +##### Excluding a `body-form` parameter + +To exclude the `password` field in a request that uses `application/x-www-form-urlencoded`, set the `body-form` property's value to an array with the field name `[ "password" ]`. For instance, the JSON document looks like this: + +```json +{ + "body-form": [ "password" ] +} +``` + +The exclude parameters uses `body-form` when the request uses a content type `application/x-www-form-urlencoded`. + +##### Excluding a specific JSON nodes using JSON Path + +To exclude the `schema` property in the root object, set the `body-json` property's value to an array with the JSON Path expression `[ "$.schema" ]`. + +The JSON Path expression uses special syntax to identify JSON nodes: `$` refers to the root of the JSON document, `.` refers to the current object (in our case the root object), and the text `schema` refers to a property name. Thus, the JSON path expression `$.schema` refers to a property `schema` in the root object. +For instance, the JSON document looks like this: + +```json +{ + "body-json": [ "$.schema" ] +} +``` + +The exclude parameters uses `body-json` when the request uses a content type `application/json`. Each entry in `body-json` is expected to be a [JSON Path expression](https://goessner.net/articles/JsonPath/). In JSON Path, characters like `$`, `*`, `.` among others have special meaning. + +##### Excluding multiple JSON nodes using JSON Path + +To exclude the property `password` on each entry of an array of `users` at the root level, set the `body-json` property's value to an array with the JSON Path expression `[ "$.users[*].paswword" ]`. + +The JSON Path expression starts with `$` to refer to the root node and uses `.` to refer to the current node. Then, it uses `users` to refer to a property and the characters `[` and `]` to enclose the index in the array you want to use, instead of providing a number as an index you use `*` to specify any index. After the index reference, we find `.` which now refers to any given selected index in the array, preceded by a property name `password`. + +For instance, the JSON document looks like this: + +```json +{ + "body-json": [ "$.users[*].paswword" ] +} +``` + +The exclude parameters uses `body-json` when the request uses a content type `application/json`. Each entry in `body-json` is expected to be a [JSON Path expression](https://goessner.net/articles/JsonPath/). In JSON Path characters like `$`, `*`, `.` among others have special meaning. + +##### Excluding an XML attribute + +To exclude an attribute named `isEnabled` located in the root element `credentials`, set the `body-xml` property's value to an array with the XPath expression `[ "/credentials/@isEnabled" ]`. + +The XPath expression `/credentials/@isEnabled`, starts with `/` to indicate the root of the XML document, then it is followed by the word `credentials` which indicates the name of the element to match. It uses a `/` to refer to a node of the previous XML element, and the character `@` to indicate that the name `isEnable` is an attribute. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/@isEnabled" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be an [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions, characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding an XML element's text + +To exclude the text of the `username` element contained in root node `credentials`, set the `body-xml` property's value to an array with the XPath expression `[/credentials/username/text()" ]`. + +In the XPath expression `/credentials/username/text()`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `username`. Last part has a `/` that refers to the current element, and uses a XPath function called `text()` which identifies the text of the current element. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/username/text()" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding an XML element + +To exclude the element `username` contained in root node `credentials`, set the `body-xml` property's value to an array with the XPath expression `[/credentials/username" ]`. + +In the XPath expression `/credentials/username`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `username`. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/username" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding an XML node with namespaces + +To exclude a XML element `login` which is defined in namespace `s`, and contained in `credentials` root node, set the `body-xml` property's value to an array with the XPath expression `[ "/credentials/s:login" ]`. + +In the XPath expression `/credentials/s:login`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `s:login`. Notice that name contains the character `:`, this character separates the namespace from the node name. + +The namespace name should have been defined in the XML document which is part of the body request. You may check the namespace in the specification document HAR, OpenAPI, or Postman Collection file. + +```json +{ + "body-xml": [ + "/credentials/s:login" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +#### Using a JSON string + +To provide the exclusion JSON document set the variable `FUZZAPI_EXCLUDE_PARAMETER_ENV` with the JSON string. In the following example, the `.gitlab-ci.yml`, the `FUZZAPI_EXCLUDE_PARAMETER_ENV` variable is set to a JSON string: + +```yaml +stages: + - fuzz + +include: + - template: API-Fuzzing.gitlab-ci.yml + +variables: + FUZZAPI_PROFILE: Quick + FUZZAPI_OPENAPI: test-api-specification.json + FUZZAPI_TARGET_URL: http://test-deployment/ + FUZZAPI_EXCLUDE_PARAMETER_ENV: '{ "headers": [ "Upgrade-Insecure-Requests" ] }' +``` + +#### Using a file + +To provide the exclusion JSON document, set the variable `FUZZAPI_EXCLUDE_PARAMETER_FILE` with the JSON file path. The file path is relative to the job current working directory. In the following example `.gitlab-ci.yml` file, the `FUZZAPI_EXCLUDE_PARAMETER_FILE` variable is set to a JSON file path: + +```yaml +stages: + - fuzz + +include: + - template: API-Fuzzing.gitlab-ci.yml + +variables: + FUZZAPI_PROFILE: Quick + FUZZAPI_OPENAPI: test-api-specification.json + FUZZAPI_TARGET_URL: http://test-deployment/ + FUZZAPI_EXCLUDE_PARAMETER_FILE: api-fuzzing-exclude-parameters.json +``` + +The `api-fuzzing-exclude-parameters.json` is a JSON document that follows the structure of [exclude parameters document](#exclude-parameters-using-a-json-document). + +### Exclude URLS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357195) in GitLab 14.10. + +As an alternative to excluding by paths, you can filter by any other component in the URL by using the `FUZZAPI_EXCLUDE_URLS` CI/CD variable. This variable can be set in your `.gitlab-ci.yml` file. The variable can store multiple values, separated by commas (`,`). Each value is a regular expression. Because each entry is a regular expression, an entry such as `.*` excludes all URLs because it is a regular expression that matches everything. + +In your job output you can check if any URLs matched any provided regular expression from `FUZZAPI_EXCLUDE_URLS`. Matching operations are listed in the **Excluded Operations** section. Operations listed in the **Excluded Operations** should not be listed in the **Tested Operations** section. For example the following portion of a job output: + +```plaintext +2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +``` + +NOTE: +Each value in `FUZZAPI_EXCLUDE_URLS` is a regular expression. Characters such as `.` , `*` and `$` among many others have special meanings in [regular expressions](https://en.wikipedia.org/wiki/Regular_expression#Standards). + +#### Examples + +##### Excluding a URL and child resources + +The following example excludes the URL `http://target/api/auth` and its child resources. + +```yaml +variables: + FUZZAPI_EXCLUDE_URLS: http://target/api/auth +``` + +##### Excluding two URLs and allow their child resources + +To exclude the URLs `http://target/api/buy` and `http://target/api/sell` but allowing to scan their child resources, for instance: `http://target/api/buy/toy` or `http://target/api/sell/chair`. You could use the value `http://target/api/buy/$,http://target/api/sell/$`. This value is using two regular expressions, each of them separated by a `,` character. Hence, it contains `http://target/api/buy$` and `http://target/api/sell$`. In each regular expression, the trailing `$` character points out where the matching URL should end. + +```yaml +variables: + FUZZAPI_EXCLUDE_URLS: http://target/api/buy/$,http://target/api/sell/$ +``` + +##### Excluding two URLs and their child resources + +In order to exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: + +```yaml +variables: + FUZZAPI_EXCLUDE_URLS: http://target/api/buy,http://target/api/sell +``` + +##### Excluding URL using regular expressions + +In order to exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more). We could use `https://target/api/v.*/user/create$`, in the previous regular expression `.` indicates any character and `*` indicates zero or more times, additionally `$` indicates that the URL should end there. + +```yaml +variables: + FUZZAPI_EXCLUDE_URLS: https://target/api/v.*/user/create$ +``` + ### Header Fuzzing Header fuzzing is disabled by default due to the high number of false positives that occur with many @@ -1495,6 +1814,19 @@ API Security can still try to consume an OpenAPI document that does not fully co FUZZAPI_OPENAPI_RELAXED_VALIDATION: On ``` +### No operation in the OpenAPI document is consuming any supported media type + +API Security uses the specified media types in the OpenAPI document to generate requests. If no request can be created due to the lack of supported media types, then an error will be thrown. + +**Error message** + +- In [GitLab 14.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/333304), `Error, no operation in the OpenApi document is consuming any supported media type. Check 'OpenAPI Specification' to check the supported media types.` + +**Solution** + +1. Review the supported media types in the [OpenAPI Specification](#openapi-specification) section. +1. Edit your OpenAPI document, allowing at least a given operation to accept any of the supported media types. Alternatively, a supported media type could be set in the OpenAPI document level and get applied to all operations. This step may require changes in your application to ensure the supported media type is accepted by the application. + ## Get support or request an improvement To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index f2d6cef669d..64566e458ee 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -186,7 +186,7 @@ The `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` CI/CD variable controls whether the findings related to programming languages. The languages supported depend on the [scanner used](#change-scanners): -- [Trivy](https://aquasecurity.github.io/trivy/latest/vulnerability/detection/language/). +- [Trivy](https://aquasecurity.github.io/trivy/latest/docs/vulnerability/detection/language/). - [Grype](https://github.com/anchore/grype#features). By default, the report only includes packages managed by the Operating System (OS) package manager @@ -222,6 +222,7 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u | `CS_DISABLE_DEPENDENCY_LIST` | `"false"` | Disable Dependency Scanning for packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DISABLE_LANGUAGE_VULNERABILITY_SCAN` | `"true"` | Disable scanning for language-specific packages installed in the scanned image. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. | All | | `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | +| `CS_IMAGE_SUFFIX` | `""` | Suffix added to `CS_ANALYZER_IMAGE`. If set to `-fips`, `FIPS-enabled` image is used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/7630) in GitLab 14.10. | All | | `CS_IGNORE_UNFIXED` | `"false"` | Ignore vulnerabilities that are not fixed. | All | | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | | `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | @@ -236,22 +237,29 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u Support depends on the scanner: - [Grype](https://github.com/anchore/grype#grype) -- [Trivy](https://aquasecurity.github.io/trivy/latest/vulnerability/detection/os/) (Default). +- [Trivy](https://aquasecurity.github.io/trivy/latest/docs/vulnerability/detection/os/) (Default). -#### UBI-based images +#### FIPS-enabled images > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5775) in GitLab 14.1. -GitLab also offers [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) -versions of the container-scanning images. You can therefore replace standard images with UBI-based -images. To configure the images, set the `CS_ANALYZER_IMAGE` variable to the standard tag plus the -`-ubi` extension. +GitLab also offers [FIPS-enabled Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) +versions of the container-scanning images. You can therefore replace standard images with FIPS-enabled +images. To configure the images, set the `CS_IMAGE_SUFFIX` to `-fips` or modify the `CS_ANALYZER_IMAGE` variable to the +standard tag plus the `-fips` extension. | Scanner name | `CS_ANALYZER_IMAGE` | | --------------- | ------------------- | -| Default (Trivy) | `registry.gitlab.com/security-products/container-scanning:4-ubi` | -| Grype | `registry.gitlab.com/security-products/container-scanning/grype:4-ubi` | -| Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:4-ubi` | +| Default (Trivy) | `registry.gitlab.com/security-products/container-scanning:4-fips` | +| Grype | `registry.gitlab.com/security-products/container-scanning/grype:4-fips` | +| Trivy | `registry.gitlab.com/security-products/container-scanning/trivy:4-fips` | + +NOTE: +Prior to GitLab 15.0, the `-ubi` image extension is also available. GitLab 15.0 and later only +support `-fips`. + +Starting with GitLab 14.10, `-fips` is automatically added to `CS_ANALYZER_IMAGE` when FIPS mode is +enabled in the GitLab instance. ### Enable Container Scanning through an automatic merge request @@ -753,7 +761,7 @@ The images include the latest advisory database available for their respective s scanner includes data from multiple sources: - [Grype](https://github.com/anchore/grype#grypes-database). -- [Trivy](https://aquasecurity.github.io/trivy/latest/vulnerability/detection/data-source/). +- [Trivy](https://aquasecurity.github.io/trivy/latest/docs/vulnerability/detection/data-source/). Database update information for other analyzers is available in the [maintenance table](../index.md#vulnerability-scanner-maintenance). diff --git a/doc/user/application_security/dast/checks/598.2.md b/doc/user/application_security/dast/checks/598.2.md new file mode 100644 index 00000000000..f6c6787128d --- /dev/null +++ b/doc/user/application_security/dast/checks/598.2.md @@ -0,0 +1,30 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Use of GET request method with sensitive query strings (password) + +## Description + +The user's password was identified in the request URL. Passwords should never be sent in GET +requests as they maybe captured by proxy systems, stored in browser history, or stored in +log files. If an attacker were to get access to these logs or logging systems, they would +be able to gain access to the target account. + +## Remediation + +Passwords should never be sent in GET requests. When authenticating users or requesting users +reset their passwords, always use POST requests to transmit sensitive data. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 598.2 | true | 598 | Passive | Medium | + +## Links + +- [OWASP](https://owasp.org/www-community/vulnerabilities/Information_exposure_through_query_strings_in_url) +- [CWE](https://cwe.mitre.org/data/definitions/598.html) diff --git a/doc/user/application_security/dast/checks/598.3.md b/doc/user/application_security/dast/checks/598.3.md new file mode 100644 index 00000000000..fa6fdf43e1c --- /dev/null +++ b/doc/user/application_security/dast/checks/598.3.md @@ -0,0 +1,31 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Use of GET request method with sensitive query strings (Authorization header details) + +## Description + +The authorization header value was identified in the request URL. These headers typically contain +usernames and passwords or JWT tokens. These values should never be sent in GET requests as they +maybe captured by proxy systems, stored in browser history, or stored in log files. If an attacker +were to get access to these logs or logging systems, they would be able to gain access to the +target account. + +## Remediation + +Authorization header details should never be sent in GET requests. When transmitting sensitive information +such as JWT tokens, always use POST requests or headers to transmit the sensitive data. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 598.3 | true | 598 | Passive | Medium | + +## Links + +- [OWASP](https://owasp.org/www-community/vulnerabilities/Information_exposure_through_query_strings_in_url) +- [CWE](https://cwe.mitre.org/data/definitions/598.html) diff --git a/doc/user/application_security/dast/checks/829.1.md b/doc/user/application_security/dast/checks/829.1.md new file mode 100644 index 00000000000..ca3d99c2bc9 --- /dev/null +++ b/doc/user/application_security/dast/checks/829.1.md @@ -0,0 +1,48 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Inclusion of Functionality from Untrusted Control Sphere + +## Description + +JavaScript or CSS source files are included from third party domains without +[Sub-Resource Integrity (SRI)](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity). +If an attacker were to compromise the sites hosting these third party resources, they could inject malicious +script or CSS data in an attempt to compromise users of your application. However, if SRI was applied and an +attacker attempted to modify the contents of the script, the browser would not load the script and your +applications users would be protected from the malicious alterations. + +## Remediation + +All identified resources should be sourced from the same domain as the target application. If this is not +possible, it is strongly recommended that all `script` tags that implement `src` values, or `link` tags +that implement the `href` values include Sub-Resource Integrity. To generate SRI integrity values the +[srihash](https://www.srihash.org/) tool can be used, or by running one of the following commands: + +- `cat FILENAME.js | openssl dgst -sha384 -binary | openssl base64 -A` +- `shasum -b -a 384 FILENAME.js | awk '{ print $1 }' | xxd -r -p | base64` + +The output of these tools must be added as additional attributes, in particular: `integrity` and either +`crossorigin=anonymous` or `crossorigin=use-credentials`. +An example of a valid SRI protected script tag can be found below: + +```html +<script src="https://example.com/example-framework.js" + integrity="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC" + crossorigin="anonymous"></script> +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 829.1 | true | 829 | Passive | Low | + +## Links + +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Javascript_Management_Cheat_Sheet.html#subresource-integrity) +- [CWE](https://cwe.mitre.org/data/definitions/829.html) +- [MDN](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) diff --git a/doc/user/application_security/dast/checks/829.2.md b/doc/user/application_security/dast/checks/829.2.md new file mode 100644 index 00000000000..e6fada117f8 --- /dev/null +++ b/doc/user/application_security/dast/checks/829.2.md @@ -0,0 +1,47 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Invalid Sub-Resource Integrity values detected + +## Description + +JavaScript or CSS source files were found to contain invalid +[Sub-Resource Integrity (SRI)](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) +`integrity` values or a missing `crossorigin` value. These scripts or links should be investigated to +ensure they have not been maliciously altered. If in doubt, contact the owner of the scripts or replace +them with known good versions. + +## Remediation + +All identified resources should be sourced from the same domain as the target application. If this is not +possible, it is strongly recommended that all `script` tags that implement `src` values, or `link` tags +that implement the `href` values include Sub-Resource Integrity. To generate SRI integrity values the +[srihash](https://www.srihash.org/) tool can be used, or by running one of the following commands: + +- `cat FILENAME.js | openssl dgst -sha384 -binary | openssl base64 -A` +- `shasum -b -a 384 FILENAME.js | awk '{ print $1 }' | xxd -r -p | base64` + +The output of these tools must be added as additional attributes, in particular: `integrity` and either +`crossorigin=anonymous` or `crossorigin=use-credentials`. +An example of a valid SRI protected script tag can be found below: + +```html +<script src="https://example.com/example-framework.js" + integrity="sha384-oqVuAfXRKap7fdgcCY5uykM6+R9GqQ8K/uxy9rx7HNQlGYl1kPzQho1wx4JwY8wC" + crossorigin="anonymous"></script> +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 829.2 | true | 829 | Passive | Medium | + +## Links + +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Javascript_Management_Cheat_Sheet.html#subresource-integrity) +- [CWE](https://cwe.mitre.org/data/definitions/829.html) +- [MDN](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index 435bc28c4aa..764e3c4a839 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -20,5 +20,9 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [200.1](200.1.md) | Exposure of sensitive information to an unauthorized actor (private IP address) | Low | Passive | | [548.1](548.1.md) | Exposure of information through directory listing | Low | Passive | | [598.1](598.1.md) | Use of GET request method with sensitive query strings (session ID) | Medium | Passive | +| [598.2](598.2.md) | Use of GET request method with sensitive query strings (password) | Medium | Passive | +| [598.3](598.3.md) | Use of GET request method with sensitive query strings (Authorization header details) | Medium | Passive | | [614.1](614.1.md) | Sensitive cookie without Secure attribute | Low | Passive | | [693.1](693.1.md) | Missing X-Content-Type-Options: nosniff | Low | Passive | +| [829.1](829.1.md) | Inclusion of Functionality from Untrusted Control Sphere | Low | Passive | +| [829.2](829.2.md) | Invalid Sub-Resource Integrity values detected | Medium | Passive | diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index fd6c39ffbf1..ee57803dfc7 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -278,7 +278,8 @@ page. You can enable or configure DAST settings using the UI. The generated settings are formatted so they can be conveniently pasted into the `.gitlab-ci.yml` file. -1. From the project's home page, go to **Security & Compliance > Configuration**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. 1. In the **Dynamic Application Security Testing (DAST)** section, select **Enable DAST** or **Configure DAST**. 1. Select the desired **Scanner profile**, or select **Create scanner profile** and save a @@ -288,12 +289,14 @@ can be conveniently pasted into the `.gitlab-ci.yml` file. 1. Select **Generate code snippet**. A modal opens with the YAML snippet corresponding to the options you selected. 1. Do one of the following: - 1. Select **Copy code only** to copy the snippet to your clipboard. - 1. Select **Copy code and open `.gitlab-ci.yml` file** to copy the snippet to your clipboard. The - CI/CD Editor then opens. + 1. To copy the snippet to your clipboard, select **Copy code only**. + 1. To add the snippet to your project's `.gitlab-ci.yml` file, select + **Copy code and open `.gitlab-ci.yml` file**. The Pipeline Editor opens. 1. Paste the snippet into the `.gitlab-ci.yml` file. 1. Select the **Lint** tab to confirm the edited `.gitlab-ci.yml` file is valid. - 1. Select **Commit changes**. + 1. Select the **Edit** tab, then select **Commit changes**. + +When the snippet is committed to the `.gitlab-ci.yml` file, pipelines include a DAST job. #### Crawling web applications dependent on JavaScript @@ -1053,7 +1056,7 @@ To run an on-demand scan either at a scheduled date or frequency, read 1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left sidebar. -1. Select **New DAST scan**. +1. Select **New scan**. 1. Complete the **Scan name** and **Description** fields. 1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown. 1. In **Scanner profile**, select a scanner profile from the dropdown. @@ -1088,7 +1091,7 @@ To schedule a scan: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > On-demand Scans**. -1. Select **New DAST scan**. +1. Select **New scan**. 1. Complete the **Scan name** and **Description** text boxes. 1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. 1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 839833d9d98..a4908204b60 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -7,73 +7,65 @@ type: reference, howto # DAST API **(ULTIMATE)** -You can add dynamic application security testing of web APIs to your [GitLab CI/CD](../../../ci/index.md) pipelines. -This helps you discover bugs and potential security issues that other QA processes may miss. +You can add dynamic application security testing (DAST) of web APIs to your +[GitLab CI/CD](../../../ci/index.md) pipelines. This helps you discover bugs and potential security +issues that other QA processes may miss. We recommend that you use DAST API testing in addition to [GitLab Secure](../index.md)'s other security scanners and your own test processes. If you're using [GitLab CI/CD](../../../ci/index.md), you can run DAST API tests as part your CI/CD workflow. -## Requirements - -- One of the following web API types: - - REST API - - SOAP - - GraphQL - - Form bodies, JSON, or XML -- One of the following assets to provide APIs to test: - - OpenAPI v2 or v3 API definition - - Postman Collection v2.0 or v2.1 - - HTTP Archive (HAR) of API requests to test +WARNING: +Do not run DAST API testing against a production server. Not only can it perform *any* function that +the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting +data. Only run DAST API against a test server. -## When DAST API scans run +You can run DAST API scanning against the following web API types: -When using the `DAST-API.gitlab-ci.yml` template, the defined jobs use the `dast` stage by default. To enable your `.gitlab-ci.yml` file must include the `dast` stage in your `stages` definition. To ensure DAST API scans the latest code, your CI pipeline should deploy changes to a test environment in a stage before the `dast` stage: +- REST API +- SOAP +- GraphQL +- Form bodies, JSON, or XML -```yaml -stages: - - build - - test - - deploy - - dast -``` +## When DAST API scans run -Note that if your pipeline is configured to deploy to the same web server on each run, running a -pipeline while another is still running could cause a race condition in which one pipeline -overwrites the code from another. The API to scan should be excluded from changes for the duration -of a DAST API scan. The only changes to the API should be from the DAST API scanner. Be aware that -any changes made to the API (for example, by users, scheduled tasks, database changes, code -changes, other pipelines, or other scanners) during a scan could cause inaccurate results. +DAST API scanning runs in the `dast` stage by default. To ensure DAST API scanning examines the latest +code, ensure your CI/CD pipeline deploys changes to a test environment in a stage before the `dast` +stage. -## Enable DAST API scanning +If your pipeline is configured to deploy to the same web server on each run, running a pipeline +while another is still running could cause a race condition in which one pipeline overwrites the +code from another. The API to be scanned should be excluded from changes for the duration of a +DAST API scan. The only changes to the API should be from the DAST API scanner. Changes made to the +API (for example, by users, scheduled tasks, database changes, code changes, other pipelines, or +other scanners) during a scan could cause inaccurate results. -There are three ways to perform scans. See the configuration section for the one you wish to use: +## Example DAST API scanning configurations -- [OpenAPI v2 or v3 specification](#openapi-specification) -- [HTTP Archive (HAR)](#http-archive-har) -- [Postman Collection v2.0 or v2.1](#postman-collection) +The following projects demonstrate DAST API scanning: -Examples of various configurations can be found here: - -- [Example OpenAPI v2 specification project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/openapi-example) +- [Example OpenAPI v2 Specification project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/openapi-example) - [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/har-example) - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/postman-example) - [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/graphql-example) - [Example SOAP project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/soap-example) -WARNING: -GitLab 14.0 will require that you place DAST API configuration files (for example, -`gitlab-dast-api-config.yml`) in your repository's `.gitlab` directory instead of your -repository's root. You can continue using your existing configuration files as they are, but -starting in GitLab 14.0, GitLab will not check your repository's root for configuration files. +## Targeting API for DAST scanning + +You can specify the API you want to scan by using: + +- [OpenAPI v2 or v3 Specification](#openapi-specification) +- [HTTP Archive (HAR)](#http-archive-har) +- [Postman Collection v2.0 or v2.1](#postman-collection) ### OpenAPI Specification > - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. > - Support to generate media type `application/xml` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.8. +> - Support to media types was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. -This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. +This section shows you how to configure DAST API scanning using an OpenAPI Specification to provide information about the target API to test. OpenAPI Specifications are provided as a file system resource or URL. Both JSON and YAML OpenAPI formats are supported. DAST API uses an OpenAPI document to generate the request body. When a request body is required, @@ -84,52 +76,40 @@ the body generation is limited to these body types: - `application/json` - `application/xml` -Follow these steps to configure DAST API in GitLab with an OpenAPI specification: +### OpenAPI and media types -1. To use DAST API, you must [include](../../../ci/yaml/index.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) - that's provided as part of your GitLab installation. Add the following to your - `.gitlab-ci.yml` file: +A media type (formerly known as MIME type) is an identifier for file formats and format contents transmitted. A OpenAPI document lets you specify that a given operation can accept different media types, hence a given request can send data using different file content. As for example, a `PUT /user` operation to update user data could accept data in either XML (media type `application/xml`) or JSON (media type `application/json`) format. +OpenAPI 2.x lets you specify the accepted media types globally or per operation, and OpenAPI 3.x lets you specify the accepted media types per operation. DAST API will check the listed media types, and try to produce sample data for each supported media type. - ```yaml - stages: - - dast +- In [GitLab 14.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/333304), the default behavior is to select one of the supported media types to use. The first supported media type is chosen from the list. This behavior is configurable. +- In GitLab 14.9 and earlier, the default behavior is to perform testing using all supported media types. This means if two media types are listed (for example, `application/json` and `application/xml`), testing are performed using JSON, and then the same tests using XML. - include: - - template: DAST-API.gitlab-ci.yml - ``` +Testing the same operation (for example, `POST /user`) using different media types (for example, `application/json` and `application/xml`) is not always desirable. +For example, if the target application executes the same code regardless of the request content type, it will take longer to finish the test session, and it may report duplicated vulnerabilities related to the request body depending on the target app. -1. The [configuration file](#configuration-files) has several testing profiles defined with different checks enabled. We recommend that you start with the `Quick` profile. - Testing with this profile completes faster, allowing for easier configuration validation. +The environment variable `DAST_API_OPENAPI_ALL_MEDIA_TYPES` lets you specify whether or not to use all supported media types instead of one when generating requests for a given operation. When the environment variable `DAST_API_OPENAPI_ALL_MEDIA_TYPES` is set to any value, DAST API tries to generate requests for all supported media types instead of one in a given operation. This will cause testing to take longer as testing is repeated for each provided media type. - Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick` for the profile you choose: +Alternatively, the variable `DAST_API_OPENAPI_MEDIA_TYPES` is used to provide a list of media types that will each be tested. Providing more than one media type causes testing to take longer, as testing is performed for each media type selected. When the environment variable `DAST_API_OPENAPI_MEDIA_TYPES` is set to a list of media types, only the listed media types are included when creating requests. - ```yaml - stages: - - dast +Multiple media types in `DAST_API_OPENAPI_MEDIA_TYPES` are separated by a colon (`:`). For example, to limit request generation to the media types `application/x-www-form-urlencoded` and `multipart/form-data`, set the environment variable `DAST_API_OPENAPI_MEDIA_TYPES` to `application/x-www-form-urlencoded:multipart/form-data`. Only supported media types in this list are included when creating requests, though non-supported media types are always skipped. A media type text may contain different sections. For example, `application/vnd.api+json; charset=UTF-8`, is a compound of `type "/" [tree "."] subtype ["+" suffix]* [";" parameter]`. Parameters are not taken into account when performing the filtering media types on request generation. - include: - - template: DAST-API.gitlab-ci.yml +The environment variables `DAST_API_OPENAPI_ALL_MEDIA_TYPES` and `DAST_API_OPENAPI_MEDIA_TYPES` allow you to decide how to handle media types. These settings are mutually exclusive. If both are enabled, DAST API reports an error. - variables: - DAST_API_PROFILE: Quick - ``` +#### Configure DAST API with an OpenAPI Specification -1. Provide the location of the OpenAPI specification. You can provide the specification as a file - or URL. Specify the location by adding the `DAST_API_OPENAPI` variable: +To configure DAST API scanning with an OpenAPI specification: - ```yaml - stages: - - dast +To configure DAST API scanning with an OpenAPI Specification: - include: - - template: DAST-API.gitlab-ci.yml +1. [Include](../../../ci/yaml/index.md#includetemplate) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) in your `.gitlab-ci.yml` file. - variables: - DAST_API_PROFILE: Quick - DAST_API_OPENAPI: test-api-specification.json - ``` +1. The [configuration file](#configuration-files) has several testing profiles defined with different checks enabled. We recommend that you start with the `Quick` profile. + Testing with this profile completes faster, allowing for easier configuration validation. + Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. + +1. Provide the location of the OpenAPI Specification as either a file or URL. + Specify the location by adding the `DAST_API_OPENAPI` variable. 1. The target API instance's base URL is also required. Provide it by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. @@ -140,20 +120,20 @@ Follow these steps to configure DAST API in GitLab with an OpenAPI specification automatically parses that file to find its scan target. You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `DAST_API_TARGET_URL`: +Complete example configuration of using an OpenAPI Specification: - ```yaml - stages: - - dast +```yaml +stages: + - dast - include: - - template: DAST-API.gitlab-ci.yml +include: + - template: DAST-API.gitlab-ci.yml - variables: - DAST_API_PROFILE: Quick - DAST_API_OPENAPI: test-api-specification.json - DAST_API_TARGET_URL: http://test-deployment/ - ``` +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ +``` This is a minimal configuration for DAST API. From here you can: @@ -161,14 +141,12 @@ This is a minimal configuration for DAST API. From here you can: - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run DAST API testing against a production server. Not only can it perform *any* function that the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting data. Only run DAST API scanning against a test server. - ### HTTP Archive (HAR) -The [HTTP Archive format (HAR)](http://www.softwareishard.com/blog/har-12-spec/) -is an archive file format for logging HTTP transactions. When used with the GitLab DAST API scanner, HAR must contain records of calling the web API to test. The DAST API scanner extracts all the requests and -uses them to perform testing. +The [HTTP Archive format (HAR)](../api_fuzzing/create_har_files.md) is an archive file format for +logging HTTP transactions. When used with the GitLab DAST API scanner, the HAR file must contain +records of calling the web API to test. The DAST API scanner extracts all of the requests and uses them +to perform testing. You can use various tools to generate HAR files: @@ -182,53 +160,20 @@ WARNING: HAR files may contain sensitive information such as authentication tokens, API keys, and session cookies. We recommend that you review the HAR file contents before adding them to a repository. -Follow these steps to configure DAST API to use a HAR file that provides information about the -target API to test: +#### DAST API scanning with a HAR file -1. To use DAST API, you must [include](../../../ci/yaml/index.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) - that's provided as part of your GitLab installation. To do so, add the following to your - `.gitlab-ci.yml` file: +To configure DAST API to use a HAR file that provides information about the target API to test: - ```yaml - stages: - - dast - - include: - - template: DAST-API.gitlab-ci.yml - ``` +1. [Include](../../../ci/yaml/index.md#includetemplate) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) in your `.gitlab-ci.yml` file. 1. The [configuration file](#configuration-files) has several testing profiles defined with different checks enabled. We recommend that you start with the `Quick` profile. Testing with this profile completes faster, allowing for easier configuration validation. - Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick` for the profile you choose: - - ```yaml - stages: - - dast - - include: - - template: DAST-API.gitlab-ci.yml - - variables: - DAST_API_PROFILE: Quick - ``` + Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. 1. Provide the location of the HAR file. You can provide the location as a file path - or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `DAST_API_HAR` variable: - - ```yaml - stages: - - dast - - include: - - template: DAST-API.gitlab-ci.yml - - variables: - DAST_API_PROFILE: Quick - DAST_API_HAR: test-api-recording.har - ``` + or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `DAST_API_HAR` variable. 1. The target API instance's base URL is also required. Provide it by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. @@ -239,20 +184,20 @@ target API to test: automatically parses that file to find its scan target. You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `DAST_API_TARGET_URL`: +Complete example configuration of using an HAR file: - ```yaml - stages: - - dast +```yaml +stages: + - dast - include: - - template: DAST-API.gitlab-ci.yml +include: + - template: DAST-API.gitlab-ci.yml - variables: - DAST_API_PROFILE: Quick - DAST_API_HAR: test-api-recording.har - DAST_API_TARGET_URL: http://test-deployment/ - ``` +variables: + DAST_API_PROFILE: Quick + DAST_API_HAR: test-api-recording.har + DAST_API_TARGET_URL: http://test-deployment/ +``` This is a minimal configuration for DAST API. From here you can: @@ -260,11 +205,6 @@ This is a minimal configuration for DAST API. From here you can: - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run DAST API testing against a production server. Not only can it perform *any* function that -the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting -data. Only run DAST API against a test server. - ### Postman Collection The [Postman API Client](https://www.postman.com/product/api-client/) is a popular tool that @@ -282,52 +222,20 @@ Postman Collection files may contain sensitive information such as authenticatio and session cookies. We recommend that you review the Postman Collection file contents before adding them to a repository. -Follow these steps to configure DAST API to use a Postman Collection file that provides -information about the target API to test: - -1. To use DAST API, you must [include](../../../ci/yaml/index.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) - that's provided as part of your GitLab installation. To do so, add the following to your - `.gitlab-ci.yml` file: +#### DAST API scanning with a Postman Collection file - ```yaml - stages: - - dast +To configure DAST API to use a Postman Collection file that provides information about the target +API to test: - include: - - template: DAST-API.gitlab-ci.yml - ``` +1. [Include](../../../ci/yaml/index.md#includetemplate) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). 1. The [configuration file](#configuration-files) has several testing profiles defined with different checks enabled. We recommend that you start with the `Quick` profile. Testing with this profile completes faster, allowing for easier configuration validation. - Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file, - substituting `Quick` for the profile you choose: + Provide the profile by adding the `DAST_API_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. - ```yaml - stages: - - dast - - include: - - template: DAST-API.gitlab-ci.yml - - variables: - DAST_API_PROFILE: Quick - ``` - -1. Provide the location of the Postman Collection file. You can provide the location as a file or URL. Specify the location by adding the `DAST_API_POSTMAN_COLLECTION` variable: - - ```yaml - stages: - - dast - - include: - - template: DAST-API.gitlab-ci.yml - - variables: - DAST_API_PROFILE: Quick - DAST_API_POSTMAN_COLLECTION: postman-collection_serviceA.json - ``` +1. Provide the location of the Postman Collection file as either a file or URL. Specify the location by adding the `DAST_API_POSTMAN_COLLECTION` variable. 1. The target API instance's base URL is also required. Provide it by using the `DAST_API_TARGET_URL` variable or an `environment_url.txt` file. @@ -338,20 +246,20 @@ information about the target API to test: automatically parses that file to find its scan target. You can see an [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). - Here's an example of using `DAST_API_TARGET_URL`: +Complete example configuration of using a Postman collection: - ```yaml - stages: - - dast +```yaml +stages: + - dast - include: - - template: DAST-API.gitlab-ci.yml +include: + - template: DAST-API.gitlab-ci.yml - variables: - DAST_API_PROFILE: Quick - DAST_API_POSTMAN_COLLECTION: postman-collection_serviceA.json - DAST_API_TARGET_URL: http://test-deployment/ - ``` +variables: + DAST_API_PROFILE: Quick + DAST_API_POSTMAN_COLLECTION: postman-collection_serviceA.json + DAST_API_TARGET_URL: http://test-deployment/ +``` This is a minimal configuration for DAST API. From here you can: @@ -359,12 +267,7 @@ This is a minimal configuration for DAST API. From here you can: - [Add authentication](#authentication). - Learn how to [handle false positives](#handling-false-positives). -WARNING: -**NEVER** run DAST API testing against a production server. Not only can it perform *any* function that -the API can, it may also trigger bugs in the API. This includes actions like modifying and deleting -data. Only run DAST API against a test server. - -#### Postman variables +##### Postman variables Postman allows the developer to define placeholders that can be used in different parts of the requests. These placeholders are called variables, as explained in [Using variables](https://learning.postman.com/docs/sending-requests/variables/). @@ -388,11 +291,11 @@ Postman file. For example, Postman does not export environment-scoped variables file. By default, the DAST API scanner uses the Postman file to resolve Postman variable values. If a JSON file -is set in a GitLab CI environment variable `DAST_API_POSTMAN_COLLECTION_VARIABLES`, then the JSON +is set in a GitLab CI/CD environment variable `DAST_API_POSTMAN_COLLECTION_VARIABLES`, then the JSON file takes precedence to get Postman variable values. -Although Postman can export environment variables into a JSON file, the format is not compatible -with the JSON expected by `DAST_API_POSTMAN_COLLECTION_VARIABLES`. +WARNING: +Although Postman can export environment variables into a JSON file, the format is not compatible with the JSON expected by `DAST_API_POSTMAN_COLLECTION_VARIABLES`. Here is an example of using `DAST_API_POSTMAN_COLLECTION_VARIABLES`: @@ -421,12 +324,12 @@ values. For example: } ``` -### Authentication +## Authentication Authentication is handled by providing the authentication token as a header or cookie. You can provide a script that performs an authentication flow or calculates the token. -#### HTTP Basic Authentication +### HTTP Basic Authentication [HTTP basic authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) is an authentication method built in to the HTTP protocol and used in conjunction with @@ -456,23 +359,23 @@ variables: DAST_API_HTTP_PASSWORD: $TEST_API_PASSWORD ``` -#### Bearer Tokens +### Bearer tokens Bearer tokens are used by several different authentication mechanisms, including OAuth2 and JSON Web -Tokens (JWT). Bearer tokens are transmitted using the `Authorization` HTTP header. To use bearer +Tokens (JWT). Bearer tokens are transmitted using the `Authorization` HTTP header. To use Bearer tokens with DAST API, you need one of the following: -- A token that doesn't expire -- A way to generate a token that lasts the length of testing -- A Python script that DAST API can call to generate the token +- A token that doesn't expire. +- A way to generate a token that lasts the length of testing. +- A Python script that DAST API can call to generate the token. -##### Token doesn't expire +#### Token doesn't expire -If the bearer token doesn't expire, use the `DAST_API_OVERRIDES_ENV` variable to provide it. This +If the Bearer token doesn't expire, use the `DAST_API_OVERRIDES_ENV` variable to provide it. This variable's content is a JSON snippet that provides headers and cookies to add to DAST API's outgoing HTTP requests. -Follow these steps to provide the bearer token with `DAST_API_OVERRIDES_ENV`: +Follow these steps to provide the Bearer token with `DAST_API_OVERRIDES_ENV`: 1. [Create a CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables), for example `TEST_API_BEARERAUTH`, with the value @@ -502,9 +405,9 @@ Follow these steps to provide the bearer token with `DAST_API_OVERRIDES_ENV`: 1. To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. -##### Token generated at test runtime +#### Token generated at test runtime -If the bearer token must be generated and doesn't expire during testing, you can provide to DAST API a file containing the token. A prior stage and job, or part of the DAST API job, can +If the Bearer token must be generated and doesn't expire during testing, you can provide DAST API a file that has the token. A prior stage and job, or part of the DAST API job, can generate this file. DAST API expects to receive a JSON file with the following structure: @@ -539,14 +442,14 @@ variables: To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. -##### Token has short expiration +#### Token has short expiration -If the bearer token must be generated and expires prior to the scan's completion, you can provide a +If the Bearer token must be generated and expires prior to the scan's completion, you can provide a program or script for the DAST API scanner to execute on a provided interval. The provided script runs in an Alpine Linux container that has Python 3 and Bash installed. If the Python script requires additional packages, it must detect this and install the packages at runtime. -The script must create a JSON file containing the bearer token in a specific format: +The script must create a JSON file containing the Bearer token in a specific format: ```json { @@ -582,7 +485,7 @@ variables: To validate that authentication is working, run an DAST API test and review the job logs and the test API's application logs. See the [overrides section](#overrides) for more information about override commands. -### Configuration files +## Configuration files To get you started quickly, GitLab provides the configuration file [`gitlab-dast-api-config.yml`](https://gitlab.com/gitlab-org/security-products/analyzers/dast/-/blob/master/config/gitlab-dast-api-config.yml). @@ -590,12 +493,12 @@ This file has several testing profiles that perform various numbers of tests. Th profile increases as the test numbers go up. To use a configuration file, add it to your repository's root as `.gitlab/gitlab-dast-api-config.yml`. -#### Profiles +### Profiles The following profiles are pre-defined in the default configuration file. Profiles can be added, removed, and modified by creating a custom configuration. -##### Quick +#### Quick - Application Information Check - Cleartext Authentication Check @@ -610,7 +513,7 @@ can be added, removed, and modified by creating a custom configuration. - Token Check - XML Injection Check -##### Full +#### Full - Application Information Check - Cleartext AuthenticationCheck @@ -630,17 +533,24 @@ can be added, removed, and modified by creating a custom configuration. - Token Check - XML Injection Check -### Available CI/CD variables +## Available CI/CD variables | CI/CD variable | Description | |------------------------------------------------------|--------------------| -| `DAST_API_VERSION` | Specify DAST API container version. Defaults to `latest`. | +| `SECURE_ANALYZERS_PREFIX` | Specify the Docker registry base address from which to download the analyzer. | +| `DAST_API_VERSION` | Specify DAST API container version. Defaults to `1`. | +| `DAST_API_IMAGE_SUFFIX` | Specify a container image suffix. Defaults to none. | | `DAST_API_TARGET_URL` | Base URL of API testing target. | |[`DAST_API_CONFIG`](#configuration-files) | DAST API configuration file. Defaults to `.gitlab-dast-api.yml`. | |[`DAST_API_PROFILE`](#configuration-files) | Configuration profile to use during testing. Defaults to `Quick`. | |[`DAST_API_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | +|[`DAST_API_EXCLUDE_URLS`](#exclude-urls) | Exclude API URL from testing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357195) in GitLab 14.10. | +|[`DAST_API_EXCLUDE_PARAMETER_ENV`](#exclude-parameters) | JSON string containing excluded parameters. | +|[`DAST_API_EXCLUDE_PARAMETER_FILE`](#exclude-parameters) | Path to a JSON file containing excluded parameters. | |[`DAST_API_OPENAPI`](#openapi-specification) | OpenAPI specification file or URL. | |[`DAST_API_OPENAPI_RELAXED_VALIDATION`](#openapi-specification) | Relax document validation. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345950) in GitLab 14.7. | +|[`DAST_API_OPENAPI_ALL_MEDIA_TYPES`](#openapi-specification) | Use all supported media types instead of one when generating requests. Causes test duration to be longer. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. | +|[`DAST_API_OPENAPI_MEDIA_TYPES`](#openapi-specification) | Colon (`:`) separated media types accepted for testing. Default is disabled. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333304) in GitLab 14.10. | |[`DAST_API_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | |[`DAST_API_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | |[`DAST_API_POSTMAN_COLLECTION_VARIABLES`](#postman-variables) | Path to a JSON file to extract postman variable values. | @@ -656,7 +566,7 @@ can be added, removed, and modified by creating a custom configuration. |`DAST_API_SERVICE_START_TIMEOUT` | How long to wait for target API to become available in seconds. Default is 300 seconds. | |`DAST_API_TIMEOUT` | How long to wait for API responses in seconds. Default is 30 seconds. | -### Overrides +## Overrides DAST API provides a method to add or override specific items in your request, for example: @@ -814,7 +724,7 @@ It is changed to: You can provide this JSON document as a file or environment variable. You may also provide a command to generate the JSON document. The command can run at intervals to support values that expire. -#### Using a file +### Using a file To provide the overrides JSON as a file, the `DAST_API_OVERRIDES_FILE` CI/CD variable is set. The path is relative to the job current working directory. @@ -834,7 +744,7 @@ variables: DAST_API_OVERRIDES_FILE: dast-api-overrides.json ``` -#### Using a CI/CD variable +### Using a CI/CD variable To provide the overrides JSON as a CI/CD variable, use the `DAST_API_OVERRIDES_ENV` variable. This allows you to place the JSON as variables that can be masked and protected. @@ -872,7 +782,7 @@ variables: DAST_API_OVERRIDES_ENV: $SECRET_OVERRIDES ``` -#### Using a command +### Using a command If the value must be generated or regenerated on expiration, you can provide a program or script for the DAST API scanner to execute on a specified interval. The provided command runs in an Alpine Linux @@ -914,7 +824,7 @@ variables: DAST_API_OVERRIDES_INTERVAL: 300 ``` -#### Debugging overrides +### Debugging overrides > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334578) in GitLab 14.8. @@ -965,10 +875,10 @@ def get_auth_response(): # In our example, access token is retrieved from a given endpoint try: - # Performs a http request, response sample: + # Performs a http request, response sample: # { "Token" : "b5638ae7-6e77-4585-b035-7d9de2e3f6b3" } response = get_auth_response() - + # Check that the request is successful. may raise `requests.exceptions.HTTPError` response.raise_for_status() @@ -995,7 +905,7 @@ except Exception as e: logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') raise -# computes object that holds overrides file content. +# computes object that holds overrides file content. # It uses data fetched from request overrides_data = { "headers": { @@ -1070,7 +980,7 @@ variables: In the previous sample, you could use the script `user-pre-scan-set-up.sh` to also install new runtimes or applications that later on you could use in our overrides command. -### Exclude Paths +## Exclude Paths > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. @@ -1088,7 +998,7 @@ To verify the paths are excluded, review the `Tested Operations` and `Excluded O 2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ ``` -#### Examples +### Examples This example excludes the `/auth` resource. This does not exclude child resources (`/auth/child`). @@ -1111,6 +1021,294 @@ variables: DAST_API_EXCLUDE_PATHS=/auth*;/v1/* ``` +### Exclude parameters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292196) in GitLab 14.10. + +While testing an API you may might want to exclude a parameter (query string, header, or body element) from testing. This may be needed because a parameter always causes a failure, slows down testing, or for other reasons. To exclude parameters, you can set one of the following variables: `DAST_API_EXCLUDE_PARAMETER_ENV` or `DAST_API_EXCLUDE_PARAMETER_FILE`. + +The `DAST_API_EXCLUDE_PARAMETER_ENV` allows providing a JSON string containing excluded parameters. This is a good option if the JSON is short and will not often change. Another option is the variable `DAST_API_EXCLUDE_PARAMETER_FILE`. This variable is set to a file path that can be checked into the repository, created by another job as an artifact, or generated at runtime with a pre script using `DAST_API_PRE_SCRIPT`. + +#### Exclude parameters using a JSON document + +The JSON document contains a JSON object, this object uses specific properties to identify which parameter should be excluded. +You can provide the following properties to exclude specific parameters during the scanning process: + +- `headers`: Use this property to exclude specific headers. The property's value is an array of header names to be excluded. Names are case-insensitive. +- `cookies`: Use this property's value to exclude specific cookies. The property's value is an array of cookie names to be excluded. Names are case-sensitive. +- `query`: Use this property to exclude specific fields from the query string. The property's value is an array of field names from the query string to be excluded. Names are case-sensitive. +- `body-form`: Use this property to exclude specific fields from a request that uses the media type `application/x-www-form-urlencoded`. The property's value is an array of the field names from the body to be excluded. Names are case-sensitive. +- `body-json`: Use this property to exclude specific JSON nodes from a request that uses the media type `application/json`. The property's value is an array, each entry of the array is a [JSON Path](https://goessner.net/articles/JsonPath/) expression. +- `body-xml`: Use this property to exclude specific XML nodes from a request that uses media type `application/xml`. The property's value is an array, each entry of the array is a [XPath v2](https://www.w3.org/TR/xpath20/) expression. + +Thus, the following JSON document is an example of the expected structure to exclude parameters. + +```json +{ + "headers": [ + "header1", + "header2" + ], + "cookies": [ + "cookie1", + "cookie2" + ], + "query": [ + "query-string1", + "query-string2" + ], + "body-form": [ + "form-param1", + "form-param2" + ], + "body-json": [ + "json-path-expression-1", + "json-path-expression-2" + ], + "body-xml" : [ + "xpath-expression-1", + "xpath-expression-2" + ] +} +``` + +#### Examples + +##### Excluding a single header + +To exclude the header `Upgrade-Insecure-Requests`, set the `header` property's value to an array with the header name: `[ "Upgrade-Insecure-Requests" ]`. For instance, the JSON document looks like this: + +```json +{ + "headers": [ "Upgrade-Insecure-Requests" ] +} +``` + +Header names are case-insensitive, so the header name `UPGRADE-INSECURE-REQUESTS` is equivalent to `Upgrade-Insecure-Requests`. + +##### Excluding both a header and two cookies + +To exclude the header `Authorization`, and the cookies `PHPSESSID` and `csrftoken`, set the `headers` property's value to an array with header name `[ "Authorization" ]` and the `cookies` property's value to an array with the cookies' names `[ "PHPSESSID", "csrftoken" ]`. For instance, the JSON document looks like this: + +```json +{ + "headers": [ "Authorization" ], + "cookies": [ "PHPSESSID", "csrftoken" ] +} +``` + +##### Excluding a `body-form` parameter + +To exclude the `password` field in a request that uses `application/x-www-form-urlencoded`, set the `body-form` property's value to an array with the field name `[ "password" ]`. For instance, the JSON document looks like this: + +```json +{ + "body-form": [ "password" ] +} +``` + +The exclude parameters uses `body-form` when the request uses a content type `application/x-www-form-urlencoded`. + +##### Excluding a specific JSON nodes using JSON Path + +To exclude the `schema` property in the root object, set the `body-json` property's value to an array with the JSON Path expression `[ "$.schema" ]`. + +The JSON Path expression uses special syntax to identify JSON nodes: `$` refers to the root of the JSON document, `.` refers to the current object (in our case the root object), and the text `schema` refers to a property name. Thus, the JSON path expression `$.schema` refers to a property `schema` in the root object. +For instance, the JSON document looks like this: + +```json +{ + "body-json": [ "$.schema" ] +} +``` + +The exclude parameters uses `body-json` when the request uses a content type `application/json`. Each entry in `body-json` is expected to be a [JSON Path expression](https://goessner.net/articles/JsonPath/). In JSON Path characters like `$`, `*`, `.` among others have special meaning. + +##### Excluding multiple JSON nodes using JSON Path + +To exclude the property `password` on each entry of an array of `users` at the root level, set the `body-json` property's value to an array with the JSON Path expression `[ "$.users[*].paswword" ]`. + +The JSON Path expression starts with `$` to refer to the root node and uses `.` to refer to the current node. Then, it uses `users` to refer to a property and the characters `[` and `]` to enclose the index in the array you want to use, instead of providing a number as an index you use `*` to specify any index. After the index reference, we find `.` which now refers to any given selected index in the array, preceded by a property name `password`. + +For instance, the JSON document looks like this: + +```json +{ + "body-json": [ "$.users[*].paswword" ] +} +``` + +The exclude parameters uses `body-json` when the request uses a content type `application/json`. Each entry in `body-json` is expected to be a [JSON Path expression](https://goessner.net/articles/JsonPath/). In JSON Path characters like `$`, `*`, `.` among others have special meaning. + +##### Excluding a XML attribute + +To exclude an attribute named `isEnabled` located in the root element `credentials`, set the `body-xml` property's value to an array with the XPath expression `[ "/credentials/@isEnabled" ]`. + +The XPath expression `/credentials/@isEnabled`, starts with `/` to indicate the root of the XML document, then it is followed by the word `credentials` which indicates the name of the element to match. It uses a `/` to refer to a node of the previous XML element, and the character `@` to indicate that the name `isEnable` is an attribute. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/@isEnabled" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding a XML text's element + +To exclude the text of the `username` element contained in root node `credentials`, set the `body-xml` property's value to an array with the XPath expression `[/credentials/username/text()" ]`. + +In the XPath expression `/credentials/username/text()`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `username`. Last part has a `/` that refers to the current element, and uses a XPath function called `text()` which identifies the text of the current element. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/username/text()" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding an XML element + +To exclude the element `username` contained in root node `credentials`, set the `body-xml` property's value to an array with the XPath expression `[/credentials/username" ]`. + +In the XPath expression `/credentials/username`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `username`. + +For instance, the JSON document looks like this: + +```json +{ + "body-xml": [ + "/credentials/username" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be a [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +##### Excluding an XML node with namespaces + +To exclude anXML element `login` which is defined in namespace `s`, and contained in `credentials` root node, set the `body-xml` property's value to an array with the XPath expression `[ "/credentials/s:login" ]`. + +In the XPath expression `/credentials/s:login`, the first character `/` refers to the root XML node, and then after it indicates an XML element's name `credentials`. Similarly, the character `/` refers to the current element, followed by a new XML element's name `s:login`. Notice that name contains the character `:`, this character separates the namespace from the node name. + +The namespace name should have been defined in the XML document which is part of the body request. You may check the namespace in the specification document HAR, OpenAPI, or Postman Collection file. + +```json +{ + "body-xml": [ + "/credentials/s:login" + ] +} +``` + +The exclude parameters uses `body-xml` when the request uses a content type `application/xml`. Each entry in `body-xml` is expected to be an [XPath v2 expression](https://www.w3.org/TR/xpath20/). In XPath, expressions characters like `@`, `/`, `:`, `[`, `]` among others have special meanings. + +#### Using a JSON string + +To provide the exclusion JSON document set the variable `DAST_API_EXCLUDE_PARAMETER_ENV` with the JSON string. In the following example, the `.gitlab-ci.yml`, the `DAST_API_EXCLUDE_PARAMETER_ENV` variable is set to a JSON string: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_EXCLUDE_PARAMETER_ENV: '{ "headers": [ "Upgrade-Insecure-Requests" ] }' +``` + +#### Using a file + +To provide the exclusion JSON document set the variable `DAST_API_EXCLUDE_PARAMETER_FILE` with the JSON file path. The file path is relative to the job current working directory. In the following example `.gitlab-ci.yml` content, the `DAST_API_EXCLUDE_PARAMETER_FILE` variable is set to a JSON file path: + +```yaml +stages: + - dast + +include: + - template: DAST-API.gitlab-ci.yml + +variables: + DAST_API_PROFILE: Quick + DAST_API_OPENAPI: test-api-specification.json + DAST_API_TARGET_URL: http://test-deployment/ + DAST_API_EXCLUDE_PARAMETER_FILE: dast-api-exclude-parameters.json +``` + +The `dast-api-exclude-parameters.json` is a JSON document that follows the structure of [exclude parameters document](#exclude-parameters-using-a-json-document). + +### Exclude URLS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357195) in GitLab 14.10. + +As an alternative to excluding by paths, you can filter by any other component in the URL by using the `DAST_API_EXCLUDE_URLS` CI/CD variable. This variable can be set in your `.gitlab-ci.yml` file. The variable can store multiple values, separated by commas (`,`). Each value is a regular expression. Because each entry is a regular expression, an entry like `.*` will exclude all URLs because it is a regular expression that matches everything. + +In your job output you can check if any URLs matched any provided regular expression from `DAST_API_EXCLUDE_URLS`. Matching operations are listed in the **Excluded Operations** section. Operations listed in the **Excluded Operations** should not be listed in the **Tested Operations** section. For example the following portion of a job output: + +```plaintext +2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +``` + +NOTE: +Each value in `DAST_API_EXCLUDE_URLS` is a regular expression. Characters such as `.` , `*` and `$` among many others have special meanings in [regular expressions](https://en.wikipedia.org/wiki/Regular_expression#Standards). + +#### Examples + +##### Excluding a URL and child resources + +The following example excludes the URL `http://target/api/auth` and its child resources. + +```yaml +variables: + DAST_API_EXCLUDE_URLS: http://target/api/auth +``` + +##### Excluding two URLs and allow their child resources + +To exclude the URLs `http://target/api/buy` and `http://target/api/sell` but allowing to scan their child resources, for instance: `http://target/api/buy/toy` or `http://target/api/sell/chair`. You could use the value `http://target/api/buy/$,http://target/api/sell/$`. This value is using two regular expressions, each of them separated by a `,` character. Hence, it contains `http://target/api/buy$` and `http://target/api/sell$`. In each regular expression, the trailing `$` character points out where the matching URL should end. + +```yaml +variables: + DAST_API_EXCLUDE_URLS: http://target/api/buy/$,http://target/api/sell/$ +``` + +##### Excluding two URLs and their child resources + +In order to exclude the URLs: `http://target/api/buy` and `http://target/api/sell`, and their child resources. To provide multiple URLs we use the `,` character as follows: + +```yaml +variables: + DAST_API_EXCLUDE_URLS: http://target/api/buy,http://target/api/sell +``` + +##### Excluding URL using regular expressions + +In order to exclude exactly `https://target/api/v1/user/create` and `https://target/api/v2/user/create` or any other version (`v3`,`v4`, and more). We could use `https://target/api/v.*/user/create$`, in the previous regular expression `.` indicates any character and `*` indicates zero or more times, additionally `$` indicates that the URL should end there. + +```yaml +variables: + DAST_API_EXCLUDE_URLS: https://target/api/v.*/user/create$ +``` + ## Running your first scan When configured correctly, a CI/CD pipeline contains a `dast` stage and an `dast_api` job. The job only fails when an invalid configuration is provided. During normal operation, the job always succeeds even if vulnerabilities are identified during testing. @@ -1165,7 +1363,7 @@ pipelines. For more information, see the [Security Dashboard documentation](../s Once a vulnerability is found, you can interact with it. Read more on how to [address the vulnerabilities](../vulnerabilities/index.md). -## Handling False Positives +### Handling False Positives False positives can be handled in several ways: @@ -1177,7 +1375,7 @@ False positives can be handled in several ways: - Turn off the Check producing the false positive. This prevents the check from generating any vulnerabilities. Example checks are the SQL Injection Check, and JSON Hijacking Check. -### Turn off a Check +#### Turn off a Check Checks perform testing of a specific type and can be turned on and off for specific configuration profiles. The provided [configuration files](#configuration-files) define several profiles that you @@ -1235,7 +1433,7 @@ This results in the following YAML: - Name: XmlInjectionCheck ``` -### Turn off an Assertion for a Check +#### Turn off an Assertion for a Check Assertions detect vulnerabilities in tests produced by checks. Many checks support multiple Assertions such as Log Analysis, Response Analysis, and Status Code. When a vulnerability is found, the Assertion used is provided. To identify which Assertions are on by default, see the Checks default configuration in the configuration file. The section is called `Checks`. @@ -1419,6 +1617,19 @@ API Security can still try to consume an OpenAPI document that does not fully co DAST_API_OPENAPI_RELAXED_VALIDATION: On ``` +### No operation in the OpenAPI document is consuming any supported media type + +API Security uses the specified media types in the OpenAPI document to generate requests. If no request can be created due to the lack of supported media types, then an error will be thrown. + +**Error message** + +- In [GitLab 14.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/333304), `Error, no operation in the OpenApi document is consuming any supported media type. Check 'OpenAPI Specification' to check the supported media types.` + +**Solution** + +1. Review supported media types in the [OpenAPI Specification](#openapi-specification) section. +1. Edit your OpenAPI document, allowing at least a given operation to accept any of the supported media types. Alternatively, a supported media type could be set in the OpenAPI document level and get applied to all operations. This step may require changes in your application to ensure the supported media type is accepted by the application. + ## Get support or request an improvement To get support for your particular problem please use the [getting help channels](https://about.gitlab.com/get-help/). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index a4a7e6703ab..924e3838d91 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -321,13 +321,13 @@ The following package managers use lockfiles that GitLab analyzers are capable o | Package Manager | Supported File Format Versions | Tested Versions | | ------ | ------ | ------ | -| Bundler | N/A | [1.17.3](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/master/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | -| Composer | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/tests/php-composer/-/blob/master/composer.lock) | -| Conan | 0.4 | [1.x](https://gitlab.com/gitlab-org/security-products/tests/c-conan/-/blob/master/conan.lock) | -| Go | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/tests/go-modules/-/blob/master/go.mod) | -| NuGet | v1 | [4.9](https://gitlab.com/gitlab-org/security-products/tests/csharp-nuget-dotnetcore/-/blob/master/src/web.api/packages.lock.json#L2) | -| npm | v1, v2 | [6.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/master/package-lock.json#L4), [7.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/lockfile-v2-FREEZE/package-lock.json#L4) | -| yarn | v1 | [1.x](https://gitlab.com/gitlab-org/security-products/tests/js-yarn/-/blob/master/yarn.lock) | +| Bundler | N/A | [1.17.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/ruby-bundler/main/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | +| Composer | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/php-composer/main/composer.lock) | +| Conan | 0.4 | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/c-conan/main/conan.lock) | +| Go | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/go-modules/main/go.sum) | +| NuGet | v1 | [4.9](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/csharp-nuget-dotnetcore/main/src/web.api/packages.lock.json#L2) | +| npm | v1, v2 | [6.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-npm/main/package-lock.json#L4), [7.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-npm/lockfileVersion2/package-lock.json#L4) | +| yarn | v1 | [1.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/master/qa/fixtures/js-yarn/main/yarn.lock#L2) | #### Obtaining dependency information by running a package manager to generate a parsable file @@ -339,19 +339,19 @@ To support the following package managers, the GitLab analyzers proceed in two s | Package Manager | Pre-installed Versions | Tested Versions | | ------ | ------ | ------ | | Bundler | [2.1.4](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit/-/blob/v2.11.3/Dockerfile#L15)<sup><b><a href="#exported-dependency-information-notes-1">1</a></b></sup> | [1.17.3](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/master/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | -| sbt | [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L330), [1.1.4](https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/-/blob/main/project/build.properties#L1), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L339), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L348), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L357), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L366), [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/.gitlab-ci.yml#L384) | -| Maven | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L3) | [3.6.3](https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/blob/master/pom.xml#L3) | -| Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [7.3.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.26.0/config/.tool-versions#L5)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup> | [5.6.4](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/master/gradle/wrapper/gradle-wrapper.properties#L3), [6.5](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14/gradle/wrapper/gradle-wrapper.properties#L3), [6.7-rc-1](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-15/gradle/wrapper/gradle-wrapper.properties#L3), [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.27.1/.gitlab-ci.yml#L289-297)<sup><b><a href="#exported-dependency-information-notes-3">3</a></b></sup>, [6.9](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-6-9/gradle/wrapper/gradle-wrapper.properties#L3), [7.0-rc-2](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-16/gradle/wrapper/gradle-wrapper.properties#L3), [7.3](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-7-3/gradle/wrapper/gradle-wrapper.properties#L3), [7.3.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.27.1/.gitlab-ci.yml#L299-317)<sup><b><a href="#exported-dependency-information-notes-3">3</a></b></sup> | -| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | [57.5.0](https://gitlab.com/gitlab-org/security-products/tests/python-setuptools/-/blob/main/setup.py) | -| pip | [20.2.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L26) | [20.x](https://gitlab.com/gitlab-org/security-products/tests/python-pip/-/blob/master/requirements.txt) | -| Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/pipfile-lock-FREEZE/Pipfile.lock#L6)<sup><b><a href="#exported-dependency-information-notes-4">4</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/master/Pipfile) | +| sbt | [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.24.6/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L443-447), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L449-453), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L455-459), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L461-465), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L467-471), [1.5.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L473-477), [1.6.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L479-483) | +| Maven | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L95-97) | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L95-97) | +| Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [7.3.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.26.0/config/.tool-versions#L5)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup> | [5.6.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L319-323), [6.7](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L286-288)<sup><b><a href="#exported-dependency-information-notes-3">3</a></b></sup>, [6.9](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L331-335), [7.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.28.1/spec/image_spec.rb#L300-302)<sup><b><a href="#exported-dependency-information-notes-3">3</a></b></sup> | +| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | [57.5.0](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.22.0/spec/image_spec.rb#L224-247) | +| pip | [20.2.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L26) | [20.x](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.22.0/spec/image_spec.rb#L77-91) | +| Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.22.0/spec/image_spec.rb#L168-191)<sup><b><a href="#exported-dependency-information-notes-4">4</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.22.0/spec/image_spec.rb#L143-166) | <!-- markdownlint-disable MD044 --> <ol> <li> <a id="exported-dependency-information-notes-1"></a> <p> - The pre-installed version of <code>Bundler</code> is only used for the <a href="https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit">bundler-audit</a> analyzer, and is not used for <a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">gemnasium</a>. + The pre-installed and tested version of <code>Bundler</code> is only used for the <a href="https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit">bundler-audit</a> analyzer, and is not used for <a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">gemnasium</a>. </p> </li> <li> @@ -508,19 +508,18 @@ always take the latest dependency scanning artifact available. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/282533) in GitLab 14.1. > - [Feature flag sec_dependency_scanning_ui_enable removed](https://gitlab.com/gitlab-org/gitlab/-/issues/326005) in GitLab 14.2. -To enable Dependency Scanning in a project, you can create a merge request -from the Security Configuration page. +To enable Dependency Scanning in a project, you can create a merge request: -1. In the project where you want to enable Dependency Scanning, navigate to - **Security & Compliance > Configuration**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. 1. In the **Dependency Scanning** row, select **Configure with a merge request**. +1. Review and merge the merge request to enable Dependency Scanning. -This automatically creates a merge request with the changes necessary to enable Dependency Scanning -that you can review and merge to complete the configuration. +Pipelines now include a dependency scanning job. ### Customizing the dependency scanning settings -The dependency scanning settings can be changed through [CI/CD variables](#available-cicd-variables) by using the +The Dependency Scanning settings can be changed through [CI/CD variables](#available-cicd-variables) by using the [`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For example: @@ -580,6 +579,7 @@ The following variables allow configuration of global dependency scanning settin | `DS_EXCLUDED_ANALYZERS` | Specify the analyzers (by name) to exclude from Dependency Scanning. For more information, see [Dependency Scanning Analyzers](analyzers.md). | | `DS_DEFAULT_ANALYZERS` | ([**DEPRECATED - use `DS_EXCLUDED_ANALYZERS` instead**](https://gitlab.com/gitlab-org/gitlab/-/issues/287691)) Override the names of the official default images. For more information, see [Dependency Scanning Analyzers](analyzers.md). | | `DS_EXCLUDED_PATHS` | Exclude files and directories from the scan based on the paths. A comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec`). Parent directories also match patterns. Default: `"spec, test, tests, tmp"`. | +| `DS_IMAGE_SUFFIX` | Suffix added to the image name. If set to `-fips`, `FIPS-enabled` images are used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354796) in GitLab 14.10. | | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the official default images (proxy). Read more about [customizing analyzers](analyzers.md). | | `SECURE_LOG_LEVEL` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. Default: `info`. | @@ -660,6 +660,40 @@ you can use the `MAVEN_CLI_OPTS` CI/CD variable. Read more on [how to use private Maven repositories](../index.md#using-private-maven-repositories). +#### FIPS-enabled images + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/354796) in GitLab 14.10. + +GitLab also offers [FIPS-enabled Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) +versions of the Gemnasium images. You can therefore replace standard images with FIPS-enabled images. + +To use FIPS-enabled images, set the `DS_IMAGE_SUFFIX` to `-fips`, +and set `DS_EXCLUDED_ANALYZERS` to `bundler-audit, retire.js` +to exclude the analyzers that don't support FIPS. + +```yaml +variables: + DS_IMAGE_SUFFIX: "-fips" + DS_EXCLUDED_ANALYZERS: "bundler-audit, retire.js" +``` + +If you want to execute `bundler-audit` or `retire.js` in your project pipeline, you can override the +Gemnasium scanning jobs, and set `DS_IMAGE_SUFFIX` to `-fips` only for those jobs. + +```yaml +gemnasium-dependency_scanning: + variables: + DS_IMAGE_SUFFIX: "-fips" + +gemnasium-maven-dependency_scanning: + variables: + DS_IMAGE_SUFFIX: "-fips" + +gemnasium-python-dependency_scanning: + variables: + DS_IMAGE_SUFFIX: "-fips" +``` + ## Interacting with the vulnerabilities Once a vulnerability is found, you can interact with it. Read more on how to @@ -890,7 +924,7 @@ Please check the [Release Process documentation](https://gitlab.com/gitlab-org/s ## Contributing to the vulnerability database You can search the [`gemnasium-db`](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project -to find a vulnerability in the Gemnasium database. +to find a vulnerability in the GitLab Advisory Database. You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md). ## Running dependency scanning in an offline environment @@ -1237,3 +1271,24 @@ analyzers, edit your `gitlab-ci.yml` file and either: For example, currently the `gemnasium-maven-dependency_scanning` job pulls the latest `gemnasium-maven` Docker image because `DS_ANALYZER_IMAGE` is set to `"$SECURE_ANALYZERS_PREFIX/gemnasium-maven:$DS_MAJOR_VERSION"`. + +### Dependency Scanning of setuptools project fails with `use_2to3 is invalid` error + +Support for [2to3](https://docs.python.org/3/library/2to3.html) +was [removed](https://setuptools.pypa.io/en/latest/history.html#v58-0-0) +in `setuptools` version `v58.0.0`. Dependency Scanning (running `python 3.9`) uses `setuptools` +version `58.1.0+`, which doesn't support `2to3`. Therefore, a `setuptools` dependency relying on +`lib2to3` will fail with this message: + +```plaintext +error in <dependency name> setup command: use_2to3 is invalid +``` + +To work around this error, downgrade the analyzer's version of `setuptools` (e.g. `v57.5.0`): + +```yaml +gemnasium-python-dependency_scanning: + before_script: + - pip install setuptools==57.5.0 + image: registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python:2-python-3.9 +``` diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md index b72f54b4493..35968a6361f 100644 --- a/doc/user/application_security/iac_scanning/index.md +++ b/doc/user/application_security/iac_scanning/index.md @@ -41,9 +41,31 @@ GitLab IaC scanning supports a variety of IaC configuration files. Our IaC secur | Google Deployment Manager | [KICS](https://kics.io/) | 14.5 | | Kubernetes | [KICS](https://kics.io/) | 14.5 | | OpenAPI | [KICS](https://kics.io/) | 14.5 | -| Terraform | [KICS](https://kics.io/) | 14.5 | +| Terraform <sup>2</sup> | [KICS](https://kics.io/) | 14.5 | 1. IaC scanning can analyze Azure Resource Manager templates in JSON format. If you write templates in the [Bicep](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/overview) language, you must use [the bicep CLI](https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-cli) to convert your Bicep files into JSON before GitLab IaC scanning can analyze them. +1. Terraform modules in a custom registry are not scanned for vulnerabilities. You can follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/357004) for the proposed feature. + +### Supported distributions + +GitLab scanners are provided with a base alpine image for size and maintainability. + +#### FIPS-enabled images + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. + +GitLab also offers [FIPS-enabled Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) +versions of the images. You can therefore replace standard images with FIPS-enabled +images. To configure the images, set the `SAST_IMAGE_SUFFIX` to `-fips` or modify the +standard tag plus the `-fips` extension. + +```yaml +variables: + SAST_IMAGE_SUFFIX: '-fips' + +include: + - template: Security/SAST-IaC.latest.gitlab-ci.yml +``` ### Making IaC analyzers available to all GitLab tiers @@ -54,13 +76,13 @@ All open source (OSS) analyzers are available with the GitLab Free tier. Future Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: -| Capability | In Free | In Ultimate | -|:---------------------------------------------------------------------------------------|:--------------------|:-------------------| -| [Configure IaC Scanners](#configuration) | **{check-circle}** | **{check-circle}** | -| View [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | -| Presentation of JSON Report in merge request | **{dotted-circle}** | **{check-circle}** | -| [Address vulnerabilities](../../application_security/vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | -| [Access to Security Dashboard](../../application_security/security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | +| Capability | In Free & Premium | In Ultimate | +|:----------------------------------------------------------------|:--------------------|:-------------------| +| [Configure IaC scanner](#configuration) | **{check-circle}** | **{check-circle}** | +| Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | +| See new findings in merge request widget | **{dotted-circle}** | **{check-circle}** | +| [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | ## Contribute your scanner @@ -92,15 +114,14 @@ that you can download and analyze. ### Enable IaC Scanning via an automatic merge request -To enable IaC Scanning in a project, you can create a merge request -from the Security Configuration page: +To enable IaC Scanning in a project, you can create a merge request: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. 1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure with a merge request**. +1. Review and merge the merge request to enable IaC Scanning. -This automatically creates a merge request with the changes necessary to enable IaC Scanning -that you can review and merge to complete the configuration. +Pipelines now include an IaC job. ## Reports JSON format diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index ff548f1d29f..3a6aa8e3485 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -52,7 +52,7 @@ The following vulnerability scanners and their databases are regularly updated: | Secure scanning tool | Vulnerabilities database updates | |:----------------------------------------------------------------|:---------------------------------| | [Container Scanning](container_scanning/index.md) | A job runs on a daily basis to build new images with the latest vulnerability database updates from the upstream scanner. For more details, see [Vulnerabilities database update](container_scanning/index.md#vulnerabilities-database-update). | -| [Dependency Scanning](dependency_scanning/index.md) | Relies on `bundler-audit` (for Ruby gems), `retire.js` (for npm packages), and `gemnasium` (the GitLab tool for all libraries). Both `bundler-audit` and `retire.js` fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated on a daily basis using [data from NVD, the `ruby-advisory-db` and the GitHub Security Advisory Database as data sources](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/SOURCES.md). See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | +| [Dependency Scanning](dependency_scanning/index.md) | Relies on `bundler-audit` (for Ruby gems), `retire.js` (for npm packages), and `gemnasium` (the GitLab tool for all libraries). Both `bundler-audit` and `retire.js` fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [GitLab Advisory Database](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated on a daily basis using [data from NVD, the `ruby-advisory-db` and the GitHub Advisory Database as data sources](https://gitlab.com/gitlab-org/security-products/gemnasium-db/-/blob/master/SOURCES.md). See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | | [Dynamic Application Security Testing (DAST)](dast/index.md) | The scanning engine is updated on a periodic basis. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/main/Dockerfile#L1). The scanning rules are downloaded at scan runtime. | | [Static Application Security Testing (SAST)](sast/index.md) | Relies exclusively on [the tools GitLab wraps](sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | @@ -218,7 +218,7 @@ security issues: WARNING: This feature is in its end-of-life process. It is [deprecated](../../update/deprecations.md#vulnerability-check) -for use in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new +in GitLab 14.8, and is planned for removal in GitLab 15.0. Users should migrate to the new [Security Approval Policies](policies/scan-result-policies.md). To prevent a merge request introducing a security vulnerability in a project, enable the @@ -381,6 +381,12 @@ Learn more on overriding security jobs: All the security scanning tools define their stage, so this error can occur with all of them. +## Self managed installation options + +For self managed installations, you can choose to run most of the GitLab security scanners even when [not connected to the internet](offline_deployments/index.md). + +Self managed installations can also run the security scanners on a GitLab Runner [running inside OpenShift](../../install/openshift_and_gitlab/index.md). + ## Security report validation > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. @@ -474,6 +480,7 @@ GitLab provides two methods of accomplishing this, each with advantages and disa - [Compliance framework pipelines](../project/settings/#compliance-pipeline-configuration) are recommended when: + - Scan execution enforcement is required for SAST or Secret Detection scans that use custom rulesets. - Scan execution enforcement is required for SAST IaC, Dependency Scanning, License Compliance, API Fuzzing, or Coverage-guided Fuzzing. - Scan execution enforcement is required for scanners external to GitLab. @@ -482,9 +489,18 @@ GitLab provides two methods of accomplishing this, each with advantages and disa - [Scan execution policies](policies/scan-execution-policies.md) are recommended when: - - Scan execution enforcement is required for DAST, SAST, Secret Detection, or Container Scanning. + - Scan execution enforcement is required for DAST. + - Scan execution enforcement is required for Container Scanning with project-specific variable + customizations. To accomplish this, users must create a separate security policy per project. - Scans are required to run on a regular, scheduled cadence. +- Either solution can be used equally well when: + + - Scan execution enforcement is required for SAST or Secret Detection when custom rulesets are not + used. + - Scan execution enforcement is required for Container Scanning with no project-specific variable + customizations. + Additional details about the differences between the two solutions are outlined below: | | Compliance Framework Pipelines | Scan Execution Policies | diff --git a/doc/user/application_security/policies/img/association_diagram.png b/doc/user/application_security/policies/img/association_diagram.png Binary files differnew file mode 100644 index 00000000000..d082e297c68 --- /dev/null +++ b/doc/user/application_security/policies/img/association_diagram.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index 8a39220da35..81d24104340 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -21,6 +21,56 @@ GitLab supports the following security policies: - [Scan Result Policy](scan-result-policies.md) - [Container Network Policy](#container-network-policy) (DEPRECATED) +## Security policy project + +All security policies are stored as YAML in a separate security policy project that gets linked to +the development project. This association can be a one-to-many relationship, allowing one security +policy project to apply to multiple development projects. Linked projects are not required to be in +the same group as the development projects to which they are linked. + + + +Although it is possible to have one project linked to itself and to serve as both the development +project and the security policy project, this is not recommended. Keeping the security policy +project separate from the development project allows for complete separation of duties between +security/compliance teams and development teams. + +All security policies are stored in the `.gitlab/security-policies/policy.yml` YAML file inside the +linked security policy project. The format for this YAML is specific to the type of policy that is +stored there. Examples and schema information are available for the following policy types: + +- [Scan execution policy](scan-execution-policies.md#example-security-policies-project) +- [Scan result policy](scan-result-policies.md#example-security-scan-result-policies-project) + +Policies created in this project are applied through a background job that runs once every 10 +minutes. Allow up to 10 minutes for any policy changes committed to this project to take effect. + +### Managing the linked security policy project + +NOTE: +Only project Owners have the [permissions](../../permissions.md#project-members-permissions) +to select, edit, and unlink a security policy project. + +As a project owner, take the following steps to create or edit an association between your current +project and a project that you would like to designate as the security policy project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Policies**. +1. Select **Edit Policy Project**, and search for and select the + project you would like to link from the dropdown menu. +1. Select **Save**. + +To unlink a security policy project, follow the same steps but instead select the trash can icon in +the modal. + + + +### Viewing the linked security policy project + +All users who have access to the project policy page and are not project owners will instead view a +button linking out to the associated security policy project. If no security policy project has been +associated then the linking button does not appear. + ## Policy management The Policies page displays deployed @@ -57,6 +107,7 @@ You can use the policy editor to create, edit, and delete policies: 1. On the top bar, select **Menu > Projects** and find your group. 1. On the left sidebar, select **Security & Compliance > Policies**. - To create a new policy, select **New policy** which is located in the **Policies** page's header. + You can then select which type of policy to create. - To edit an existing policy, select **Edit policy** in the selected policy drawer. The policy editor has two modes: @@ -78,44 +129,12 @@ by the Rule mode, Rule mode is automatically disabled. If the YAML is incorrect, you must use YAML mode to fix your policy before Rule mode is available again. -## Security Policies project - -NOTE: -We recommend using the [Security Policies project](#security-policies-project) -exclusively for managing policies for the project. Do not add your application's source code to such -projects. - -The Security Policies feature is a repository to store policies. All security policies are stored in -the `.gitlab/security-policies/policy.yml` YAML file. The format for this YAML is specific to the type of policy that is being stored there. Examples and schema information are available for the following policy types: - -- [Scan execution policy](scan-execution-policies.md#example-security-policies-project) -- [Scan result policy](scan-result-policies.md#example-security-scan-result-policies-project) - -Policies created in this project are applied through a background job that runs once every 10 -minutes. Allow up to 10 minutes for any policy changes committed to this project to take effect. - -## Security Policy project selection - -NOTE: -Only project Owners have the [permissions](../../permissions.md#project-members-permissions) -to select Security Policy Project. - -When the Security Policy project is created and policies are created within that repository, you -must create an association between that project and the project you want to apply policies to: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **Security & Compliance > Policies**. -1. Select **Edit Policy Project**, and search for and select the - project you would like to link from the dropdown menu. -1. Select **Save**. - -  - -### Unlink Security Policy projects - -Project owners can unlink Security Policy projects from development projects. To do this, follow -the steps described in [Security Policy project selection](#security-policy-project-selection), -but select the trash can icon in the modal. +When you finish creating or editing your policy, save and apply it by selecting the +**Configure with a merge request** button and then merging the resulting merge request. When you +press this button, the policy YAML is validated and any resulting errors are displayed. +Additionally, if you are a project owner and a security policy project has not been previously +associated with this project, then a new project is created and associated automatically at the same +time that the first policy merge request is created. ## Scan execution policies @@ -132,7 +151,7 @@ See [Scan result policies](scan-result-policies.md). WARNING: Container Network Policy is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. The **Container Network Policy** section provides packet flow metrics for diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index c3778ac97de..7e8e60768b9 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -132,8 +132,8 @@ Note the following: ## Example security policies project -You can use this example in a `.gitlab/security-policies/policy.yml`, as described in -[Security policies project](index.md#security-policies-project). +You can use this example in a `.gitlab/security-policies/policy.yml` file stored in a +[security policy project](index.md#security-policy-project): ```yaml --- diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 8215316bcab..d2cce207bfd 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -65,7 +65,7 @@ This rule enforces the defined actions based on the information provided. | `scanners` | `array` of `string` | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. | | `vulnerabilities_allowed` | `integer` | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | | `severity_levels` | `array` of `string` | `info`, `unknown`, `low`, `medium`, `high`, `critical`| The severity levels for this rule to consider. | -| `vulnerability_states` | `array` of `string` | `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed` | The vulnerability states for this rule to consider when the target branch is set to the default branch. | +| `vulnerability_states` | `array` of `string` | `newly_detected`, `detected`, `confirmed`, `resolved`, `dismissed` | The vulnerability states for this rule to consider when the target branch is set to the default branch. The `newly_detected` state considers all newly detected vulnerabilities regardless of their status or dismissal. The other states consider findings that match the selected state and already exist in the default branch. | ## `require_approval` action type @@ -90,8 +90,8 @@ Requirements and limitations: ## Example security scan result policies project -You can use this example in a `.gitlab/security-policies/policy.yml`, as described in -[Security policies project](index.md#security-policies-project): +You can use this example in a `.gitlab/security-policies/policy.yml` file stored in a +[security policy project](index.md#security-policy-project): ```yaml --- diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index d3a79410eea..8f006f258b6 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -81,6 +81,7 @@ You can also [view our language roadmap](https://about.gitlab.com/direction/secu | Go | [Semgrep](https://semgrep.dev) | 14.4 | | Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | | Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | +| Java (any build system) | [Semgrep](https://semgrep.dev) | 14.10 | | Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | | Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | | JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | @@ -132,6 +133,30 @@ The following analyzers have multi-project support: Multi-project support in the Security Code Scan requires a Solution (`.sln`) file in the root of the repository. For details on the Solution format, see the Microsoft reference [Solution (`.sln`) file](https://docs.microsoft.com/en-us/visualstudio/extensibility/internals/solution-dot-sln-file?view=vs-2019). +### Supported distributions + +The default scanner images are build off a base Alpine image for size and maintainability. + +#### FIPS-enabled images + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. + +GitLab offers [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) +versions of the images that are FIPS-enabled. To use the FIPS-enabled images, you can either: + +- Set the `SAST_IMAGE_SUFFIX` to `-fips`. +- Add the `-fips` extension to the default image name. + +For example: + +```yaml +variables: + SAST_IMAGE_SUFFIX: '-fips' + +include: + - template: Security/SAST.gitlab-ci.yml +``` + ### Making SAST analyzers available to all GitLab tiers All open source (OSS) analyzers have been moved to the GitLab Free tier as of GitLab 13.3. @@ -141,17 +166,17 @@ All open source (OSS) analyzers have been moved to the GitLab Free tier as of Gi Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: -| Capability | In Free | In Ultimate | -|:---------------------------------------------------------------------------------------|:--------------------|:-------------------| -| [Configure SAST Scanners](#configuration) | **{check-circle}** | **{check-circle}** | -| [Customize SAST Settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | -| View [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | -| Presentation of JSON Report in Merge Request | **{dotted-circle}** | **{check-circle}** | -| [Address vulnerabilities](../../application_security/vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | -| [Access to Security Dashboard](../../application_security/security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | -| [Configure SAST in the UI](#configure-sast-in-the-ui) | **{dotted-circle}** | **{check-circle}** | -| [Customize SAST Rulesets](#customize-rulesets) | **{dotted-circle}** | **{check-circle}** | -| [False Positive Detection](#false-positive-detection) | **{dotted-circle}** | **{check-circle}** | +| Capability | In Free & Premium | In Ultimate | +|:----------------------------------------------------------------|:--------------------|:-------------------| +| [Configure SAST scanners](#configuration) | **{check-circle}** | **{check-circle}** | +| [Customize SAST settings](#available-cicd-variables) | **{check-circle}** | **{check-circle}** | +| Download [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | +| See new findings in merge request widget | **{dotted-circle}** | **{check-circle}** | +| [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Configure SAST in the UI](#configure-sast-in-the-ui) | **{dotted-circle}** | **{check-circle}** | +| [Customize SAST rulesets](#customize-rulesets) | **{dotted-circle}** | **{check-circle}** | +| [Detect False Positives](#false-positive-detection) | **{dotted-circle}** | **{check-circle}** | ## Contribute your scanner @@ -190,28 +215,28 @@ always take the latest SAST artifact available. ### Configure SAST in the UI You can enable and configure SAST in the UI, either with default settings, or with customizations. -Use the method that best meets your needs. +The method you can use depends on your GitLab license tier. -- [Configure SAST in the UI with default settings](#configure-sast-in-the-ui-with-default-settings) -- [Configure SAST in the UI with customizations](#configure-sast-in-the-ui-with-customizations) +- [Configure SAST in the UI with default settings](#configure-sast-in-the-ui-with-default-settings). +- [Configure SAST in the UI with customizations](#configure-sast-in-the-ui-with-customizations). **(ULTIMATE)** ### Configure SAST in the UI with default settings > [Introduced](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#security-configuration-page-for-all-users) in GitLab 13.9 +NOTE: +The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal +configuration file. If you have a complex GitLab configuration file it may not be parsed +successfully, and an error may occur. + To enable and configure SAST with default settings: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance** > **Configuration**. -1. In the SAST section, select `Enable via MR`. -1. Review the draft MR that enables SAST with the default recommended settings in the - `.gitlab-ci.yml` file. -1. Merge the MR to enable SAST. You should see SAST jobs run in that MR's pipeline. +1. In the SAST section, select **Configure with a merge request**. +1. Review and merge the merge request to enable SAST. -NOTE: -The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal -configuration file. If you have a complex GitLab configuration file it may not be parsed -successfully, and an error may occur. +Pipelines now include a SAST job. ### Configure SAST in the UI with customizations **(ULTIMATE)** @@ -219,27 +244,28 @@ successfully, and an error may occur. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in GitLab 13.4. > - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/3635) in GitLab 13.5. +NOTE: +The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal +configuration file. If you have a complex GitLab configuration file it may not be parsed +successfully, and an error may occur. + To enable and configure SAST with customizations: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Configuration**. -1. If the project does not have a `.gitlab-ci.yml` file, select **Enable** in the Static Application - Security Testing (SAST) row, otherwise select **Configure**. +1. If the project does not have a `.gitlab-ci.yml` file, select **Enable SAST** in the Static + Application Security Testing (SAST) row, otherwise select **Configure SAST**. 1. Enter the custom SAST values. Custom values are stored in the `.gitlab-ci.yml` file. For CI/CD variables not in the SAST - Configuration page, their values are left unchanged. Default values are inherited from the GitLab - SAST template. + Configuration page, their values are inherited from the GitLab SAST template. 1. Optionally, expand the **SAST analyzers** section, select individual [SAST analyzers](analyzers.md) and enter custom analyzer values. 1. Select **Create Merge Request**. 1. Review and merge the merge request. -NOTE: -The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal -configuration file. If you have a complex GitLab configuration file it may not be parsed -successfully, and an error may occur. +Pipelines now include a SAST job. ### Overriding SAST jobs @@ -399,7 +425,7 @@ and `value` of identifiers and then overridden: ``` If a vulnerability is found with a type `CWE` with a value of `703` then -the vulnerability severity is overwritten to `Critical`. +the vulnerability severity is overwritten to `Critical`. #### Synthesize a custom configuration @@ -523,7 +549,7 @@ Several passthrouh types generate a configuration for the target analyzer: the configuration. - If there is a filename collision between files in both repositories, files from the `sast` repository overwrite files from the `myrules` repository, - as `sast-rules` has higher precedence. + as `sast-rules` has higher precedence. - The `raw` entry creates a file named `insecure.yml` under `/sgrules`. The full path is `/sgrules/insecure.yml`. - The `url` entry fetches a configuration made available through a URL and @@ -831,6 +857,7 @@ The following are Docker image-related CI/CD variables. | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the default images (proxy). Read more about [customizing analyzers](analyzers.md). | | `SAST_EXCLUDED_ANALYZERS` | Names of default images that should never run. Read more about [customizing analyzers](analyzers.md). | | `SAST_ANALYZER_IMAGE_TAG` | Override the default version of analyzer image. Read more about [pinning the analyzer image version](#pinning-to-minor-image-version). | +| `SAST_IMAGE_SUFFIX` | Suffix added to the image name. If set to `-fips`, `FIPS-enabled` images are used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355518) in GitLab 14.10. | #### Vulnerability filters @@ -936,7 +963,7 @@ To use SAST in an offline environment, you need: - A Docker Container Registry with locally available copies of SAST [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers) images. - Configure certificate checking of packages (optional). -GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), +GitLab Runner has a [default `pull_policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local copy is available. The GitLab Runner [`pull_policy` can be set to `if-not-present`](https://docs.gitlab.com/runner/executors/docker.html#using-the-if-not-present-pull-policy) in an offline environment if you prefer using only locally available Docker images. However, we @@ -990,7 +1017,7 @@ Support for custom certificate authorities was introduced in the following versi | `phpcs-security-audit` | [v2.8.2](https://gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit/-/releases/v2.8.2) | | `pmd-apex` | [v2.1.0](https://gitlab.com/gitlab-org/security-products/analyzers/pmd-apex/-/releases/v2.1.0) | | `security-code-scan` | [v2.7.3](https://gitlab.com/gitlab-org/security-products/analyzers/security-code-scan/-/releases/v2.7.3) | -| `semgrep` | [v0.0.1](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/releases/v0.0.1) | +| `semgrep` | [v0.0.1](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep/-/releases/v0.0.1) | | `sobelow` | [v2.2.0](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow/-/releases/v2.2.0) | | `spotbugs` | [v2.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs/-/releases/v2.7.1) | diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 582497eb465..0a18e7d5f45 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -60,13 +60,14 @@ as shown in the following table: | Capability | In Free & Premium | In Ultimate | |:----------------------------------------------------------------|:--------------------|:-------------------| -| [Configure Secret Detection Scanners](#configuration) | **{check-circle}** | **{check-circle}** | -| [Customize Secret Detection Settings](#customizing-settings) | **{check-circle}** | **{check-circle}** | -| View [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** | **{check-circle}** | -| Presentation of JSON Report in merge request | **{dotted-circle}** | **{check-circle}** | +| [Configure Secret Detection scanner](#configuration) | **{check-circle}** | **{check-circle}** | +| [Customize Secret Detection settings](#customizing-settings) | **{check-circle}** | **{check-circle}** | +| Download [JSON Report](../sast/index.md#reports-json-format) | **{check-circle}** | **{check-circle}** | +| See new findings in the merge request widget | **{dotted-circle}** | **{check-circle}** | | View identified secrets in the pipelines' **Security** tab | **{dotted-circle}** | **{check-circle}** | -| [Interaction with Vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | -| [Access to Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Manage vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Access the Security Dashboard](../security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Customize Secret Detection rulesets](#custom-rulesets) | **{dotted-circle}** | **{check-circle}** | ## Configuration @@ -107,25 +108,48 @@ The results are saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. +### Supported distributions + +The default scanner images are build off a base Alpine image for size and maintainability. + +#### FIPS-enabled images + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. + +GitLab offers [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) +versions of the images that are FIPS-enabled. To use the FIPS-enabled images, you can either: + +- Set the `SAST_IMAGE_SUFFIX` to `-fips`. +- Add the `-fips` extension to the default image name. + +For example: + +```yaml +variables: + SECRET_DETECTION_IMAGE_SUFFIX: '-fips' + +include: + - template: Security/Secret-Detection.gitlab-ci.yml +``` + ### Enable Secret Detection via an automatic merge request > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4496) in GitLab 13.11, deployed behind a feature flag, enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/329886) in GitLab 14.1. -To enable Secret Detection in a project, you can create a merge request -from the Security Configuration page. +NOTE: +This method works best with no existing `.gitlab-ci.yml` file, or with a minimal configuration +file. If you have a complex GitLab configuration file it may not be parsed successfully, and an +error may occur. -1. In the project where you want to enable Secret Detection, go to - **Security & Compliance > Configuration**. -1. In the **Secret Detection** row, select **Configure with a merge request**. +To enable Secret Detection in a project, you can create a merge request: -This automatically creates a merge request with the changes necessary to enable Secret Detection -that you can review and merge to complete the configuration. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Secret Detection** row, select **Configure with a merge request**. +1. Review and merge the merge request to enable Secret Detection. -NOTE: -The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal -configuration file. If you have a complex GitLab configuration file it may not be parsed -successfully, and an error may occur. +Pipelines now include a Secret Detection job. ### Customizing settings @@ -176,6 +200,7 @@ Secret Detection can be customized by defining available CI/CD variables: | `SECRET_DETECTION_COMMITS` | - | The list of commits that Gitleaks should scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/243564) in GitLab 13.5. | | `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | | `SECRET_DETECTION_HISTORIC_SCAN` | false | Flag to enable a historic Gitleaks scan. | +| `SECRET_DETECTION_IMAGE_SUFFIX` | Suffix added to the image name. If set to `-fips`, `FIPS-enabled` images are used for scan. See [FIPS-enabled images](#fips-enabled-images) for more details. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355519) in GitLab 14.10. | ### Custom rulesets **(ULTIMATE)** diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index a0cfd6d9d77..488ec336646 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -52,6 +52,9 @@ To view vulnerabilities in a pipeline: 1. From the list, select the pipeline you want to check for vulnerabilities. 1. Select the **Security** tab. +**Scan details** shows vulnerabilities introduced by the merge request, in addition to existing vulnerabilities +from the latest successful pipeline in your project's default branch. + A pipeline consists of multiple jobs, such as SAST and DAST scans. If a job fails to finish, the security dashboard doesn't show SAST scanner output. For example, if the SAST job finishes but the DAST job fails, the security dashboard doesn't show SAST results. On failure, @@ -66,7 +69,8 @@ To view the total number of vulnerabilities per scan: 1. Select the **Status** of a branch. 1. Select the **Security** tab. -**Scan details** show the total number of vulnerabilities found per scan in the pipeline. +**Scan details** shows vulnerabilities introduced by the merge request, in addition to existing vulnerabilities +from the latest successful pipeline in your project's default branch. ### Download security scan outputs diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index 390882d4326..9b8dd2825ea 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Threat Monitoring is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. The **Threat Monitoring** page provides alerts and metrics diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 0b27760b4bb..18b99e06299 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -28,7 +28,7 @@ On the vulnerability's page, you can: - [Create an issue](#create-an-issue-for-a-vulnerability). - [Link issues to the vulnerability](#linked-issues). - [Resolve a vulnerability](#resolve-a-vulnerability) if a solution is - available. + available. - [View security training specific to the detected vulnerability](#view-security-training-for-a-vulnerability). ## Vulnerability status values diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index eb59c289700..a9cef15e3e8 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -222,12 +222,8 @@ To undo this action, select a different status from the same menu. ## Manually add a vulnerability finding -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301003) in GitLab 14.9. Disabled by default. - -FLAG: -This feature is not enabled by default. To make it available, ask an administrator to -[enable the feature flag](../../feature_flags.md) named `new_vulnerability_form`. -On GitLab.com, this feature is not yet available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/301003) in GitLab 14.9. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/353796) in GitLab 14.10. To add a new vulnerability finding from your project level Vulnerability Report page: @@ -236,7 +232,7 @@ To add a new vulnerability finding from your project level Vulnerability Report 1. Click on **Submit Vulnerability**. 1. Complete the fields and submit the form. -You will be brought to the newly created vulnerability's detail page. Manually created records appear in the +You will be brought to the newly created vulnerability's detail page. Manually created records appear in the Group, Project, and Security Center Vulnerability Reports. To filter them, use the Generic Tool filter. ## Operational vulnerabilities diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md index 73a8470e025..c15041f6b0d 100644 --- a/doc/user/clusters/agent/ci_cd_tunnel.md +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -8,6 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327409) in GitLab 14.1. > - The pre-configured `KUBECONFIG` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324275) in GitLab 14.2. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) the `ci_access` attribute in GitLab 14.3. > - The ability to authorize groups was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. > - Support for Omnibus installations was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5686) in GitLab 14.5. @@ -69,6 +70,7 @@ To authorize the agent to access the GitLab project where you keep Kubernetes ma ``` - The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is. + - You can install additional agents into the same cluster to accommodate additional hierarchies. - You can authorize up to 100 projects. All CI/CD jobs now include a `KUBECONFIG` with contexts for every shared agent connection. @@ -91,9 +93,11 @@ To authorize the agent to access all of the GitLab projects in a group or subgro ``` - The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is. + - You can install additional agents into the same cluster to accommodate additional hierarchies. + - All of the subgroups of an authorized group also have access to the same agent (without being specified individually). - You can authorize up to 100 groups. -All the projects that belong to the group are now authorized to access the agent. +All the projects that belong to the group and its subgroups are now authorized to access the agent. All CI/CD jobs now include a `KUBECONFIG` with contexts for every shared agent connection. Choose the context to run `kubectl` commands from your CI/CD scripts. @@ -123,7 +127,7 @@ Run `kubectl config get-contexts`. When you deploy to an environment that has both a [certificate-based cluster](../../infrastructure/clusters/index.md) (deprecated) and an agent connection: -- The certificate-based cluster's context is called `gitlab-deploy`. This context +- The certificate-based cluster's context is called `gitlab-deploy`. This context is always selected by default. - In GitLab 14.9 and later, agent contexts are included in the `KUBECONFIG`. You can select them by using `kubectl config use-context diff --git a/doc/user/clusters/agent/gitops.md b/doc/user/clusters/agent/gitops.md index 8f0e2255121..e99e3b00ec7 100644 --- a/doc/user/clusters/agent/gitops.md +++ b/doc/user/clusters/agent/gitops.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Using a GitOps workflow for Kubernetes **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. With GitOps, you can manage containerized clusters and applications from a Git repository that: @@ -18,6 +19,8 @@ By combining GitLab, Kubernetes, and GitOps, you can have: - GitLab as the GitOps operator. - Kubernetes as the automation and convergence system. - GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment. +- Built-in automatic drift remediation. +- Resource management with [server-side applies](https://kubernetes.io/docs/reference/using-api/server-side-apply/) for transparent multi-actor field management. This diagram shows the repositories and main actors in a GitOps deployment: @@ -90,6 +93,37 @@ gitops: | `prune_propagation_policy` | The deletion propagation policy that [should be used for pruning](https://github.com/kubernetes/apimachinery/blob/44113beed5d39f1b261a12ec398a356e02358307/pkg/apis/meta/v1/types.go#L456-L470). Can be: `orphan`, `background`, or `foreground`. Default is `foreground`. | | `inventory_policy` | Determines whether an inventory object can take over objects that belong to another inventory object or don't belong to any inventory object. This is done by determining if the apply/prune operation can go through for a resource based on comparison of the `inventory-id` value in the package and the `owning-inventory` annotation (`config.k8s.io/owning-inventory`) [in the live object](https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/inventory/policy.go#L12-L66). Can be: `must_match`, `adopt_if_no_inventory`, or `adopt_all`. Default is `must_match`. | +## GitOps annotations + +The GitLab agent for Kubernetes has annotations you can use to: + +- **Sort resources**: Apply or delete resources in a specific order. +- **Use apply-time mutation**: Dynamically substitute fields from one resource configuration to another. + +The agent has [default sorting](https://github.com/kubernetes-sigs/cli-utils/blob/d7d63f4b62897f584ca9e02b6faf4d2f327a9b09/pkg/ordering/sort.go#L74), +but with annotations, you can fine-tune the order and apply time-value injection. + +To provide the GitOps functionality, the GitLab agent for Kubernetes uses the [`cli-utils` library](https://github.com/kubernetes-sigs/cli-utils/), +a Kubernetes SIG project. You can read more about the available annotations in the [`cli-utils` documentation](https://github.com/kubernetes-sigs/cli-utils/blob/master/README.md#apply-sort-ordering). + +- [Learn more about apply sort ordering](https://github.com/kubernetes-sigs/cli-utils#apply-sort-ordering). +- [Learn more about apply-time mutation](https://github.com/kubernetes-sigs/cli-utils#apply-time-mutation). + +## Automatic drift remediation + +Drift happens when the current configuration of an infrastructure resource differs from its expected configuration. +Typically, this is caused by manually editing resources directly through the service that created the resource. Minimizing the +risk of drift helps to ensure configuration consistency and successful operations. + +In GitLab, the agent for Kubernetes regularly compares the expected state from the `git` repository with +the known state from the `cluster`. Deviations from the `git` state are fixed at every check. These checks +happen automatically every 5 minutes. They are not configurable. + +The agent uses [server-side applies](https://kubernetes.io/docs/reference/using-api/server-side-apply/). +As a result, every field in a resource can have different managers. Only fields managed by `git` +are checked for drift. This facilitates the use of in-cluster controllers to modify resources like +[Horizontal Pod Autoscalers](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). + ## Additional resources The following documentation and examples can help you get started with a GitOps workflow. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index a8ad19df2e1..bab3f3137fe 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -9,12 +9,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in GitLab 13.4. > - Support for `grpcs` [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6. > - Agent server [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program in GitLab 13.10. -> - The agent became available to every project on GitLab.com in GitLab 13.11. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the GitLab agent became available on GitLab.com. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. > - [Renamed](https://gitlab.com/groups/gitlab-org/-/epics/7167) from "GitLab Kubernetes Agent" to "GitLab agent for Kubernetes" in GitLab 14.6. You can connect your Kubernetes cluster with GitLab to deploy, manage, -and monitor your cloud-native solutions. +and monitor your cloud-native solutions. To connect a Kubernetes cluster to GitLab, you must first [install an agent in your cluster](install/index.md). @@ -43,8 +43,8 @@ This workflow is considered push-based, because GitLab is pushing requests from GitLab supports the following Kubernetes versions. You can upgrade your Kubernetes version to a supported version at any time: +- 1.21 (support ends on November 22, 2022) - 1.20 (support ends on July 22, 2022) -- 1.19 (support ends on February 22, 2022) GitLab supports at least two production-ready Kubernetes minor versions at any given time. GitLab regularly reviews the supported versions and diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index fca80a4a291..e76ef9e827d 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -39,6 +39,9 @@ To install the agent in your cluster: You must register an agent with GitLab. +FLAG: +In GitLab 14.10, a [flag](../../../../administration/feature_flags.md) named `certificate_based_clusters` changed the **Actions** menu to focus on the agent rather than certificates. The flag is [enabled on GitLab.com and self-managed](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). + Prerequisites: - For a [GitLab CI/CD workflow](../ci_cd_tunnel.md), ensure that @@ -48,8 +51,7 @@ To register an agent with GitLab: 1. On the top bar, select **Menu > Projects** and find your project. 1. From the left sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select **Actions**. -1. From the **Select an agent** dropdown list: +1. Select **Connect a cluster (agent)**. - If you want to create a configuration with CI/CD defaults, type a name for the agent. - If you already have an [agent configuration file](#create-an-agent-configuration-file), select it from the list. 1. Select **Register an agent**. @@ -61,11 +63,12 @@ To register an agent with GitLab: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7, the agent configuration file can be added to multiple directories (or subdirectories) of the repository. > - Group authorization was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. -You can use an agent configuration file to specify details about your implementation. -Creating a file is optional but is needed if: +The agent is configured through a configuration file. This file is optional. Without a configuration file, you can still use the CI/CD workflow in the project where the agent is registered. + +You need a configuration file if: -- You use [a GitOps workflow](../gitops.md#gitops-configuration-reference) and you want a more advanced configuration. -- You use a GitLab CI/CD workflow. In that workflow, you must [authorize the agent](../ci_cd_tunnel.md#authorize-the-agent). +- You want to use [a GitOps workflow](../gitops.md#gitops-configuration-reference). +- You want to authorize a different project to use the agent for a [GitLab CI/CD workflow](../ci_cd_tunnel.md#authorize-the-agent). To create an agent configuration file, go to the GitLab project. In the repository, create a file called `config.yaml` at this path: @@ -79,104 +82,62 @@ To create an agent configuration file, go to the GitLab project. In the reposito - For a GitOps workflow, view [the configuration reference](../gitops.md#gitops-configuration-reference) for details. - For a GitLab CI/CD workflow, you can leave the file blank for now. -The agent bootstraps with the GitLab installation URL and an access token, -and you provide the rest of the configuration in your repository, following -Infrastructure as Code (IaaC) best practices. - ### Install the agent in the cluster -To connect your cluster to GitLab, install the registered agent -in your cluster. To install it, you can use either: +> Introduced in GitLab 14.10, GitLab recommends using Helm to install the agent. -- [The one-liner installation method](#one-liner-installation). -- [The advanced installation method](#advanced-installation). +To connect your cluster to GitLab, install the registered agent +in your cluster. You can either: -You can use the one-liner installation for trying to use the agent for the first time, to do internal setups with -high trust, and to quickly get started. For long-term production usage, you may want to use the advanced installation -method to benefit from more configuration options. +- [Install the agent with Helm](#install-the-agent-with-helm). +- Or, follow the [advanced installation method](#advanced-installation-method). -#### One-liner installation +If you do not know which one to choose, we recommend starting with Helm. -The one-liner installation is the simplest process, but you need -Docker installed locally. If you don't have it, you can either install -it or opt to the [advanced installation method](#advanced-installation). +#### Install the agent with Helm -To install the agent on your cluster using the one-liner installation: +To install the agent on your cluster using Helm: +1. [Install Helm](https://helm.sh/docs/intro/install/) 1. In your computer, open a terminal and [connect to your cluster](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/). -1. Run the command you copied when registering your cluster in the previous step. +1. Run the command you copied when registering your agent with GitLab. -Optionally, you can [customize the one-liner installation command](#customize-the-one-liner-installation). +Optionally, you can [customize the Helm installation](#customize-the-helm-installation). -##### Customize the one-liner installation +##### Customize the Helm installation -By default, the one-liner command generated by GitLab: +By default, the Helm installation command generated by GitLab: -- Creates a namespace for the deployment (`gitlab-agent`). -- Sets up a service account with `cluster-admin` rights (see [how to restrict this service account](#customize-the-permissions-for-the-agentk-service-account)). -- Creates a `Secret` resource for the agent's access token. +- Creates a namespace `gitlab-agent` for the deployment (`--namespace gitlab-agent`). You can skip creating the namespace by omitting the `--create-namespace` flag. +- Sets up a service account for the agent with `cluster-admin` rights. You can: + - Skip creating the service account by adding `--set serviceAccount.create=false` to the `helm install` command. In this case, you must set `serviceAccount.name` to a pre-existing service account. + - Skip creating the RBAC permissions by adding `--set rbac.create=false` to the `helm install` command. In this case, you must bring your own RBAC permissions for the agent. Otherwise, it has no permissions at all. +- Creates a `Secret` resource for the agent's access token. To instead bring your own secret with a token, omit the token (`--set token=...`) and instead use `--set config.secretName=<your secret name>`. - Creates a `Deployment` resource for the `agentk` pod. -You can edit these parameters to customize the one-liner installation command. -To view all available options, open a terminal and run this command: - -```shell -docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --help - -Usage: - cli generate [flags] - -Flags: - --agent-token string Access token registered for agent - --agent-version string Version of the agentk image to use (default "v14.8.1") - -h, --help help for generate - --kas-address string GitLab agent server for Kubernetes address - --name-prefix string The prefix to use for names of Kubernetes objects - --namespace string Kubernetes namespace to create resources in (default "gitlab-agent") - --no-rbac Do not include corresponding Roles and RoleBindings for the agent service account -``` - -WARNING: -Use `--agent-version stable` to refer to the latest stable -release at the time when the command runs. For production, however, -you should explicitly specify a matching version. - -#### Advanced installation - -For advanced installation options, use [the `kpt` installation method](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). - -##### Customize the permissions for the `agentk` service account - -You own your cluster and can grant GitLab the permissions you want. -By default, however, the generated manifests provide `cluster-admin` rights to the agent. - -You can restrict the agent's access rights by using Kustomize overlays. [An example is commented out](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/build/deployment/gitlab-agent/cluster/kustomization.yaml) in the `kpt` package you retrieved as part of the installation. - -To restrict permissions: +To see the full list of customizations available, see the Helm chart's [default values file](https://gitlab.com/gitlab-org/charts/gitlab-agent/-/blob/main/values.yaml). -1. Copy the `cluster` directory. -1. Edit the `kustomization.yaml` and `components/*` files based on your requirements. -1. Run `kustomize build <your copied directory> | kubectl apply -f -` to apply your configuration. +#### Advanced installation method -#### Update the advanced installation base layer - -Now you can update from the upstream package by using `kpt pkg update gitlab-agent --strategy resource-merge`. -When the advanced installation setup changes, you will not need to change your custom overlays. +GitLab also provides a [KPT package for the agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). This method provides greater flexibility, but is only recommended for advanced users. ## Install multiple agents in your cluster -For total separation between teams, you might need to run multiple `agentk` instances in your cluster. -You might want multiple agents so you can restrict RBAC for every `agentk` deployment. +To install a second agent in your cluster, you can follow the [previous steps](#register-the-agent-with-gitlab) a second time. To avoid resource name collisions within the cluster, you must either: + +- Use a different release name for the agent, e.g. `second-gitlab-agent`: -To install multiple agents, follow the -[advanced installation steps](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent) -a second time and: + ```shell + helm upgrade --install second-gitlab-agent gitlab/gitlab-agent ... + ``` -1. Change the agent name and create a new configuration file. -1. Register the new agent. You receive a new access token. Each token should be used only with one agent. -1. Change the namespace or prefix you use for the installation. +- Or, install the agent in a different namespace, e.g. `different-namespace`: -You should also change the RBAC for the installed `agentk`. + ```shell + helm upgrade --install gitlab-agent gitlab/gitlab-agent \ + --namespace different-namespace \ + ... + ``` ## Example projects @@ -187,50 +148,37 @@ The following example projects can help you get started with the agent. - [Auto DevOps setup that uses the CI/CD workflow](https://gitlab.com/gitlab-examples/ops/gitops-demo/hello-world-service) - [Cluster management project template example that uses the CI/CD workflow](https://gitlab.com/gitlab-examples/ops/gitops-demo/cluster-management) -## Upgrades and version compatibility - -The agent has two major components: `agentk` and `kas`. -GitLab provides `kas` installers built into the various GitLab installation methods. -The required `kas` version corresponds to the GitLab `major.minor` (X.Y) versions. - -At the same time, `agentk` and `kas` can differ by 1 minor version in either direction. For example, -`agentk` 14.4 supports `kas` 14.3, 14.4, and 14.5 (regardless of the patch). +## Updates and version compatibility -A feature introduced in a given GitLab minor version might work with other `agentk` or `kas` versions. -To ensure it works, use at least the same `agentk` and `kas` minor version. For example, -if your GitLab version is 14.2, use at least `agentk` 14.2 and `kas` 14.2. - -We recommend upgrading your `kas` installations together with GitLab instances' upgrades, and to -[upgrade the `agentk` installations](#update-the-agent-version) after upgrading GitLab. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340882) in GitLab 14.8, GitLab warns you on the agent's list page to update the agent version installed on your cluster. -The available `agentk` and `kas` versions are available in -[the Container Registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/). +For the best experience, the version of the agent installed in your cluster should match the GitLab major and minor version. The previous minor version is also supported. For example, if your GitLab version is v14.9.4 (major version 14, minor version 9), then versions v14.9.0 and v14.9.1 of the agent are ideal, but any v14.8.x version of the agent is also supported. See [this page](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/releases) of releases of the GitLab agent. ### Update the agent version -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340882) in GitLab 14.8, GitLab warns you on the agent's list page to update the agent version installed on your cluster. - -To update the agent's version, re-run the [installation command](#install-the-agent-in-the-cluster) -with a newer `--agent-version`. Make sure to specify the other required parameters: `--kas-address`, `--namespace`, and `--agent-token`. -The available `agentk` versions are in [the Container Registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/1223205?sort=desc). +To update the agent to the latest version, you can run: -If you don't have access to your agent's access token, you can retrieve it from your cluster: - -1. Open a terminal and connect to your cluster. -1. To retrieve the namespace, run: +```shell +helm repo update +helm upgrade --install gitlab-agent gitlab/gitlab-agent \ + --namespace gitlab-agent \ + --reuse-values +``` - ```shell - kubectl get namespaces - ``` +To set a specific version, you can override the `image.tag` value. For example, to install version `v14.9.1`, run: -1. To retrieve the secret, run: +```shell +helm upgrade gitlab-agent gitlab/gitlab-agent \ + --namespace gitlab-agent \ + --reuse-values \ + --set image.tag=v14.9.1 +``` - ```shell - kubectl -n <namespace> get secrets - ``` +## Uninstall the agent -1. To retrieve the access token, run: +If you [installed the agent with Helm](#install-the-agent-with-helm), then you can also uninstall with Helm. For example, if the release and namespace are both called `gitlab-agent`, then you can uninstall the agent using the following command: - ```shell - kubectl -n <namespace> get secret <secret-name> --template={{.data.token}} | base64 --decode - ``` +```shell +helm uninstall gitlab-agent \ + --namespace gitlab-agent +``` diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index 3f743d34e0a..2087c804e26 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -6,11 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Working with the agent for Kubernetes **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the GitLab agent became available on GitLab.com. -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) the `ci_access` attribute in GitLab 14.3. -> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. +Use the following tasks when working with the agent for Kubernetes. ## View your agents @@ -171,7 +167,7 @@ To remove an agent from the UI: WARNING: Cilium integration is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. The agent for Kubernetes also provides an integration with Cilium. This integration provides a simple way to diff --git a/doc/user/clusters/agent/troubleshooting.md b/doc/user/clusters/agent/troubleshooting.md index a5e568837ad..c5c7e46c078 100644 --- a/doc/user/clusters/agent/troubleshooting.md +++ b/doc/user/clusters/agent/troubleshooting.md @@ -27,9 +27,8 @@ If you are a GitLab administrator, you can also view the [GitLab agent server lo } ``` -This error is shown if there are some connectivity issues between the address -specified as `kas-address`, and your agent pod. To fix it, make sure that you -specified the `kas-address` correctly. +This error occurs when there are connectivity issues between the `kas-address` +and your agent pod. To fix this issue, make sure the `kas-address` is accurate. ```json { @@ -41,8 +40,8 @@ specified the `kas-address` correctly. } ``` -This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the -`wss` or `ws` URL ends with a trailing slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` +This error occurs when the `kas-address` doesn't include a trailing slash. To fix this issue, make sure that the +`wss` or `ws` URL ends with a trailing slash, like `wss://GitLab.host.tld:443/-/kubernetes-agent/` or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. ## ValidationError(Deployment.metadata) @@ -58,9 +57,10 @@ or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. } ``` -This error is shown if a manifest file is malformed, and Kubernetes can't -create specified objects. Make sure that your manifest files are valid. You -may try using them to create objects in Kubernetes directly for more troubleshooting. +This error occurs when a manifest file is malformed and Kubernetes can't +create the specified objects. Make sure that your manifest files are valid. + +For additional troubleshooting, try to use the manifest files to create objects in Kubernetes directly. ## Error while dialing failed to WebSocket dial: failed to send handshake request @@ -73,16 +73,10 @@ may try using them to create objects in Kubernetes directly for more troubleshoo } ``` -This error is shown if you configured `wss` as `kas-address` on the agent side, -but KAS on the server side is not available via `wss`. To fix it, make sure the +This error occurs when you configured `wss` as `kas-address` on the agent side, +but the agent server is not available at `wss`. To fix this issue, make sure the same schemes are configured on both sides. -It's not possible to set the `grpc` scheme due to the issue -[It is not possible to configure KAS to work with `grpc` without directly editing GitLab KAS deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/276888). To use `grpc` while the -issue is in progress, directly edit the deployment with the -`kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use -`grpc://gitlab-kas.<YOUR-NAMESPACE>:8150`. - ## Decompressor is not installed for grpc-encoding ```json @@ -94,8 +88,8 @@ issue is in progress, directly edit the deployment with the } ``` -This error is shown if the version of the agent is newer that the version of KAS. -To fix it, make sure that both `agentk` and KAS use the same versions. +This error occurs when the version of the agent is newer that the version of the agent server (KAS). +To fix it, make sure that both `agentk` and the agent server are the same version. ## Certificate signed by unknown authority @@ -109,9 +103,11 @@ To fix it, make sure that both `agentk` and KAS use the same versions. } ``` -This error is shown if your GitLab instance is using a certificate signed by an internal CA that -is unknown to the agent. One approach to fixing it is to present the CA certificate file to the agent -via a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it +This error occurs when your GitLab instance is using a certificate signed by an internal +certificate authority that is unknown to the agent. + +To fix this issue, you can present the CA certificate file to the agent +by using a Kubernetes `configmap` and mount the file in the agent `/etc/ssl/certs` directory from where it will be picked up automatically. For example, if your internal CA certificate is `myCA.pem`: @@ -153,7 +149,7 @@ Then in `resources.yml`: path: myCA.pem ``` -Alternatively, you can mount the certificate file at a different location and include it using the +Alternatively, you can mount the certificate file at a different location and specify it for the `--ca-cert-file` agent parameter: ```yaml @@ -188,5 +184,5 @@ Alternatively, you can mount the certificate file at a different location and in } ``` -This error is shown if the manifest project is not public. To fix it, make sure your manifest project is public or your manifest files -are stored in the agent's configuration repository. +This error occurs when the project where you keep your manifests is not public. To fix it, make sure your project is public or your manifest files +are stored in the repository where the agent is configured. diff --git a/doc/user/compliance/compliance_report/img/failed_icon_v13_3.png b/doc/user/compliance/compliance_report/img/failed_icon_v13_3.png Binary files differdeleted file mode 100644 index c3f386c9dee..00000000000 --- a/doc/user/compliance/compliance_report/img/failed_icon_v13_3.png +++ /dev/null diff --git a/doc/user/compliance/compliance_report/img/success_icon_v13_3.png b/doc/user/compliance/compliance_report/img/success_icon_v13_3.png Binary files differdeleted file mode 100644 index ea6ca924f81..00000000000 --- a/doc/user/compliance/compliance_report/img/success_icon_v13_3.png +++ /dev/null diff --git a/doc/user/compliance/compliance_report/img/warning_icon_v13_3.png b/doc/user/compliance/compliance_report/img/warning_icon_v13_3.png Binary files differdeleted file mode 100644 index 168a7021948..00000000000 --- a/doc/user/compliance/compliance_report/img/warning_icon_v13_3.png +++ /dev/null diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index 27783a063da..77dbefa0755 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -9,17 +9,19 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36524) in GitLab 12.8 as Compliance Dashboard. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/299360) to compliance report in GitLab 14.2. +> - [Replaced](https://gitlab.com/groups/gitlab-org/-/epics/5237) by merge request violations in GitLab 14.6 [with a flag](../../../administration/feature_flags.md) named `compliance_violations_report`. Disabled by default. +> - GraphQL API [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7222) in GitLab 14.9. +> - [Generally available](https://gitlab.com/groups/gitlab-org/-/epics/5237) in GitLab 14.10. [Feature flag `compliance_violations_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/346266) removed. Compliance report gives you the ability to see a group's merge request activity. It provides a high-level view for all projects in the group. For example, code approved for merging into production. -You can use the report to: +You can use the report to get: -- Get an overview of the latest merge request for each project. -- See if merge requests were approved and by whom. -- See merge request authors. -- See the latest [CI/CD pipeline](../../../ci/pipelines/index.md) result for each merge request. +- A list of compliance violations from all merged merge requests within the group. +- The reason and severity of each compliance violation. +- A link to the merge request that caused each compliance violation. ## View the compliance report for a group @@ -32,8 +34,36 @@ To view the compliance report: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Compliance report**. -NOTE: -The compliance report shows only the latest merge request on each project. +### Severity levels scale + +The following is a list of available violation severity levels, ranked from most to least severe: + +| Icon | Severity level | +|:----------------------------------------------|:---------------| +| **{severity-critical, 18, gl-fill-red-800}** | Critical | +| **{severity-high, 18, gl-fill-red-600}** | High | +| **{severity-medium, 18, gl-fill-orange-400}** | Medium | +| **{severity-low, 18, gl-fill-orange-300}** | Low | +| **{severity-info, 18, gl-fill-blue-400}** | Info | + +### Violation types + +The following is a list of violations that are either: + +- Already available. +- Aren't available, but which we are tracking in issues. + +| Violation | Severity level | Category | Description | Availability | +|:-------------------------------------|:----------------|:---------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| +| Author approved merge request | High | [Separation of duties](#separation-of-duties) | The author of the merge request approved their own merge request. [Learn more](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | +| Committers approved merge request | High | [Separation of duties](#separation-of-duties) | The committers of the merge request approved the merge request they contributed to. [Learn more](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | +| Fewer than two approvals | High | [Separation of duties](#separation-of-duties) | The merge request was merged with fewer than two approvals. [Learn more](../../project/merge_requests/approvals/rules.md). | [Available in GitLab 14.10](https://gitlab.com/groups/gitlab-org/-/epics/6870) | +| Pipeline failed | Medium | [Pipeline results](../../../ci/pipelines/index.md) | The merge requests pipeline failed and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Pipeline passed with warnings | Info | [Pipeline results](../../../ci/pipelines/index.md) | The merge request pipeline passed with warnings and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Code coverage down more than 10% | High | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of more than 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | +| Code coverage down less than 1% | Info | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of less than 1%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | ## Merge request drawer @@ -51,30 +81,15 @@ request: - A list of users that approved the merge request. - The user that merged the merge request. -## Approval status and separation of duties - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217939) in GitLab 13.3. +## Separation of duties We support a separation of duties policy between users who create and approve merge requests. -The approval status column can help you identify violations of this policy. Our criteria for the separation of duties is as follows: - [A merge request author is **not** allowed to approve their merge request](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author) - [A merge request committer is **not** allowed to approve a merge request they have added commits to](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits) - [The minimum number of approvals required to merge a merge request is **at least** two](../../project/merge_requests/approvals/rules.md) -The **Approval status** column shows you at a glance whether a merge request is complying with the above. -This column has four states: - -| State | Description | -|:------|:------------| -| Empty | The merge request approval status is unknown | -|  | The merge request **does not** comply with any of the above criteria | -|  | The merge request complies with **some** of the above criteria | -|  | The merge request complies with **all** of the above criteria | - -If you see a non-success state, review the criteria for the merge request's project to ensure it complies with the separation of duties. - ## Chain of Custody report > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213364) in GitLab 13.3. @@ -105,64 +120,3 @@ You can generate a commit-specific Chain of Custody report for a given commit SH NOTE: The Chain of Custody report download is a CSV file, with a maximum size of 15 MB. The remaining records are truncated when this limit is reached. - -## Merge request violations - -> - Introduced in GitLab 14.6. [Deployed behind the `compliance_violations_report` flag](../../../administration/feature_flags.md). Disabled by default. -> - GraphQL API [introduced](https://gitlab.com/groups/gitlab-org/-/epics/7222) in GitLab 14.9. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `compliance_violations_report`. -On GitLab.com, this feature is not available. This feature is not ready for production use. - -Merge request violations provide a view of all the [separation of duties](#approval-status-and-separation-of-duties) compliance violations -that exist in projects in a specific group. For each separation of duties compliance violation, you can see: - -- A list of compliance violations. -- The severity of each compliance violation. -- Reason for the compliance violation. -- A link to the merge request that caused the compliance violation. - -Merge request violations can be accessed: - -- In the GitLab UI. -- Using the [GraphQL API](../../../api/graphql/reference/index.md#complianceviolation) (GitLab 14.9 and later). - -### View merge request violations - -To view merge request violations: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance report**. - -### Severity levels scale - -The following is a list of available violation severity levels, ranked from most to least severe: - -| Icon | Severity level | -|:----------------------------------------------|:---------------| -| **{severity-critical, 18, gl-fill-red-800}** | Critical | -| **{severity-high, 18, gl-fill-red-600}** | High | -| **{severity-medium, 18, gl-fill-orange-400}** | Medium | -| **{severity-low, 18, gl-fill-orange-300}** | Low | -| **{severity-info, 18, gl-fill-blue-400}** | Info | - -### Violation types - -The following is a list of violations that are either: - -- Already available. -- Aren't available, but which we are tracking in issues. - -| Violation | Severity level | Category | Description | Availability | -|:-------------------------------------|:----------------|:----------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------| -| Author approved merge request | High | [Separation of duties](#approval-status-and-separation-of-duties) | The author of the merge request approved their own merge request. [Learn more](../../project/merge_requests/approvals/settings.md#prevent-approval-by-author). | [Unavailable](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Committers approved merge request | High | [Separation of duties](#approval-status-and-separation-of-duties) | The committers of the merge request approved the merge request they contributed to. [Learn more](../../project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits). | [Unavailable](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Fewer than two approvals | High | [Separation of duties](#approval-status-and-separation-of-duties) | The merge request was merged with fewer than two approvals. [Learn more](../../project/merge_requests/approvals/rules.md). | [Unavailable](https://gitlab.com/groups/gitlab-org/-/epics/6870) | -| Pipeline failed | Medium | [Pipeline results](../../../ci/pipelines/index.md) | The merge requests pipeline failed and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Pipeline passed with warnings | Info | [Pipeline results](../../../ci/pipelines/index.md) | The merge request pipeline passed with warnings and was merged. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down more than 10% | High | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of more than 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | -| Code coverage down less than 1% | Info | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | The code coverage report for the merge request indicates a reduction in coverage of less than 1%. | [Unavailable](https://gitlab.com/gitlab-org/gitlab/-/issues/346011) | diff --git a/doc/user/crm/crm_contacts_v14_10.png b/doc/user/crm/crm_contacts_v14_10.png Binary files differnew file mode 100644 index 00000000000..f82878f9d4b --- /dev/null +++ b/doc/user/crm/crm_contacts_v14_10.png diff --git a/doc/user/crm/crm_contacts_v14_6.png b/doc/user/crm/crm_contacts_v14_6.png Binary files differdeleted file mode 100644 index 37a615f3926..00000000000 --- a/doc/user/crm/crm_contacts_v14_6.png +++ /dev/null diff --git a/doc/user/crm/crm_organizations_v14_10.png b/doc/user/crm/crm_organizations_v14_10.png Binary files differnew file mode 100644 index 00000000000..256c9c3f83b --- /dev/null +++ b/doc/user/crm/crm_organizations_v14_10.png diff --git a/doc/user/crm/crm_organizations_v14_6.png b/doc/user/crm/crm_organizations_v14_6.png Binary files differdeleted file mode 100644 index 2bde3823cc7..00000000000 --- a/doc/user/crm/crm_organizations_v14_6.png +++ /dev/null diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index 1fb628cf505..7fc11add2cd 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -49,7 +49,7 @@ To view a group's contacts: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Customer relations > Contacts**. - + ### Create a contact @@ -86,7 +86,7 @@ To view a group's organizations: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Customer relations > Organizations**. - + ### Create an organization @@ -103,7 +103,15 @@ organizations using the GraphQL API. ### Edit an organization -You can only [edit](../../api/graphql/reference/index.md#mutationcustomerrelationsorganizationupdate) +To edit an existing organization: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Customer relations > Organizations**. +1. Next to the organization you wish to edit, select **Edit** (**{pencil}**). +1. Edit the required fields. +1. Select **Save changes**. + +You can also [edit](../../api/graphql/reference/index.md#mutationcustomerrelationsorganizationupdate) organizations using the GraphQL API. ## Issues @@ -171,3 +179,23 @@ When you use the `/add_contacts` or `/remove_contacts` quick actions, follow the /add_contacts [contact: /remove_contacts [contact: ``` + +## Moving objects with CRM entries + +The root group is the topmost group in the group hierarchy. + +When you move an issue, project, or group **within the same group hierarchy**, +issues retain their contacts. + +When you move an issue or project and the **root group changes**, +issues lose their contacts. + +When you move a group and its **root group changes**: + +- All unique contacts and organizations are migrated to the new root group. +- Contacts that already exist (by email address) are deemed duplicates and deleted. +- Organizations that already exist (by name) are deemed duplicates and deleted. +- All issues retain their contacts or are updated to point at contacts with the same email address. + +If you do not have permission to create contacts and organizations in the new +root group, the group transfer fails. diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index a6343e0d1cf..8537856ef25 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -112,12 +112,27 @@ you can reply to comments by sending an email. You can use [Markdown](../markdown.md) and [quick actions](../project/quick_actions.md) in your email replies. -## Who can edit comments +## Edit a comment You can edit your own comment at any time. - Anyone with at least the Maintainer role can also edit a comment made by someone else. +To edit a comment: + +1. On the comment, select **Edit comment** (**{pencil}**). +1. Make your edits. +1. Select **Save changes**. + +### Editing a comment to add a mention + +By default, when you mention a user, GitLab [creates a to-do item](../todos.md#actions-that-create-to-do-items) +for them, and sends them a [notification email](../profile/notifications.md). + +If you edit an existing comment to add a user mention that wasn't there before, GitLab: + +- Creates a to-do item for the mentioned user. +- Does not send a notification email. + ## Prevent comments by locking an issue You can prevent public comments in an issue or merge request. @@ -137,7 +152,8 @@ If an issue or merge request is locked and closed, you cannot reopen it. ## Mark a comment as confidential **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9 [with a flag](../../administration/feature_flags.md) named `confidential_notes`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9 [with a flag](../../administration/feature_flags.md) named `confidential_notes`. Disabled by default. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/351143) in GitLab 14.10: you can only mark comments in issues and epics as confidential. Previously, it was also possible for comments in merge requests and snippets. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, @@ -145,9 +161,25 @@ ask an administrator to [enable the feature flag](../../administration/feature_f On GitLab.com, this feature is not available. You should not use this feature for production environments. -You can make a comment confidential, so that it is visible only to project members -who have at least the Reporter role. +You can make a comment **in an issue or an epic** confidential, so that it is visible only to you (the commenting user) and +the project members who have at least the Reporter role. + +Keep in mind: + +- You can only mark comments as confidential when you create them. +- You can't change the confidentiality of existing comments. +- Replies to comments use same confidentiality as the original comment. + +Prerequisites: + +- You must either: + - Have at least the Reporter role for the project. + - Be the issue assignee. + - Be the issue author. + +To mark a comment as confidential: +1. Start adding a new comment. 1. Below the comment, select the **Make this comment confidential** checkbox. 1. Select **Comment**. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 5cf6a505bee..6bb45575fc3 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -48,9 +48,16 @@ gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAA ## Mail configuration GitLab.com sends emails from the `mg.gitlab.com` domain by using [Mailgun](https://www.mailgun.com/), -and has its own dedicated IP address (`192.237.158.143`). +and has its own dedicated IP addresses: -The IP address for `mg.gitlab.com` is subject to change at any time. +- `161.38.202.219` +- `159.135.226.146` +- `192.237.158.143` +- `198.61.254.136` +- `23.253.183.236` +- `69.72.35.190` + +The IP addresses for `mg.gitlab.com` are subject to change at any time. ### Service Desk custom mailbox @@ -138,7 +145,7 @@ the related documentation. | Artifacts maximum size (compressed) | 1 GB | See [Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size) | | Artifacts [expiry time](../../ci/yaml/index.md#artifactsexpire_in) | From June 22, 2020, deleted after 30 days unless otherwise specified (artifacts created before that date have no expiry). | See [Default artifacts expiration](../admin_area/settings/continuous_integration.md#default-artifacts-expiration) | | Scheduled Pipeline Cron | `*/5 * * * *` | See [Pipeline schedules advanced configuration](../../administration/cicd.md#change-maximum-scheduled-pipeline-frequency) | -| Maximum jobs in active pipelines | `500` for Free tier, unlimited otherwise | See [Number of jobs in active pipelines](../../administration/instance_limits.md#number-of-jobs-in-active-pipelines) | +| Maximum jobs in active pipelines | `500` for Free tier, `1000` for all trial tiers, and unlimited otherwise. | See [Number of jobs in active pipelines](../../administration/instance_limits.md#number-of-jobs-in-active-pipelines) | | Maximum CI/CD subscriptions to a project | `2` | See [Number of CI/CD subscriptions to a project](../../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) | | Maximum number of pipeline triggers in a project | `25000` for Free tier, Unlimited for all paid tiers | See [Limit the number of pipeline triggers](../../administration/instance_limits.md#limit-the-number-of-pipeline-triggers) | | Maximum pipeline schedules in projects | `10` for Free tier, `50` for all paid tiers | See [Number of pipeline schedules](../../administration/instance_limits.md#number-of-pipeline-schedules) | @@ -315,24 +322,30 @@ limiting responses](#rate-limiting-responses). The following table describes the rate limits for GitLab.com, both before and after the limits change in January, 2021: -| Rate limit | Before 2021-02-12 | From 2021-02-12 | From 2022-02-03 | -|:--------------------------------------------------------------------------|:------------------------------|:------------------------------|:----------------------------------------| -| **Protected paths** (for a given **IP address**) | **10** requests per minute | **10** requests per minute | **10** requests per minute | -| **Raw endpoint** traffic (for a given **project, commit, and file path**) | **300** requests per minute | **300** requests per minute | **300** requests per minute | -| **Unauthenticated** traffic (from a given **IP address**) | **500** requests per minute | **500** requests per minute | **500** requests per minute | -| **Authenticated** API traffic (for a given **user**) | **2,000** requests per minute | **2,000** requests per minute | **2,000** requests per minute | -| **Authenticated** non-API HTTP traffic (for a given **user**) | **1,000** requests per minute | **1,000** requests per minute | **1,000** requests per minute | -| **All** traffic (from a given **IP address**) | **2,000** requests per minute | **2,000** requests per minute | **2,000** requests per minute | -| **Issue creation** | **300** requests per minute | **300** requests per minute | **300** requests per minute | -| **Note creation** (on issues and merge requests) | **300** requests per minute | **60** requests per minute | **60** requests per minute | -| **Advanced, project, and group search** API (for a given **IP address**) | | **10** requests per minute | **10** requests per minute | -| **GitLab Pages** requests (for a given **IP address**) | | | **1000** requests per **50 seconds** | -| **GitLab Pages** requests (for a given **GitLab Pages domain**) | | | **5000** requests per **10 seconds** | +| Rate limit | From 2021-02-12 | From 2022-02-03 | +|:--------------------------------------------------------------------------|:------------------------------|:----------------------------------------| +| **Protected paths** (for a given **IP address**) | **10** requests per minute | **10** requests per minute | +| **Raw endpoint** traffic (for a given **project, commit, and file path**) | **300** requests per minute | **300** requests per minute | +| **Unauthenticated** traffic (from a given **IP address**) | **500** requests per minute | **500** requests per minute | +| **Authenticated** API traffic (for a given **user**) | **2,000** requests per minute | **2,000** requests per minute | +| **Authenticated** non-API HTTP traffic (for a given **user**) | **1,000** requests per minute | **1,000** requests per minute | +| **All** traffic (from a given **IP address**) | **2,000** requests per minute | **2,000** requests per minute | +| **Issue creation** | **300** requests per minute | **300** requests per minute | +| **Note creation** (on issues and merge requests) | **60** requests per minute | **60** requests per minute | +| **Advanced, project, and group search** API (for a given **IP address**) | **10** requests per minute | **10** requests per minute | +| **GitLab Pages** requests (for a given **IP address**) | | **1000** requests per **50 seconds** | +| **GitLab Pages** requests (for a given **GitLab Pages domain**) | | **5000** requests per **10 seconds** | More details are available on the rate limits for [protected paths](#protected-paths-throttle) and [raw endpoints](../../user/admin_area/settings/rate_limits_on_raw_endpoints.md). +GitLab can rate-limit requests at several layers. The rate limits listed here +are configured in the application. These limits are the most +restrictive per IP address. To learn more about the rate limiting +for GitLab.com, read our runbook page +[Overview of rate limits for GitLab.com](https://gitlab.com/gitlab-com/runbooks/-/tree/master/docs/rate-limiting). + ### Rate limiting responses For information on rate limiting responses, see: @@ -395,7 +408,7 @@ doesn't return the following headers: ### Visibility settings If created before GitLab 12.2 (July 2019), these items have the -[Internal visibility](../../public_access/public_access.md#internal-projects-and-groups) +[Internal visibility](../public_access.md#internal-projects-and-groups) setting [disabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/12388): - Projects diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index 3b866c4a1b0..cd2d5c190a1 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3090) in GitLab 12.2 for subgroups. -With Contribution Analytics, you can get an overview of the [contribution events](../../index.md#user-contribution-events) in your +With Contribution Analytics, you can get an overview of the [contribution events](../../profile/index.md#user-contribution-events) in your group. - Analyze your team's contributions over a period of time. diff --git a/doc/user/group/epics/img/related_epic_block_v14_9.png b/doc/user/group/epics/img/related_epic_block_v14_9.png Binary files differindex 7b5824b84d1..20fdce0151d 100644 --- a/doc/user/group/epics/img/related_epic_block_v14_9.png +++ b/doc/user/group/epics/img/related_epic_block_v14_9.png diff --git a/doc/user/group/epics/img/related_epics_add_v14_9.png b/doc/user/group/epics/img/related_epics_add_v14_9.png Binary files differindex 3da6eeaff43..112b900f2e3 100644 --- a/doc/user/group/epics/img/related_epics_add_v14_9.png +++ b/doc/user/group/epics/img/related_epics_add_v14_9.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 149c5362ac9..f2cb437ffda 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -45,20 +45,6 @@ have a start or due date, a visual  -## Permissions - -If you have access to view an epic and an issue added to that epic, you can view the issue in the -epic's issue list. - -If you have access to edit an epic and an issue added to that epic, you can add the issue to or -remove it from the epic. - -For a given group, the visibility of all projects must be the same as -the group, or less restrictive. That means if you have access to a group's epic, -then you already have access to its projects' issues. - -You can also consult the [group permissions table](../../permissions.md#group-members-permissions). - ## Related topics - [Manage epics](manage_epics.md) and multi-level child epics. diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 3350b0f1169..e8f720238ed 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -16,6 +16,10 @@ to them. > - In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/229621) and later, the New Epic button on the Epics list opens the New Epic form. > - In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45948) and later, you can create a new epic from an empty roadmap. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + To create an epic in the group you're in: 1. Get to the New Epic form: @@ -68,6 +72,10 @@ After you create an epic, you can edit the following details: - Due date - Labels +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + To edit an epic's title or description: 1. Select **Edit title and description** **{pencil}**. @@ -87,6 +95,10 @@ Users with at least the Reporter role can manage epics. When bulk editing epics in a group, you can edit their labels. +Prerequisites: + +- You must have at least the Reporter role for the parent epic's group. + To update multiple epics at the same time: 1. In a group, go to **Epics > List**. @@ -97,8 +109,9 @@ To update multiple epics at the same time: ## Delete an epic -NOTE: -To delete an epic, you must be an Owner of a group or subgroup. +Prerequisites: + +- You must have the Owner role for the epic's group. To delete the epic: @@ -112,6 +125,10 @@ If you delete an epic, all its child epics and their descendants are deleted as ## Close an epic +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + Whenever you decide that there is no longer need for that epic, close the epic by: @@ -123,13 +140,19 @@ close the epic by: ## Reopen a closed epic -You can reopen an epic that was closed by: +You can reopen an epic that was closed. + +Prerequisites: -- Selecting **Reopen epic**. +- You must have at least the Reporter role for the epic's group. + +To do so, either: + +- Select **Reopen epic**.  -- Using the `/reopen` [quick action](../../project/quick_actions.md). +- Use the `/reopen` [quick action](../../project/quick_actions.md). ## Go to an epic from an issue @@ -144,6 +167,13 @@ In a group, the left sidebar displays the total count of open epics. This number indicates all epics associated with the group and its subgroups, including epics you might not have permission to view. +Prerequisites: + +- You must be a member of either: + - The group + - A project in the group + - A project in one of the group's subgroups + To view epics in a group: 1. On the top bar, select **Menu > Groups** and find your group. @@ -225,6 +255,10 @@ and confidential child epics. However, merge requests are public, if created in Read [Merge requests for confidential issues](../../project/merge_requests/confidential.md) to learn how to create a confidential merge request. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + To make an epic confidential: - **When creating an epic:** select the checkbox under **Confidentiality**. @@ -236,6 +270,15 @@ To make an epic confidential: This section collects instructions for all the things you can do with [issues](../../project/issues/index.md) in relation to epics. +### View issues assigned to an epic + +On the **Epics and Issues** tab, you can see epics and issues assigned to this epic. +Only epics and issues that you can access show on the list. + +You can always view the issues assigned to the epic if they are in the group's child project. +It's possible because the visibility setting of a project must be the same as or less restrictive than +of its parent group. + ### View count of issues in an epic On the **Epics and Issues** tab, under each epic name, hover over the total counts. @@ -258,7 +301,12 @@ An epic contains a list of issues and an issue can be associated with at most on When you add a new issue that's already linked to an epic, the issue is automatically unlinked from its current parent. -To add a new issue to an epic: +Prerequisites: + +- You must have at least the Reporter role for the epic's group. +- You must be able to [edit the issue](../../project/issues/managing_issues.md#edit-an-issue). + +To add an existing issue to an epic: 1. On the epic's page, under **Epics and Issues**, select **Add**. 1. Select **Add an existing issue**. @@ -277,19 +325,30 @@ To add a new issue to an epic: Creating an issue from an epic enables you to maintain focus on the broader context of the epic while dividing work into smaller parts. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + To create an issue from an epic: 1. On the epic's page, under **Epics and Issues**, select **Add**. 1. Select **Add a new issue**. 1. Under **Title**, enter the title for the new issue. -1. From the **Project** dropdown, select the project in which the issue should be created. +1. From the **Project** dropdown list, select the project in which the issue should be created. 1. Select **Create issue**. +The new issue is assigned to the epic. + ### Remove an issue from an epic You can remove issues from an epic when you're on the epic's details page. After you remove an issue from an epic, the issue is no longer associated with this epic. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. +- You must be able to [edit the issue](../../project/issues/managing_issues.md#edit-an-issue). + To remove an issue from an epic: 1. Next to the issue you want to remove, select **Remove** (**{close}**). @@ -305,6 +364,10 @@ To remove an issue from an epic: New issues appear at the top of the list in the **Epics and Issues** tab. You can reorder the list of issues by dragging them. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. + To reorder issues assigned to an epic: 1. Go to the **Epics and Issues** tab. @@ -317,32 +380,47 @@ To reorder issues assigned to an epic: New issues appear at the top of the list in the **Epics and Issues** tab. You can move issues from one epic to another. +Prerequisites: + +- You must have at least the Reporter role for the epic's group. +- You must be able to [edit the issue](../../project/issues/managing_issues.md#edit-an-issue). + To move an issue to another epic: 1. Go to the **Epics and Issues** tab. -1. Drag issues into the desired parent epic. +1. Drag issues into the desired parent epic in the visible hierarchy. ### Promote an issue to an epic > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab 11.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. -If you have the necessary [permissions](../../permissions.md) to close an issue and create an -epic in the immediate parent group, you can promote an issue to an epic with the `/promote` +Prerequisites: + +- The project to which the issue belongs must be in a group. +- You must have at least the Reporter role the project's immediate parent group. +- You must either: + - Have at least the Reporter role for the project. + - Be the author of the issue. + - Be assigned to the issue. + +You can promote an issue to an epic with the `/promote` [quick action](../../project/quick_actions.md#issues-merge-requests-and-epics). -Only issues from projects that are in groups can be promoted. When you attempt to promote a confidential -issue, a warning is displayed. Promoting a confidential issue to an epic makes all information + +NOTE: +Promoting a confidential issue to an epic makes all information related to the issue public as epics are public to group members. When an issue is promoted to an epic: +- If the issue was confidential, an additional warning is displayed first. - An epic is created in the same group as the project of the issue. - Subscribers of the issue are notified that the epic was created. The following issue metadata is copied to the epic: - Title, description, activity/comment thread. -- Upvotes/downvotes. +- Upvotes and downvotes. - Participants. - Group labels that the issue already has. - Parent epic. @@ -367,6 +445,10 @@ Epics can contain multiple nested child epics, up to a total of seven levels dee ### Add a child epic to an epic +Prerequisites: + +- You must have at least the Reporter role for the parent epic's group. + To add a child epic to an epic: 1. Select **Add**. @@ -388,6 +470,10 @@ You can move child epics from one epic to another. When you add a new epic that's already linked to a parent epic, the link to its current parent is removed. Issues and child epics cannot be intermingled. +Prerequisites: + +- You must have at least the Reporter role for the parent epic's group. + To move child epics to another epic: 1. Go to the **Epics and Issues** tab. @@ -400,6 +486,10 @@ To move child epics to another epic: New child epics appear at the top of the list in the **Epics and Issues** tab. You can reorder the list of child epics. +Prerequisites: + +- You must have at least the Reporter role for the parent epic's group. + To reorder child epics assigned to an epic: 1. Go to the **Epics and Issues** tab. @@ -407,6 +497,10 @@ To reorder child epics assigned to an epic: ### Remove a child epic from a parent epic +Prerequisites: + +- You must have at least the Reporter role for the parent epic's group. + To remove a child epic from a parent epic: 1. Select **Remove** (**{close}**) in the parent epic's list of epics. diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 0c16b535ed1..0185cb9cfdf 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -138,9 +138,9 @@ migrated: In a [rails console session](../../../administration/operations/rails_console.md#starting-a-rails-console-session), you can find the failure or error messages for the group import attempt using: -```shell +```ruby # Get relevant import records -import = BulkImports::Entity.where(namespace_id: Group.id).bulk_import +import = BulkImports::Entity.where(namespace_id: Group.id).map(&:bulk_import) # Alternative lookup by user import = BulkImport.where(user_id: User.find(...)).last @@ -154,3 +154,18 @@ entities.map(&:failures).flatten # Alternative failure lookup by status entities.where(status: [-1]).pluck(:destination_name, :destination_namespace, :status) ``` + +### Stale imports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/352985) in GitLab 14.10. + +When troubleshooting group migration, an import may not complete because the import workers took +longer than 8 hours to execute. In this case, the `status` of either a `BulkImport` or +`BulkImport::Entity` is `3` (`timeout`): + +```ruby +# Get relevant import records +import = BulkImports::Entity.where(namespace_id: Group.id).map(&:bulk_import) + +import.status #=> 3 means that the import timed out. +``` diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 4b9ff7f64e8..085cd054c14 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -1,7 +1,6 @@ --- -type: reference, howto stage: Manage -group: Authentication and Authorization +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -68,7 +67,7 @@ To create a group: 1. Enter a name for the group in **Group name**. For a list of words that cannot be used as group names, see [reserved names](../reserved_names.md). 1. Enter a path for the group in **Group URL**, which is used for the [namespace](#namespaces). -1. Choose the [visibility level](../../public_access/public_access.md). +1. Choose the [visibility level](../public_access.md). 1. Personalize your GitLab experience by answering the following questions: - What is your role? - Who will be using this group? @@ -279,8 +278,8 @@ To view the activity feed in Atom format, select the [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. Similar to how you [share a project with a group](../project/members/share_project_with_groups.md), -you can share a group with another group. Members get direct access -to the shared group. This includes members who inherited group membership from a parent group. +you can share a group with another group. To invite a group, you must be a member of it. Members get direct access +to the shared group. This includes members who inherited group membership from a parent group. To share a given group, for example, `Frontend` with another group, for example, `Engineering`: @@ -289,10 +288,14 @@ To share a given group, for example, `Frontend` with another group, for example, 1. On the left sidebar, select **Group information > Members**. 1. Select **Invite a group**. 1. In the **Select a group to invite** list, select `Engineering`. -1. Select a [role](../permissions.md). +1. Select a [role](../permissions.md) as maximum access level. 1. Select **Invite**. -All the members of the `Engineering` group are added to the `Frontend` group. +After sharing the `Frontend` group with the `Engineering` group: + +- The **Groups** tab lists the `Engineering` group. +- The **Groups** tab lists a group regardless of whether it is a public or private group. +- All members of the `Engineering` group have access to the `Frontend` group. The same access levels of the members apply up to the maximum access level selected when sharing the group. ## Manage group memberships via LDAP **(PREMIUM SELF)** @@ -793,23 +796,25 @@ The group's new subgroups have push rules set for them based on either: - The closest parent group with push rules defined. - Push rules set at the instance level, if no parent groups have push rules defined. -## Group approval rules **(PREMIUM)** +## Group approval settings **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285458) in GitLab 13.9. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md), disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.5. > - [Feature flag `group_merge_request_approval_settings_feature_flag`](https://gitlab.com/gitlab-org/gitlab/-/issues/343872) removed in GitLab 14.9. -Group approval rules manage [project merge request approval rules](../project/merge_requests/approvals/index.md) -at the top-level group level. These rules [cascade to all projects](../project/merge_requests/approvals/settings.md#settings-cascading) +Group approval settings manage [project merge request approval settings](../project/merge_requests/approvals/settings.md) +at the top-level group level. These settings [cascade to all projects](../project/merge_requests/approvals/settings.md#settings-cascading) that belong to the group. -To view the merge request approval rules for a group: +To view the merge request approval settings for a group: 1. Go to the top-level group's **Settings > General** page. 1. Expand the **Merge request approvals** section. 1. Select the settings you want. 1. Select **Save changes**. +Support for group-level settings for merge request approval rules is tracked in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/4367). + ## Related topics - [Group wikis](../project/wiki/index.md) diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index b5912a0b40e..1c316f2157d 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -12,6 +12,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Moved to GitLab Premium in 13.9. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) in GitLab 14.6. [Feature flag `group_iterations`](https://gitlab.com/gitlab-org/gitlab/-/issues/221047) removed. +WARNING: +After [Iteration Cadences](#iteration-cadences) becomes generally available, +manual iteration scheduling will be [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) in GitLab 15.6. +To enhance the role of iterations as time boundaries, we will also deprecate the title field. + Iterations are a way to track issues over a period of time. This allows teams to track velocity and volatility metrics. Iterations can be used with [milestones](../../project/milestones/index.md) for tracking over different time periods. @@ -28,54 +33,6 @@ In GitLab, iterations are similar to milestones, with a few differences: - Iterations require both a start and an end date. - Iteration date ranges cannot overlap. -## Iteration cadences - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5077) in GitLab 14.1. -> - Deployed behind a [feature flag](../../feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-iteration-cadences). - -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. - -Iteration cadences automate some common iteration tasks. They can be used to -automatically create iterations every 1, 2, 3, 4, or 6 weeks. They can also -be configured to automatically roll over incomplete issues to the next iteration. - -With iteration cadences enabled, you must first -[create an iteration cadence](#create-an-iteration-cadence) before you can -[create an iteration](#create-an-iteration). - -### Create an iteration cadence - -Prerequisites: - -- You must have at least the Developer role for a group. - -To create an iteration cadence: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Issues > Iterations**. -1. Select **New iteration cadence**. -1. Fill out required fields, and select **Create iteration cadence**. The cadence list page opens. - -### Delete an iteration cadence - -Prerequisites: - -- You must have at least the Developer role for a group. - -Deleting an iteration cadence also deletes all iterations within that cadence. - -To delete an iteration cadence: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Issues > Iterations**. -1. Select the three-dot menu (**{ellipsis_v}**) > **Delete cadence** for the cadence you want to delete. -1. Select **Delete cadence** in the confirmation modal. - ## View the iterations list To view the iterations list, go to **{issues}** **Issues > Iterations**. @@ -84,12 +41,16 @@ From there you can create a new iteration or select an iteration to get a more d ## Create an iteration +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) in GitLab 14.10. + +WARNING: +Manual iteration management is in its end-of-life process. Creating an iteration is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) +in GitLab 14.10, and is planned for removal in GitLab 16.0. + Prerequisites: - You must have at least the Developer role for a group. -For manually scheduled iteration cadences, you create and add iterations yourself. - To create an iteration: 1. On the top bar, select **Menu > Groups** and find your group. @@ -100,7 +61,13 @@ To create an iteration: ## Edit an iteration -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218277) in GitLab 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218277) in GitLab 13.2. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) in GitLab 14.10. + +WARNING: +Editing all attributes, with the exception of `description` is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) +in GitLab 14.10, and is planned for removal in GitLab 16.0. +In the future only editing an iteration's `description` will be allowed. Prerequisites: @@ -110,7 +77,12 @@ To edit an iteration, select the three-dot menu (**{ellipsis_v}**) > **Edit**. ## Delete an iteration -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292268) in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292268) in GitLab 14.3. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) in GitLab 14.10. + +WARNING: +Manual iteration management is in its end-of-life process. Deleting an iteration is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/356069) +in GitLab 14.10, and is planned for removal in GitLab 16.0. Prerequisites: @@ -136,7 +108,7 @@ The report also shows a breakdown of total issues in an iteration. Open iteration reports show a summary of completed, unstarted, and in-progress issues. Closed iteration reports show the total number of issues completed by the due date. -To view an iteration report, go to the iterations list page and select an iteration's title. +To view an iteration report, go to the iterations list page and select an iteration's period. ### Iteration burndown and burnup charts @@ -195,33 +167,61 @@ To group issues by label: You can also search for labels by typing in the search input. 1. Select any area outside the label dropdown list. The page is now grouped by the selected labels. -### Enable or disable iteration cadences **(PREMIUM SELF)** +## Iteration cadences + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5077) in GitLab 14.1. +> - Deployed behind a [feature flag](../../feature_flags.md), named `iteration_cadences`, disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an +administrator to [enable the feature flag](../../../administration/feature_flags.md) named +`iteration_cadences` for a root group. +On GitLab.com, this feature is not available. This feature is not ready for production use. + +Iteration cadences automate iteration scheduling. You can use them to +automate creating iterations every 1, 2, 3, 4, or 6 weeks. You can also +configure iteration cadences to automatically roll over incomplete issues to the next iteration. + +### Create an iteration cadence + +Prerequisites: + +- You must have at least the Developer role for a group. + +To create an iteration cadence: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues > Iterations**. +1. Select **New iteration cadence**. +1. Fill out required fields, and select **Create iteration cadence**. The cadence list page opens. + +### Delete an iteration cadence + +Prerequisites: + +- You must have at least the Developer role for a group. -Iteration Cadences feature is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. +Deleting an iteration cadence also deletes all iterations within that cadence. -To enable it: +To delete an iteration cadence: -```ruby -Feature.enable(:iteration_cadences) -``` +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues > Iterations**. +1. Select the three-dot menu (**{ellipsis_v}**) > **Delete cadence** for the cadence you want to delete. +1. Select **Delete cadence** in the confirmation modal. -To disable it: +### Convert manual cadence to use automatic scheduling -```ruby -Feature.disable(:iteration_cadences) -``` +WARNING: +The upgrade is irreversible. After it's done, manual iteration cadences cannot be created. -<!-- ## Troubleshooting +When you **enable** the iteration cadences feature, all iterations are added +to a default iteration cadence. +In this default iteration cadence, you can continue to add, edit, and remove iterations. -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. +To upgrade the iteration cadence to use the automation features: -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Issues > Iterations**. +1. Select the three-dot menu (**{ellipsis_v}**) > **Edit cadence** for the cadence you want to upgrade. +1. Fill out required fields, and select **Save changes**. diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 8ebcd9f62d0..4d122e337db 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -23,12 +23,14 @@ If required, you can find [a glossary of common terms](../../../integration/saml ## Configure your identity provider -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Settings > SAML SSO**. -1. Configure your SAML identity provider using the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. - Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). +1. Find the information in GitLab required for configuration: + 1. On the top bar, select **Menu > Groups** and find your group. + 1. On the left sidebar, select **Settings > SAML SSO**. + 1. Note the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. +1. Configure your SAML identity provider app using the noted details. + Alternatively, GitLab provides a [metadata XML configuration](#metadata-configuration). See [specific identity provider documentation](#providers) for more details. -1. Configure the SAML response to include a NameID that uniquely identifies each user. +1. Configure the SAML response to include a [NameID](#nameid) that uniquely identifies each user. 1. Configure the required [user attributes](#user-attributes), ensuring you include the user's email address. 1. While the default is enabled for most SAML providers, please ensure the app is set to have service provider initiated calls in order to link existing GitLab accounts. @@ -245,7 +247,7 @@ To migrate users to a new email domain, users must: ## User access and management -> [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/268142) in GitLab 13.7. +> SAML user provisioning [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268142) in GitLab 13.7. Once Group SSO is configured and enabled, users can access the GitLab.com group through the identity provider's dashboard. If [SCIM](scim_setup.md) is configured, please see the [user access and linking setup section on the SCIM page](scim_setup.md#user-access-and-linking-setup). diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 331288e33a1..3960c97142e 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -20,7 +20,7 @@ The GitLab [SCIM API](../../../api/scim.md) implements part of [the RFC7644 prot The following actions are available: - Create users -- Deactivate users +- Remove users (deactivate SCIM identity) The following identity providers are supported: @@ -133,7 +133,7 @@ After the above steps are complete: The Okta GitLab application currently only supports SCIM. Continue using the separate Okta [SAML SSO](index.md) configuration along with the new SCIM -application described above. +application described above. An [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/216173) to add SAML support to the Okta GitLab application. ### OneLogin @@ -150,7 +150,7 @@ The following diagram is a general outline on what happens when you add users to ```mermaid graph TD - A[Add User to SCIM app] -->|IdP sends user info to GitLab| B(GitLab: Does the email exists?) + A[Add User to SCIM app] -->|IdP sends user info to GitLab| B(GitLab: Does the email exist?) B -->|No| C[GitLab creates user with SCIM identity] B -->|Yes| D[GitLab sends message back 'Email exists'] ``` diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index 6b20f1763d4..0666303bcf8 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -16,14 +16,17 @@ You can use a group access token to authenticate: - With the [GitLab API](../../../api/index.md#personalprojectgroup-access-tokens). - In [GitLab 14.2](https://gitlab.com/gitlab-org/gitlab/-/issues/330718) and later, authenticate with Git over HTTPS. + Use: -After you configure a group access token, you don't need a password when you authenticate. -Instead, you can enter any non-blank value. + - Any non-blank value as a username. + - The group access token as the password. Group access tokens are similar to [project access tokens](../../project/settings/project_access_tokens.md) and [personal access tokens](../../profile/personal_access_tokens.md), except they are associated with a group rather than a project or user. +In self-managed instances, group access tokens are subject to the same [maximum lifetime limits](../../admin_area/settings/account_and_limit_settings.md#limit-the-lifetime-of-personal-access-tokens) as personal access tokens if the limit is set. + You can use group access tokens: - On GitLab SaaS if you have the Premium license tier or higher. Group access tokens are not available with a [trial license](https://about.gitlab.com/free-trial/). @@ -33,6 +36,8 @@ You can use group access tokens: - Consider [disabling group access tokens](#enable-or-disable-group-access-token-creation) to lower potential abuse. +You cannot use group access tokens to create other access tokens. + Group access tokens inherit the [default prefix setting](../../admin_area/settings/account_and_limit_settings.md#personal-access-token-prefix) configured for personal access tokens. @@ -45,7 +50,7 @@ To create a group access token: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > Access Tokens**. 1. Enter a name. The token name is visible to any user with permissions to view the group. -1. Optional. Enter an expiry date for the token. The token will expire on that date at midnight UTC. +1. Optional. Enter an expiry date for the token. The token will expire on that date at midnight UTC. An instance-wide [maximum lifetime](../../admin_area/settings/account_and_limit_settings.md#limit-the-lifetime-of-personal-access-tokens) setting can limit the maximum allowable lifetime in self-managed instances. 1. Select a role for the token. 1. Select the [desired scopes](#scopes-for-a-group-access-token). 1. Select **Create group access token**. diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index a98f213bb3f..2cddbaa9723 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication and Authorization +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -49,9 +49,16 @@ graph TD ## Create a subgroup -Users with the at least the Maintainer role on a group can create subgroups immediately below the group, unless -[configured otherwise](#change-who-can-create-subgroups). These users can create subgroups even if group creation is -[disabled by an Administrator](../../admin_area/index.md#prevent-a-user-from-creating-groups) in the user's settings. +Prerequisites: + +- You must either: + - Have at least the Maintainer role for a group to create subgroups for it. + - Have the [role determined by a setting](#change-who-can-create-subgroups). These users can create + subgroups even if group creation is + [disabled by an Administrator](../../admin_area/index.md#prevent-a-user-from-creating-groups) in the user's settings. + +NOTE: +You cannot host a GitLab Pages subgroup website with a top-level domain name. For example, `subgroupname.example.io`. To create a subgroup: diff --git a/doc/user/group/value_stream_analytics/img/vsa_aggregated_data_toggle_v14_9.png b/doc/user/group/value_stream_analytics/img/vsa_aggregated_data_toggle_v14_9.png Binary files differnew file mode 100644 index 00000000000..5ad8026b8fd --- /dev/null +++ b/doc/user/group/value_stream_analytics/img/vsa_aggregated_data_toggle_v14_9.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png b/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png Binary files differdeleted file mode 100644 index 834556df051..00000000000 --- a/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png +++ /dev/null diff --git a/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png b/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png Binary files differdeleted file mode 100644 index 8d77c53db7f..00000000000 --- a/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png +++ /dev/null diff --git a/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png b/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png Binary files differdeleted file mode 100644 index 5dd79d06463..00000000000 --- a/doc/user/group/value_stream_analytics/img/vsa_path_nav_v13_11.png +++ /dev/null diff --git a/doc/user/group/value_stream_analytics/img/vsa_stage_table_v14_7.png b/doc/user/group/value_stream_analytics/img/vsa_stage_table_v14_7.png Binary files differdeleted file mode 100644 index 7c3305792bb..00000000000 --- a/doc/user/group/value_stream_analytics/img/vsa_stage_table_v14_7.png +++ /dev/null diff --git a/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png b/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png Binary files differdeleted file mode 100644 index 68d9741bed8..00000000000 --- a/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png +++ /dev/null diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 5ea8512ed5a..3fce9baa9ce 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -16,6 +16,7 @@ Use value stream analytics to identify: - The amount of time it takes to go from an idea to production. - The velocity of a given project. - Bottlenecks in the development process. +- Detecting long-running issues or merge requests. - Factors that cause your software development lifecycle to slow down. Value stream analytics is also available for [projects](../../analytics/value_stream_analytics.md). @@ -26,7 +27,10 @@ Value stream analytics is also available for [projects](../../analytics/value_st > - Filtering [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13216) in GitLab 13.3 > - Horizontal stage path [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12196) in 13.0 and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323982) in 13.12 -You must have at least the Reporter role to view value stream analytics for groups. +Prerequisite: + +- You must have at least the Reporter role to view value stream analytics for groups. +- You must create a [custom value stream](#custom-value-streams). Value stream analytics only shows custom value streams created for your group. To view value stream analytics for your group: @@ -37,6 +41,9 @@ To view value stream analytics for your group: 1. Select the **Filter results** text box. 1. Select a parameter. 1. Select a value or enter text to refine the results. + 1. Select whether to view metrics for items with a start or stop event: + - To view items with a stop event in the date range, turn on the **Filter by stop date** toggle. Enabled by default. + - To view items with a start event in the date range, turn off the **Filter by stop date** toggle. 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. The charts and list show workflow items created @@ -71,6 +78,9 @@ To view the median time spent in each stage by a group: 1. Select the **Filter results** text box. 1. Select a parameter. 1. Select a value or enter text to refine the results. + 1. Select whether to view metrics for items with a start or stop event: + - To view items with a stop event in the date range, turn on the **Filter by stop date** toggle. Enabled by default. + - To view items with a start event in the date range, turn off the **Filter by stop date** toggle. 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. @@ -91,6 +101,9 @@ To view the lead time and cycle time for issues: 1. Select the **Filter results** text box. 1. Select a parameter. 1. Select a value or enter text to refine the results. + 1. Select whether to view metrics for items with a start or stop event: + - To view items with a stop event in the date range, turn on the **Filter by stop date** toggle. Enabled by default. + - To view items with a start event in the date range, turn off the **Filter by stop date** toggle. 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. @@ -111,6 +124,9 @@ To view the lead time for changes for merge requests in your group: 1. Select the **Filter results** text box. 1. Select a parameter. 1. Select a value or enter text to refine the results. + 1. Select whether to view metrics for items with a start or stop event: + - To view items with a stop event in the date range, turn on the **Filter by stop date** toggle. Enabled by default. + - To view items with a start event in the date range, turn off the **Filter by stop date** toggle. 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. @@ -125,7 +141,7 @@ To view deployment metrics, you must have a [production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). Value stream analytics shows the following deployment metrics for your group: - + - Deploys: The number of successful deployments in the date range. - Deployment Frequency: The average number of successful deployments per day in the date range. @@ -137,6 +153,9 @@ To view deployment metrics for your group: 1. Select the **Filter results** text box. 1. Select a parameter. 1. Select a value or enter text to refine the results. + 1. Select whether to view metrics for items with a start or stop event: + - To view items with a stop event in the date range, turn on the **Filter by stop date** toggle. Enabled by default. + - To view items with a start event in the date range, turn off the **Filter by stop date** toggle. 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. @@ -150,25 +169,24 @@ NOTE: In GitLab 13.9 and later, metrics are calculated based on when the deployment was finished. In GitLab 13.8 and earlier, metrics are calculated based on when the deployment was created. -## Upcoming date filter change +### How value stream analytics aggregates data + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.5 [with a flag](../../../administration/feature_flags.md) named `use_vsa_aggregated_tables`. Disabled by default. +> - Filter by stop date toggle [added](https://gitlab.com/gitlab-org/gitlab/-/issues/352428) in GitLab 14.9 +> - Data refresh badge [added](https://gitlab.com/gitlab-org/gitlab/-/issues/341739) in GitLab 14.9 + +Plans for value stream analytics to filter items by stop event instead of start event are tracked in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/6046). With the completion of this work, value stream analytics will only display items with a stop event in the date range. -In the [epics](https://gitlab.com/groups/gitlab-org/-/epics/6046), we plan to alter -the date filter behavior to filter the end event time of the currently selected stage. +To preview this functionality, you can use the **Filter by stop date** toggle to enable or disable this filter until the [default filtering mode is introduced](../../../update/deprecations.md#value-stream-analytics-filtering-calculation-change) and the toggle is removed. -The change makes it possible to get a much better picture about the completed items within the -stage and helps uncover long-running items. +If you turn on the **Filter by stop date** toggle, the results show items with a stop event within the date range. When this function is enabled, it may take up to 10 minutes for results to show due to data aggregation. There are occasions when it may take longer than 10 minutes for results to display: -For example, an issue was created a year ago and the current stage was finished in the current month. -If you were to look at the metrics for the last three months, this issue would not be included in the calculation of -the stage metrics. With the new date filter, this item would be included. +- If this is the first time you are viewing value stream analytics and have not yet [created a value stream](#create-a-value-stream). +- If the group hierarchy has been re-arranged. +- If there have been bulk updates on issues and merge requests. -DISCLAIMER: -This section contains information related to upcoming products, features, and functionality. -It is important to note that the information presented is for informational purposes only. -Please do not rely on this information for purchasing or planning purposes. -As with all projects, the items mentioned on this page are subject to change or delay. -The development, release, and timing of any products, features, or functionality remain at the -sole discretion of GitLab Inc. +To view when the data was most recently updated, in the right corner next to **Edit**, hover over the **Last updated** badge. This badge is only available if you have turned on the **Filter by start date** toggle. + ## How value stream analytics measures stages @@ -177,7 +195,10 @@ Value stream analytics measures each stage from its start event to its end event For example, a stage might start when a user adds a label to an issue, and ends when they add another label. Items aren't included in the stage time calculation if they have not reached the end event. -Each stage of value stream analytics is further described in the table below. +Value stream analytics allows you to customize your stages based on pre-defined events. To make the +configuration easier, GitLab provides a pre-defined list of stages that can be used as a template + +Each pre-defined stages of value stream analytics is further described in the table below. | Stage | Measurement method | | ------- | -------------------- | @@ -188,21 +209,21 @@ Each stage of value stream analytics is further described in the table below. | Review | The median time taken to review a merge request that has a closing issue pattern, between its creation and until it's merged. | | Staging | The median time between merging a merge request that has a closing issue pattern until the very first deployment to a [production environment](#how-value-stream-analytics-identifies-the-production-environment). If there isn't a production environment, this is not tracked. | -## Example workflow +### Example workflow -This example shows a workflow through all seven stages in one day. +This example shows a workflow through all seven stages in one day. -If a stage does not include a start and a stop time, its data is not included in the median time. +If a stage does not include a start and a stop time, its data is not included in the median time. In this example, milestones have been created and CI/CD for testing and setting environments is configured. - 09:00: Create issue. **Issue** stage starts. -- 11:00: Add issue to a milestone, start work on the issue, and create a branch locally. - **Issue** stage stops and **Plan** stage starts. +- 11:00: Add issue to a milestone, start work on the issue, and create a branch locally. + **Issue** stage stops and **Plan** stage starts. - 12:00: Make the first commit. - 12:30: Make the second commit to the branch that mentions the issue number. **Plan** stage stops and **Code** stage starts. -- 14:00: Push branch and create a merge request that contains the - [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically). +- 14:00: Push branch and create a merge request that contains the + [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically). **Code** stage stops and **Test** and **Review** stages start. - GitLab CI/CD takes 5 minutes to run scripts defined in [`.gitlab-ci.yml`](../../../ci/yaml/index.md). - 19:00: Merge the merge request. **Review** stage stops and **Staging** stage starts. @@ -214,7 +235,7 @@ Value stream analytics records the following times for each stage: - **Plan**: 11:00 to 12:00: 1 hr - **Code**: 12:00 to 14:00: 2 hrs - **Test**: 5 minutes -- **Review**: 14:00 to 19:00: 5 hrs +- **Review**: 14:00 to 19:00: 5 hrs - **Staging**: 19:00 to 19:30: 30 minutes There are some additional considerations for this example: @@ -263,13 +284,16 @@ To create a value stream: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value Stream**. -1. In the top right, select the dropdown list and then **Create new Value Stream**. +1. If this is the first time you are creating a value stream, select **Create custom value stream**. Otherwise, in the top right, select the dropdown list and then **Create new Value Stream**. 1. Enter a name for the new Value Stream. - You can [customize the stages](#create-a-value-stream-with-stages). 1. Select **Create Value Stream**.  +NOTE: +If you have recently upgraded to GitLab Premium, it can take up to 30 minutes for data to collect and display. + ### Create a value stream with stages > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50229) in GitLab 13.7. @@ -283,7 +307,7 @@ To create a value stream with stages: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value Stream**. -1. In the top right, select the dropdown list and then **Create new Value Stream**. +1. If this is the first time you are creating a value stream, select **Create custom value stream**. Otherwise, in the top right, select the dropdown list and then **Create new Value Stream**. 1. Select either **Create from default template** or **Create from no template**. - You can hide or re-order default stages in the value stream. diff --git a/doc/user/index.md b/doc/user/index.md index 4ce46c5b46f..af106b85ce9 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -2,253 +2,20 @@ stage: none group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, index -description: 'Read through the GitLab User documentation to learn how to use, configure, and customize GitLab and GitLab.com to your own needs.' --- -# User Docs **(FREE)** - -Welcome to GitLab! We're glad to have you here! - -As a GitLab user you have access to all the features -your [subscription](https://about.gitlab.com/pricing/) -includes, except [GitLab administrator](../administration/index.md) -settings, unless you have administrator privileges to install, configure, -and upgrade your GitLab instance. - -Administrator privileges for [GitLab.com](https://gitlab.com/) are restricted to the GitLab team. - -For more information on configuring GitLab self-managed instances, see the [Administrator documentation](../administration/index.md). - -## Overview - -GitLab is a fully integrated software development platform that enables your team to be transparent, fast, effective, and cohesive from discussion on a new idea to production, all on the same platform. - -For more information, see [All GitLab Features](https://about.gitlab.com/features/). - -### Concepts - -To get familiar with the concepts needed to develop code on GitLab, read the following articles: - -- [Demo: Mastering Code Review With GitLab](https://about.gitlab.com/blog/2017/03/17/demo-mastering-code-review-with-gitlab/). -- [What is GitLab Flow?](https://about.gitlab.com/topics/version-control/what-is-gitlab-flow/). -- [Tutorial: It's all connected in GitLab](https://about.gitlab.com/blog/2016/03/08/gitlab-tutorial-its-all-connected/): an overview on code collaboration with GitLab. -- [Trends in Version Control Land: Microservices](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/). -- [Trends in Version Control Land: Innersourcing](https://about.gitlab.com/topics/version-control/what-is-innersource/). - -## Use cases - -GitLab is a Git-based platform that integrates a great number of essential tools for software development and deployment, and project management: - -- Hosting code in repositories with version control. -- Tracking proposals for new implementations, bug reports, and feedback with a - fully featured [Issue tracker](project/issues/index.md). -- Organizing and prioritizing with [issue boards](project/issue_board.md). -- Reviewing code in [merge requests](project/merge_requests/index.md) with live-preview changes per - branch with [Review Apps](../ci/review_apps/index.md). -- Building, testing, and deploying with built-in [Continuous Integration](../ci/index.md). -- View the current health and status of each CI environment running on Kubernetes with [deploy boards](project/deploy_boards.md). -- Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). -- Deploying personal and professional static websites with [GitLab Pages](project/pages/index.md). -- Integrating with Docker by using [GitLab Container Registry](packages/container_registry/index.md). -- Tracking the development lifecycle by using [GitLab Value Stream Analytics](analytics/value_stream_analytics.md). -- Provide support with [Service Desk](project/service_desk.md). -- [Export issues as CSV](project/issues/csv_export.md). - -With GitLab Enterprise Edition, you can also: - -- Improve collaboration with: - - [Merge request approvals](project/merge_requests/approvals/index.md). - - [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md). - - [Multiple issue boards](project/issue_board.md#multiple-issue-boards). -- Create formal relationships between issues with [linked issues](project/issues/related_issues.md). -- Use [Burndown Charts](project/milestones/burndown_and_burnup_charts.md) to track progress during a sprint or while working on a new version of their software. -- Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Search](search/advanced_search.md) for faster, more advanced code search across your entire GitLab instance. -- [Authenticate users with Kerberos](../integration/kerberos.md). -- [Mirror a repository](project/repository/mirror/index.md) from elsewhere on your local server. -- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](../ci/pipelines/multi_project_pipelines.md). -- [Lock files](project/file_lock.md) to prevent conflicts. -- Scan your code for vulnerabilities and [display them in merge requests](application_security/sast/index.md). - -You can also [integrate](project/integrations/overview.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, Trello, Slack, Bamboo CI, Jira, and a lot more. - -## User types - -There are several types of users in GitLab: - -- Regular users. -- [Internal users](../development/internal_users.md) often referred to as bot or system users. -- [Auditor](permissions.md#auditor-users) with read access to self-managed instances. -- [GitLab Administrator](../administration/index.md) with full access to - self-managed instances including settings and the [Admin Area](admin_area/index.md). - -Each user can be a member in a [group](group/index.md). - -See the [permissions page](permissions.md) for details on how each user type is used. - -## User activity - -GitLab tracks user contribution activity. -You can follow or unfollow other users from their [user profiles](profile/index.md#access-your-user-profile). -To view a user's activity in a top-level Activity view: - -1. From a user's profile, select **Follow**. -1. In the GitLab menu, select **Activity**. -1. Select the **Followed users** tab. - -### User contribution events - -Each of these contribution events is tracked: - -- `approved` - - Merge request -- `closed` - - [Epic](group/epics/index.md) - - Issue - - Merge request - - Milestone -- `commented` on any `Noteable` record. - - Alert - - Commit - - Design - - Issue - - Merge request - - Snippet -- `created` - - Design - - [Epic](group/epics/index.md) - - Issue - - Merge request - - Milestone - - Project - - Wiki page -- `destroyed` - - Design - - Milestone - - Wiki page -- `expired` - - Project membership -- `joined` - - Project membership -- `left` - - Project membership -- `merged` - - Merge request -- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. - - Project -- `reopened` - - [Epic](group/epics/index.md) - - Issue - - Merge request - - Milestone -- `updated` - - Design - - Wiki page - -## Projects - -In GitLab, you can create [projects](project/index.md) to host -your code, track issues, collaborate on code, and continuously -build, test, and deploy your app with built-in GitLab CI/CD. Or, you can do -it all at once, from one single project. - -- [Repositories](project/repository/index.md): Host your codebase in - repositories with version control and as part of a fully integrated platform. -- [Issues](project/issues/index.md): Explore the best of GitLab Issues' features. -- [Merge requests](project/merge_requests/index.md): Collaborate on code, - reviews, live preview changes per branch, and request approvals with merge requests. -- [Milestones](project/milestones/index.md): Work on multiple issues and merge - requests towards the same target date with Milestones. - -## Account - -There is a lot you can customize and configure to enjoy the best of GitLab. - -- [Settings](profile/index.md): Manage your user settings to change your personal information, - personal access tokens, authorized applications, etc. -- [Authentication](../topics/authentication/index.md): Read through the authentication - methods available in GitLab. -- [Permissions](permissions.md): Learn the different set of permissions levels for each - user type (guest, reporter, developer, maintainer, owner). -- [Feature highlight](feature_highlight.md): Learn more about the little blue dots - around the app that explain certain features. -- [Abuse reports](report_abuse.md): Report abuse from users to GitLab administrators. - -## Groups - -With GitLab [Groups](group/index.md) you can assemble related projects together -and grant members access to several projects at once. - -Groups can also be nested in [subgroups](group/subgroups/index.md). - -## Discussions - -In GitLab, you can comment and mention collaborators in issues, -merge requests, code snippets, and commits. - -When performing inline reviews to implementations -to your codebase through merge requests you can -gather feedback through [resolvable threads](discussions/index.md#resolve-a-thread). - -### GitLab Flavored Markdown (GFM) - -Read through the [GFM documentation](markdown.md) to learn how to apply -the best of GitLab Flavored Markdown in your threads, comments, -issues and merge requests descriptions, and everywhere else GFM is -supported. - -## To-Do List - -Never forget to reply to your collaborators. [GitLab To-Do List](todos.md) -is a tool for working faster and more effectively with your team, -by listing all user or group mentions, as well as issues and merge -requests you're assigned to. - -## Search - -[Search and filter](search/index.md) through groups, projects, issues, merge requests, files, code, and more. - -## Snippets - -[Snippets](snippets.md) are code blocks that you want to store in GitLab, from which -you have quick access to. You can also gather feedback on them through [Discussions](#discussions). - -## GitLab CI/CD - -Use built-in [GitLab CI/CD](../ci/index.md) to test, build, and deploy your applications -directly from GitLab. No third-party integrations needed. - -## Features behind feature flags - -Understand what [features behind feature flags](feature_flags.md) mean. - -## Keyboard shortcuts - -There are many [keyboard shortcuts](shortcuts.md) in GitLab to help you navigate between -pages and accomplish tasks faster. - -## Integrations - -[Integrate GitLab](../integration/index.md) with your preferred tool, such as Trello, Jira, etc. - -## Webhooks - -Configure [webhooks](project/integrations/webhooks.md) to listen for -specific events like pushes, issues or merge requests. GitLab sends a -POST request with data to the webhook URL. - -## API - -Automate GitLab via [API](../api/index.md). - -## Git and GitLab - -Learn what is [Git](../topics/git/index.md) and its best practices. - -## Instance-level analytics - -See [various statistics](admin_area/analytics/index.md) of your GitLab instance. - -## Operations Dashboard - -See [Operations Dashboard](operations_dashboard/index.md) for a summary of each project's operational health. +# Use GitLab **(FREE)** + +Get to know the GitLab end-to-end workflow. Configure permissions, +organize your work, create and secure your application, and analyze its performance. Report on team productivity throughout the process. + +- [Set up your organization](../topics/set_up_organization.md) +- [Organize work with projects](../user/project/index.md) +- [Plan and track work](../topics/plan_and_track.md) +- [Build your application](../topics/build_your_application.md) +- [Secure your application](../user/application_security/index.md) +- [Deploy and release your application](../topics/release_your_application.md) +- [Monitor application performance](../operations/index.md) +- [Monitor runner performance](https://docs.gitlab.com/runner/monitoring/index.html) +- [Manage your infrastructure](../user/infrastructure/index.md) +- [Analyze GitLab usage](../user/analytics/index.md) diff --git a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md index 87b8f510289..50899053cad 100644 --- a/doc/user/infrastructure/clusters/connect/new_eks_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_eks_cluster.md @@ -48,10 +48,13 @@ This project provides you with: ## Register the agent +FLAG: +In GitLab 14.10, a [flag](../../../../administration/feature_flags.md) named `certificate_based_clusters` changed the **Actions** menu to focus on the agent rather than certificates. The flag is [enabled on GitLab.com and self-managed](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). + To create a GitLab agent for Kubernetes: 1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select **Actions**. +1. Select **Connect a cluster (agent)**. 1. From the **Select an agent** dropdown list, select `eks-agent` and select **Register an agent**. 1. GitLab generates a registration token for the agent. Securely store this secret token, as you will need it later. 1. GitLab provides an address for the agent server (KAS), which you will also need later. diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 1ed8b0ef350..ab04187284d 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -6,6 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Create a Google GKE cluster +INFO: +Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial), +and in partnership with Google, GitLab is able to offer an additional $200 for new +GCP accounts to get started with the GitLab integration with Google Kubernetes Engine. +[Follow this link](https://cloud.google.com/partners/partnercredit/?pcn_code=0014M00001h35gDQAQ#contact-form) +and apply for credit. + Learn how to create a new cluster on Google Kubernetes Engine (GKE) through [Infrastructure as Code (IaC)](../../index.md). This process uses the Google and Kubernetes Terraform providers create GKE clusters. You connect the clusters to GitLab @@ -48,10 +55,13 @@ with defaults for name, location, node count, and Kubernetes version. ## Register the agent +FLAG: +In GitLab 14.10, a [flag](../../../../administration/feature_flags.md) named `certificate_based_clusters` changed the **Actions** menu to focus on the agent rather than certificates. The flag is [enabled on GitLab.com and self-managed](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). + To create a GitLab agent for Kubernetes: 1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select **Actions**. +1. Select **Connect a cluster (agent)**. 1. From the **Select an agent** dropdown list, select `gke-agent` and select **Register an agent**. 1. GitLab generates a registration token for the agent. Securely store this secret token, as you will need it later. 1. GitLab provides an address for the agent server (KAS), which you will also need later. diff --git a/doc/user/infrastructure/clusters/deploy/inventory_object.md b/doc/user/infrastructure/clusters/deploy/inventory_object.md index d76ef0b9359..d184d969ace 100644 --- a/doc/user/infrastructure/clusters/deploy/inventory_object.md +++ b/doc/user/infrastructure/clusters/deploy/inventory_object.md @@ -23,7 +23,7 @@ gitops: default_namespace: my-ns ``` -The agent creates an inventory object for every item in the `manifest_projects` list. +The agent creates an inventory object for every item in the `manifest_projects` list. The inventory object is stored in the namespace you specify for `default_namespace`. The name and location of the inventory object is based on: @@ -58,7 +58,7 @@ This action changes the location of the object in the cluster. inventory objects in the same namespace in the future. 1. Ensure the value for `cli-utils.sigs.k8s.io/inventory-id` is unique. This value is used for objects tracked by this inventory object. Their `config.k8s.io/owning-inventory` annotation is set to this value. - + The value doesn't have to match the `name` but it's convenient to set them to the same value. 1. Save the file with the manifest files as a single logical group. diff --git a/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md b/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md index 01530422e4a..61ec0a559f0 100644 --- a/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md +++ b/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md @@ -18,7 +18,7 @@ in GitLab 14.5. It is expected to be [turned off by default in 15.0](../../../update/deprecations.md#certificate-based-integration-with-kubernetes) and removed in GitLab 15.6. -If you are using the certificate-based integration, you should move to another workflow as soon as possible. +If you are using the certificate-based integration, you should move to another workflow as soon as possible. As a general rule, to migrate clusters that rely on GitLab CI/CD, you can use the [CI/CD workflow](../../clusters/agent/ci_cd_tunnel.md). @@ -60,7 +60,7 @@ To configure your Auto DevOps project to use the GitLab agent: 1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. 1. From the certificate-based clusters section, open the cluster that serves the same environment scope. 1. Select the **Details** tab and disable the cluster. -1. To activate the changes, on the left sidebar, select **CI/CD > Variables > Run pipeline**. +1. To activate the changes, on the left sidebar, select **CI/CD > Pipelines** and then **Run pipeline**. For an example, [view this project](https://gitlab.com/gitlab-examples/ops/gitops-demo/hello-world-service). @@ -70,7 +70,11 @@ Follow the process for the [CI/CD workflow](../../clusters/agent/ci_cd_tunnel.md ## Migrate from GitLab Managed applications -Follow the process to [migrate from GitLab Managed Apps to the cluster management project](../../clusters/migrating_from_gma_to_project_template.md). +[GitLab Managed Apps (GMA)](../../clusters/applications.md#gitlab-managed-apps-deprecated) were deprecated in GitLab 14.0, and +the agent for Kubernetes does not support them. To migrate from GMA to the agent, go through the following steps: + +1. [Migrate from GitLab Managed Apps to a cluster management project](../../clusters/migrating_from_gma_to_project_template.md). +1. [Migrate the cluster management project to use the agent](../../clusters/management_project_template.md). ## Migrate a cluster management project diff --git a/doc/user/infrastructure/iac/index.md b/doc/user/infrastructure/iac/index.md index 3bc7495d4be..bc7a3c0d069 100644 --- a/doc/user/infrastructure/iac/index.md +++ b/doc/user/infrastructure/iac/index.md @@ -6,110 +6,121 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Infrastructure as Code with Terraform and GitLab **(FREE)** -With Terraform in GitLab, you can use GitLab authentication and authorization with -your GitOps and Infrastructure-as-Code (IaC) workflows. -Use these features if you want to collaborate on Terraform code within GitLab or would like to use GitLab as a Terraform state storage that incorporates best practices out of the box. +To manage your infrastructure with GitLab, you can use the integration with +Terraform to define resources that you can version, reuse, and share: + +- Manage low-level components like compute, storage, and networking resources. +- Manage high-level components like DNS entries and SaaS features. +- Incorporate GitOps deployments and Infrastructure-as-Code (IaC) workflows. +- Use GitLab as a Terraform state storage. +- Store and use Terraform modules to simplify common and complex infrastructure patterns. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [a video overview](https://www.youtube.com/watch?v=iGXjUrkkzDI) of the features GitLab provides with the integration with Terraform. ## Integrate your project with Terraform > SAST test was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/6655) in GitLab 14.6. -In GitLab 14.0 and later, to integrate your project with Terraform, add the following -to your `.gitlab-ci.yml` file: +The integration with GitLab and Terraform happens through GitLab CI/CD. +Use an `include` attribute to add the Terraform template to your project and +customize from there. -```yaml -include: - - template: Terraform.latest.gitlab-ci.yml +To get started, choose the template that best suits your needs: -variables: - # If you do not use the GitLab HTTP backend, remove this line and specify TF_HTTP_* variables - TF_STATE_NAME: default - TF_CACHE_KEY: default - # If your terraform files are in a subdirectory, set TF_ROOT accordingly - # TF_ROOT: terraform/production -``` +- [Latest template](#latest-terraform-template) +- [Stable template and advanced template](#stable-and-advanced-terraform-templates) -The `Terraform.latest.gitlab-ci.yml` template: +All templates: -- Uses the latest [GitLab Terraform image](https://gitlab.com/gitlab-org/terraform-images). -- Uses the [GitLab-managed Terraform state](#gitlab-managed-terraform-state) as +- Use the [GitLab-managed Terraform state](#gitlab-managed-terraform-state) as the Terraform state storage backend. -- Creates [four pipeline stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml): - `test`, `validate`, `build`, and `deploy`. These stages - [run the Terraform commands](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.latest.gitlab-ci.yml) - `test`, `validate`, `plan`, `plan-json`, and `apply`. The `apply` command only runs on the default branch. -- Runs the [Terraform SAST scanner](../../application_security/iac_scanning/index.md#configure-iac-scanning-manually), - that you can disable by creating a `SAST_DISABLED` environment variable and setting it to `1`. +- Trigger four pipeline stages: `test`, `validate`, `build`, and `deploy`. +- Run Terraform commands: `test`, `validate`, `plan`, and `plan-json`. It also runs the `apply` only on the default branch. +- Run the [Terraform SAST scanner](../../application_security/iac_scanning/index.md#configure-iac-scanning-manually). -You can override the values in the default template by updating your `.gitlab-ci.yml` file. +### Latest Terraform template -The latest template might contain breaking changes between major GitLab releases. -For a more stable template, we recommend: +The [latest template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml) +is compatible with the most recent GitLab version. It provides the most recent +GitLab features, but can potentially include breaking changes. -- [A ready-to-use version](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) -- [A base template for customized setups](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml) +You can safely use the latest Terraform template: -This video from January 2021 walks you through all the GitLab Terraform integration features: +- If you use GitLab.com. +- If you use a self-managed instance updated with every new GitLab release. -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=iGXjUrkkzDI">Terraform with GitLab</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/iGXjUrkkzDI" frameborder="0" allowfullscreen="true"> </iframe> -</figure> +### Stable and advanced Terraform templates -## GitLab-managed Terraform state +If you use earlier versions of GitLab, you might face incompatibility errors +between the GitLab version and the template version. In this case, you can opt +to use one of these templates: + +- [The stable template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) with an skeleton that you can built on top of. +- [The advanced template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml) to fully customize your setup. + +### Use a Terraform template + +To use a Terraform template: -[Terraform remote backends](https://www.terraform.io/language/settings/backends) -enable you to store the state file in a remote, shared store. GitLab uses the -[Terraform HTTP backend](https://www.terraform.io/language/settings/backends/http) -to securely store the state files in local storage (the default) or -[the remote store of your choice](../../../administration/terraform_state.md). +1. On the top bar, select **Menu > Projects** and find the project you want to integrate with Terraform. +1. On the left sidebar, select **Repository > Files**. +1. Edit your `.gitlab-ci.yml` file, use the `include` attribute to fetch the Terraform template: -The GitLab-managed Terraform state backend can store your Terraform state easily and -securely. It spares you from setting up additional remote resources like -Amazon S3 or Google Cloud Storage. Its features include: + ```yaml + include: + # To fetch the latest template, use: + - template: Terraform.latest.gitlab-ci.yml + # To fetch the stable template, use: + - template: Terraform/Base.gitlab-ci.yml + # To fetch the advanced template, use: + - template: Terraform/Base.latest.gitlab-ci.yml + ``` -- Supporting encryption of the state file both in transit and at rest. -- Locking and unlocking state. -- Remote Terraform plan and apply execution. +1. Add the variables as described below: -Read more about setting up and [using GitLab-managed Terraform states](terraform_state.md). + ```yaml + variables: + TF_STATE_NAME: default + TF_CACHE_KEY: default + # If your terraform files are in a subdirectory, set TF_ROOT accordingly. For example: + # TF_ROOT: terraform/production + ``` + +1. (Optional) Override in your `.gitlab-ci.yaml` file the attributes present +in the template you fetched to customize your configuration. + +## GitLab-managed Terraform state + +Use the [GitLab-managed Terraform state](terraform_state.md) to store state +files in local storage or in a remote store of your choice. ## Terraform module registry -GitLab can be used as a [Terraform module registry](../../packages/terraform_module_registry/index.md) -to create and publish Terraform modules to a private registry specific to your -top-level namespace. +Use GitLab as a [Terraform module registry](../../packages/terraform_module_registry/index.md) +to create and publish Terraform modules to a private registry. ## Terraform integration in merge requests -Collaborating around Infrastructure as Code (IaC) changes requires both code changes -and expected infrastructure changes to be checked and approved. GitLab provides a -solution to help collaboration around Terraform code changes and their expected -effects using the merge request pages. This way users don't have to build custom -tools or rely on 3rd party solutions to streamline their IaC workflows. - -Read more on setting up and [using the merge request integrations](mr_integration.md). +Use the [Terraform integration in merge requests](mr_integration.md) +to collaborate on Terraform code changes and Infrastructure-as-Code +workflows. ## The GitLab Terraform provider -WARNING: +NOTE: The GitLab Terraform provider is released separately from GitLab. -We are working on migrating the GitLab Terraform provider for GitLab.com. - -You can use the [GitLab Terraform provider](https://github.com/gitlabhq/terraform-provider-gitlab) -to manage various aspects of GitLab using Terraform. The provider is an open source project, -owned by GitLab, where everyone can contribute. +We are working on migrating the GitLab Terraform provider to GitLab.com. -The [documentation of the provider](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs) -is available as part of the official Terraform provider documentation. +The [GitLab Terraform provider](https://github.com/gitlabhq/terraform-provider-gitlab) is a plugin for Terraform to facilitate +managing of GitLab resources such as users, groups, and projects. +Its documentation is available on [Terraform](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs). ## Create a new cluster through IaC - Learn how to [create a new cluster on Amazon Elastic Kubernetes Service (EKS)](../clusters/connect/new_eks_cluster.md). - Learn how to [create a new cluster on Google Kubernetes Engine (GKE)](../clusters/connect/new_gke_cluster.md). -## Troubleshooting +## Related topics -See the [troubleshooting](troubleshooting.md) documentation. +- [Terraform images](https://gitlab.com/gitlab-org/terraform-images). +- [Troubleshooting](troubleshooting.md) issues with GitLab and Terraform. diff --git a/doc/user/infrastructure/iac/mr_integration.md b/doc/user/infrastructure/iac/mr_integration.md index bcad5c9279a..0fea05a3f03 100644 --- a/doc/user/infrastructure/iac/mr_integration.md +++ b/doc/user/infrastructure/iac/mr_integration.md @@ -23,7 +23,7 @@ recommend encrypting plan output or modifying the project visibility settings. ## Configure Terraform report artifacts -GitLab ships with a [pre-built CI template](index.md#integrate-your-project-with-terraform) that uses GitLab Managed Terraform state and integrates Terraform changes into merge requests. We recommend customizing the pre-built image and relying on the `gitlab-terraform` helper provided within for a quick setup. +GitLab [integrates with Terraform](index.md#integrate-your-project-with-terraform) through CI/CD templates that use GitLab-managed Terraform state and display Terraform changes on merge requests. We recommend customizing the pre-built image and relying on the `gitlab-terraform` helper provided within for a quick setup. To manually configure a GitLab Terraform Report artifact: diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index 39a57b60787..60f97f522cf 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -454,29 +454,6 @@ query ProjectTerraformStates { For those new to the GitLab GraphQL API, read [Getting started with GitLab GraphQL API](../../../api/graphql/getting_started.md). -## Troubleshooting +## Related topics -### Unable to lock Terraform state files in CI jobs for `terraform apply` using a plan created in a previous job - -When passing `-backend-config=` to `terraform init`, Terraform persists these values inside the plan -cache file. This includes the `password` value. - -As a result, to create a plan and later use the same plan in another CI job, you might get the error -`Error: Error acquiring the state lock` errors when using `-backend-config=password=$CI_JOB_TOKEN`. -This happens because the value of `$CI_JOB_TOKEN` is only valid for the duration of the current job. - -As a workaround, use [http backend configuration variables](https://www.terraform.io/docs/language/settings/backends/http.html#configuration-variables) in your CI job, -which is what happens behind the scenes when following the -[Get started using GitLab CI](#get-started-using-gitlab-ci) instructions. - -### Error: "address": required field is not set - -By default, we set `TF_ADDRESS` to `${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}`. -If you don't set `TF_STATE_NAME` or `TF_ADDRESS` in your job, the job fails with the error message -`Error: "address": required field is not set`. - -To resolve this, ensure that either `TF_ADDRESS` or `TF_STATE_NAME` is accessible in the -job that returned the error: - -1. Configure the [CI/CD environment scope](../../../ci/variables/#add-a-cicd-variable-to-a-project) for the job. -1. Set the job's [environment](../../../ci/yaml/#environment), matching the environment scope from the previous step. +- [Troubleshooting GitLab-managed Terraform state](troubleshooting.md). diff --git a/doc/user/infrastructure/iac/troubleshooting.md b/doc/user/infrastructure/iac/troubleshooting.md index ecefa20db99..bc0aa39bc70 100644 --- a/doc/user/infrastructure/iac/troubleshooting.md +++ b/doc/user/infrastructure/iac/troubleshooting.md @@ -66,3 +66,30 @@ with better Terraform-specific names. To resolve the syntax error, you can: my-Terraform-job: extends: .terraform:init # The updated name. ``` + +## Troubleshooting Terraform state + +### Unable to lock Terraform state files in CI jobs for `terraform apply` using a plan created in a previous job + +When passing `-backend-config=` to `terraform init`, Terraform persists these values inside the plan +cache file. This includes the `password` value. + +As a result, to create a plan and later use the same plan in another CI job, you might get the error +`Error: Error acquiring the state lock` errors when using `-backend-config=password=$CI_JOB_TOKEN`. +This happens because the value of `$CI_JOB_TOKEN` is only valid for the duration of the current job. + +As a workaround, use [http backend configuration variables](https://www.terraform.io/docs/language/settings/backends/http.html#configuration-variables) in your CI job, +which is what happens behind the scenes when following the +[Get started using GitLab CI](terraform_state.md#get-started-using-gitlab-ci) instructions. + +### Error: "address": required field is not set + +By default, we set `TF_ADDRESS` to `${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${TF_STATE_NAME}`. +If you don't set `TF_STATE_NAME` or `TF_ADDRESS` in your job, the job fails with the error message +`Error: "address": required field is not set`. + +To resolve this, ensure that either `TF_ADDRESS` or `TF_STATE_NAME` is accessible in the +job that returned the error: + +1. Configure the [CI/CD environment scope](../../../ci/variables/#add-a-cicd-variable-to-a-project) for the job. +1. Set the job's [environment](../../../ci/yaml/#environment), matching the environment scope from the previous step. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index c81fdc275d9..fc2f1de5ce2 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -4,7 +4,9 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Flavored Markdown **(FREE)** +# GitLab Flavored Markdown (GLFM) **(FREE)** + +> The abbreviation [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/24592) from `GFM` to `GLFM` in GitLab 14.10. GitLab automatically renders Markdown content. For example, when you add a comment to an issue, you type the text in the Markdown language. When you save the issue, the text is rendered @@ -525,6 +527,7 @@ GitLab Flavored Markdown recognizes the following: | merge request | `!123` | `namespace/project!123` | `project!123` | | snippet | `$123` | `namespace/project$123` | `project$123` | | [epic](group/epics/index.md) | `&123` | `group1/subgroup&123` | | +| [iteration](group/iterations/index.md) | `*iteration:"iteration title"`| | | | [vulnerability](application_security/vulnerabilities/index.md) <sup>1</sup> | `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | | feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | | label by ID | `~123` | `namespace/project~123` | `project~123` | @@ -819,7 +822,8 @@ Regardless of the tag names, the relative order of the reference tags determines numbering. <!-- -Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLinks test. +The following codeblock uses HTML to skip the Vale ReferenceLinks test. +Do not change it back to a markdown codeblock. --> <pre class="highlight"><code>A footnote reference tag looks like this: [^1] @@ -926,7 +930,8 @@ ___ Examples: <!-- -Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLinks test. +The following codeblock uses HTML to skip the Vale ReferenceLinks test. +Do not change it back to a markdown codeblock. --> <pre class="highlight"><code>Inline-style (hover to see title text): @@ -1192,17 +1197,18 @@ A new line due to the previous backslash. You can create links two ways: inline-style and reference-style. For example: <!-- -Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLinks test. +The following codeblock uses HTML to skip the Vale ReferenceLinks test. +Do not change it back to a markdown codeblock. --> <pre class="highlight"><code>- This line shows an [inline-style link](https://www.google.com) -- This line shows a [link to a repository file in the same directory](index.md) -- This line shows a [relative link to a readme one directory higher](../index.md) +- This line shows a [link to a repository file in the same directory](permissions.md) +- This line shows a [relative link to a file one directory higher](../index.md) - This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") Using header ID anchors: -- This line links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) +- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-features-permissions) - This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) Using references: @@ -1219,13 +1225,13 @@ Some text to show that the reference links can follow later. </code></pre> - This line shows an [inline-style link](https://www.google.com) -- This line shows a [link to a repository file in the same directory](index.md) -- This line shows a [relative link to a README one directory higher](../index.md) +- This line shows a [link to a repository file in the same directory](permissions.md) +- This line shows a [relative link to a file one directory higher](../index.md) - This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") Using header ID anchors: -- This line links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) +- This line links to [a section on a different Markdown page, using a "#" and the header ID](permissions.md#project-features-permissions) - This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) Using references: @@ -1406,6 +1412,16 @@ while the equation for the theory of relativity is E = mc<sup>2</sup>. <!-- vale gitlab.Spelling = YES --> +### Keyboard HTML tag + +The `<kbd>` element is used to identify text that represents user keyboard input. Text surrounded by `<kbd>` tags is typically displayed in the browser's default monospace font. + +```html +Press <kbd>Enter</kbd> to go to the next page. +``` + +Press <kbd>Enter</kbd> to go to the next page. + ### Tables Tables are not part of the core Markdown spec, but they are part of GitLab Flavored Markdown. diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index ea12b225717..901fb740717 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -171,6 +171,8 @@ When you publish: ## Install a Composer package +> Authorization to [download a package archive](../../../api/packages/composer.md#download-a-package-archive) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331601) in GitLab 14.10. + Install a package from the Package Registry so you can use it as a dependency. Prerequisites: @@ -354,6 +356,8 @@ used to access them: ## Troubleshooting +### Caching + To improve performance, Composer caches files related to a package. Note that Composer doesn't remove data by itself. The cache grows as new packages are installed. If you encounter issues, clear the cache with this command: @@ -362,6 +366,14 @@ this command: composer clearcache ``` +### Authorization requirement when using `composer install` + +In GitLab 14.9 and earlier, you did not require authorization to use `composer install` if you already had a generated `composer.lock`. +If you committed your `composer.lock`, you could do a `composer install` in CI without setting up credentials. + +In GitLab 14.10 and later, authorization is required for the [downloading a package archive](../../../api/packages/composer.md#download-a-package-archive) endpoint. +If you encounter a credentials prompt when you are using `composer install`, follow the instructions in the [install a composer package](#install-a-composer-package) section to create an `auth.json` file. + ## Supported CLI commands The GitLab Composer repository supports the following Composer CLI commands: diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 731ba04a9f7..b3eadc13772 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -428,3 +428,25 @@ The GitLab Conan repository supports the following Conan CLI commands: packages you have permission to view. - `conan info`: View the information on a given package from the Package Registry. - `conan remove`: Delete the package from the Package Registry. + +## Troubleshooting + +### Make output verbose + +For more verbose output when troubleshooting a Conan issue: + +```shell +export CONAN_TRACE_FILE=/tmp/conan_trace.log # Or SET in windows +conan <command> +``` + +You can find more logging tips in the [Conan documentation](https://docs.conan.io/en/latest/mastering/logging.html). + +### SSL Errors + +If you are using a self-signed certificate, there are two methods to manage SSL errors with Conan: + +- Use the `conan remote` command to disable the SSL verification. +- Append your server `crt` file to the `cacert.pem` file. + +Read more about this in the [Conan Documentation](https://docs.conan.io/en/latest/howtos/use_tls_certificates.html). diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 5b6c71d4dd2..59bb4a89b0b 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -626,13 +626,18 @@ Use your own URLs to complete the following steps: docker pull gitlab.example.com/org/build/sample_project/cr:v2.9.1 ``` +NOTE: +For container registry authentication, use either a +[personal access token](../../profile/personal_access_tokens.md) or a +[deploy token](../../project/deploy_tokens/index.md). + 1. Rename the images to match the new project name: ```shell docker tag gitlab.example.com/org/build/sample_project/cr:v2.9.1 gitlab.example.com/new_org/build/new_sample_project/cr:v2.9.1 ``` -1. Delete the images in both projects by using the [UI](#delete-images) or [API](../../../api/packages.md#delete-a-project-package). +1. Delete the images in the old project by using the [UI](#delete-images) or [API](../../../api/packages.md#delete-a-project-package). There may be a delay while the images are queued and deleted. 1. Change the path or transfer the project by going to **Settings > General** and expanding **Advanced**. @@ -698,7 +703,7 @@ Follow [this issue](https://gitlab.com/gitlab-org/container-registry/-/issues/55 ### Tags temporarily cannot be marked for deletion -GitLab is [migrating to the next generation of the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/5523). -During the migration, you may encounter difficulty deleting tags. -If you encounter an error, it's likely that your image repository is in the process of being migrated. +GitLab is [migrating to the next generation of the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/5523). +During the migration, you may encounter difficulty deleting tags. +If you encounter an error, it's likely that your image repository is in the process of being migrated. Please wait a few minutes and try again. diff --git a/doc/user/packages/container_registry/reduce_container_registry_storage.md b/doc/user/packages/container_registry/reduce_container_registry_storage.md index 7e8b2865b6e..2cfe99876fa 100644 --- a/doc/user/packages/container_registry/reduce_container_registry_storage.md +++ b/doc/user/packages/container_registry/reduce_container_registry_storage.md @@ -79,7 +79,7 @@ the Container Registry after the policy runs. The next time the policy runs, the so it may take multiple runs for all tags to be deleted. WARNING: -GitLab self-managed installs support for third-party container registries that comply with the +GitLab self-managed installations support third-party container registries that comply with the [Docker Registry HTTP API V2](https://docs.docker.com/registry/spec/api/) specification. However, this specification does not include a tag delete operation. Therefore, when interacting with third-party container registries, GitLab uses a workaround to delete tags. See the @@ -217,7 +217,23 @@ Valid values for `cadence` when using the API are: - `1month` (every month) - `3month` (every quarter) -See the API documentation for further details: [Edit project](../../../api/projects.md#edit-project). +Valid values for `keep_n` (number of tags kept per image name) when using the API are: + +- `1` +- `5` +- `10` +- `25` +- `50` +- `100` + +Valid values for `older_than` (days until tags are automatically removed) when using the API are: + +- `7d` +- `14d` +- `30d` +- `90d` + +See the API documentation for further details: [Edit project API](../../../api/projects.md#edit-project). ### Use with external container registries diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index e431d4d7de3..5e66c8ed7a5 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -96,6 +96,14 @@ You can authenticate using: Users accessing the Dependency Proxy with a personal access token or username and password must have at least the Guest role for the group they pull images from. +The Dependency Proxy follows the [Docker v2 token authentication flow](https://docs.docker.com/registry/spec/auth/token/), +issuing the client a JWT to use for the pull requests. The JWT issued as a result of authenticating +expires after some time. When the token expires, most Docker clients store your credentials and +automatically request a new token without further action. + +The token expiration time is a [configurable setting](../../../administration/packages/dependency_proxy.md#changing-the-jwt-expiration). +On GitLab.com, the expiration time is 15 minutes. + #### SAML SSO When [SSO enforcement](../../group/saml_sso/index.md#sso-enforcement) @@ -132,7 +140,8 @@ There are other additional predefined CI/CD variables you can also use: - `CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX`: the image prefix for pulling images through the dependency proxy from the direct group or subgroup that the project belongs to. -`CI_DEPENDENCY_PROXY_SERVER` and `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX` +`CI_DEPENDENCY_PROXY_SERVER`, `CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX`, and +`CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX` include the server port. If you explicitly include the Dependency Proxy path, the port must be included, unless you have logged into the Dependency Proxy manually without including the port: diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 6e3af6a92d5..6a515b78fc1 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -283,7 +283,8 @@ To authenticate to the Package Registry, you need either a personal access token ### Authenticate with a personal access token in Gradle -Create a file `~/.gradle/gradle.properties` with the following content: +In [your `GRADLE_USER_HOME` directory](https://docs.gradle.org/current/userguide/directory_layout.html#dir:gradle_user_home), +create a file `gradle.properties` with the following content: ```groovy gitLabPrivateToken=REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN @@ -586,7 +587,7 @@ To publish a package by using Gradle: url "https://gitlab.example.com/api/v4/projects/<PROJECT_ID>/packages/maven" credentials(HttpHeaderCredentials) { name = "Private-Token" - value = gitLabPrivateToken // the variable resides in ~/.gradle/gradle.properties + value = gitLabPrivateToken // the variable resides in $GRADLE_USER_HOME/gradle.properties } authentication { header(HttpHeaderAuthentication) @@ -820,7 +821,7 @@ rm -rf ~/.m2/repository If you're using Gradle, run this command to clear the cache: ```shell -rm -rf ~/.gradle/caches +rm -rf ~/.gradle/caches # Or replace ~/.gradle with your custom GRADLE_USER_HOME ``` ### Review network trace logs diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index b980f6a5694..1f278bd1476 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -114,6 +114,26 @@ You can also remove the Package Registry for your project specifically: The **Packages & Registries > Package Registry** entry is removed from the sidebar. +## Package Registry visibility permissions + +[Project-level permissions](../../permissions.md) +determine actions such as downloading, pushing, or deleting packages. + +The visibility of the Package Registry is independent of the repository and can't be controlled from +your project's settings. For example, if you have a public project and set the repository visibility +to **Only Project Members**, the Package Registry is then public. However, disabling the Package +Registry disables all Package Registry operations. + +[GitLab-#329253](https://gitlab.com/gitlab-org/gitlab/-/issues/329253) +proposes adding the ability to control Package Registry visibility from the UI. + +| | | Anonymous<br/>(everyone on internet) | Guest | Reporter, Developer, Maintainer, Owner | +| -------------------- | --------------------- | --------- | ----- | ------------------------------------------ | +| Public project with Package Registry enabled | View Package Registry <br/> and pull packages | Yes | Yes | Yes | +| Internal project with Package Registry enabled | View Package Registry <br/> and pull packages | No | Yes | Yes | +| Private project with Package Registry enabled | View Package Registry <br/> and pull packages | No | No | Yes | +| Any project with Package Registry disabled | All operations on Package Registry | No | No | No | + ## Supported package managers WARNING: diff --git a/doc/user/permissions.md b/doc/user/permissions.md index c90f575e83a..2282a7d876e 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -54,155 +54,158 @@ The following table lists project permissions available for each role: <!-- Keep this table sorted: By topic first, then by minimum role, then alphabetically. --> -| Action | Guest | Reporter | Developer | Maintainer | Owner | -|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------|-----------|------------|----------| -| [Analytics](analytics/index.md):<br>View issue analytics | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View value stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [CI/CD analytics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View [threats list](application_security/threat_monitoring/index.md#threat-monitoring) | | | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | -| [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>View clusters | | | ✓ | ✓ | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete cleanup policies | | | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Push an image to the Container Registry | | | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (*20*) | ✓ (*20*) | ✓ | ✓ | ✓ | -| [Container Registry](packages/container_registry/index.md):<br>Remove a Container Registry image | | | ✓ | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | -| [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*16*) | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | -| [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md) | | | | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Add Labels | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Assign | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Create (*18*) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Create [confidential issues](project/issues/confidential_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View [Design Management](project/issues/design_management.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View related issues | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Set weight | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Close / reopen (*19*) | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Manage related issues | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Move issues (*14*) | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Set issue [time tracking](project/time_tracking.md) estimate and time spent | | ✓ | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | -| [Issues](project/issues/index.md):<br>Delete | | | | | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View allowed and denied licenses | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View License Compliance reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>View License list | | ✓ | ✓ | ✓ | ✓ | -| [License Compliance](compliance/license_compliance/index.md):<br>Manage license policy | | | | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Assign reviewer | | ✓ | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>See list | | ✓ | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Approve (*8*) | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Create (*17*) | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Manage merge approval rules (project settings) | | | | ✓ | ✓ | -| [Merge requests](project/merge_requests/index.md):<br>Delete | | | | | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Manage user-starred metrics dashboards (*6*) | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | -| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Pull a package | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Publish a package | | | ✓ | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Delete a package | | | | ✓ | ✓ | -| [Package registry](packages/index.md):<br>Delete a file associated with a package | | | | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>View [Error Tracking](../operations/error_tracking.md) list | | ✓ | ✓ | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>Manage [Feature Flags](../operations/feature_flags.md) | | | ✓ | ✓ | ✓ | -| [Project operations](../operations/index.md):<br>Manage [Error Tracking](../operations/error_tracking.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Reposition comments on images (posted by any user) | ✓ (*9*) | ✓ (*9*) | ✓ (*9*) | ✓ | ✓ | -| [Projects](project/index.md):<br>View Insights | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [releases](project/releases/index.md) | ✓ (*5*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View Requirements | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [time tracking](project/time_tracking.md) reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [wiki](project/wiki/index.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create [snippets](snippets.md) | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage labels | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View [project traffic statistics](../api/project_statistics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create, edit, delete [milestones](project/milestones/index.md). | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Create, edit, delete [releases](project/releases/index.md) | | | ✓ (*12*) | ✓ (*12*) | ✓ (*12*) | -| [Projects](project/index.md):<br>Create, edit [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Enable Review Apps | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View project [Audit Events](../administration/audit_events.md) | | | ✓ (*10*) | ✓ | ✓ | -| [Projects](project/index.md):<br>Add deploy keys | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Add new team members | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Change [project features visibility](../public_access/public_access.md) level | | | | ✓ (*13*) | ✓ | -| [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) (*11*) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (*7*) | ✓ (*7*) | -| [Projects](project/index.md):<br>View 2FA status of members | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Administer project compliance frameworks | | | | | ✓ | -| [Projects](project/index.md):<br>Archive project | | | | | ✓ | -| [Projects](project/index.md):<br>Change project visibility level | | | | | ✓ | -| [Projects](project/index.md):<br>Delete project | | | | | ✓ | -| [Projects](project/index.md):<br>Disable notification emails | | | | | ✓ | -| [Projects](project/index.md):<br>Transfer project to another namespace | | | | | ✓ | -| [Repository](project/repository/index.md):<br>Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>View a commit status | | ✓ | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Add tags | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Create new branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Create or update commit status | | | ✓ (*4*) | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Force push to non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Push to non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Remove non-protected branches | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Rewrite or remove Git tags | | | ✓ | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Enable or disable branch protection | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Enable or disable tag protection | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Manage [push rules](project/repository/push_rules.md) | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Push to protected branches (*4*) | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Turn on or off protected branch push for developers | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Remove fork relationship | | | | | ✓ | -| [Repository](project/repository/index.md):<br>Force push to protected branches (*3*) | | | | | | -| [Repository](project/repository/index.md):<br>Remove protected branches (*3*) | | | | | | -| [Requirements Management](project/requirements/index.md):<br>Archive / reopen | | ✓ | ✓ | ✓ | ✓ | -| [Requirements Management](project/requirements/index.md):<br>Create / edit | | ✓ | ✓ | ✓ | ✓ | -| [Requirements Management](project/requirements/index.md):<br>Import / export | | ✓ | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability | | | ✓ | ✓ | ✓ | -| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | -| [Terraform](infrastructure/index.md):<br>Read Terraform state | | | ✓ | ✓ | ✓ | -| [Terraform](infrastructure/index.md):<br>Manage Terraform state | | | | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Archive | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Create | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Move | | ✓ | ✓ | ✓ | ✓ | -| [Test cases](../ci/test_cases/index.md):<br>Reopen | | ✓ | ✓ | ✓ | ✓ | +| Action | Guest | Reporter | Developer | Maintainer | Owner | +|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------|-----------|------------|----------| +| [Analytics](analytics/index.md):<br>View [issue analytics](analytics/issue_analytics.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [merge request analytics](analytics/merge_request_analytics.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [value stream analytics](analytics/value_stream_analytics.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [CI/CD analytics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View [threats list](application_security/threat_monitoring/index.md#threat-monitoring) | | | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | +| [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>View clusters | | | ✓ | ✓ | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete [cleanup policies](packages/container_registry/index.md#delete-images-by-using-a-cleanup-policy) | | | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Push an image to the Container Registry | | | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Pull an image from the Container Registry | ✓ (*20*) | ✓ (*20*) | ✓ | ✓ | ✓ | +| [Container Registry](packages/container_registry/index.md):<br>Remove a Container Registry image | | | ✓ | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>View Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | +| [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*16*) | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md) | | | | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Add Labels | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Assign | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Create (*18*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Create [confidential issues](project/issues/confidential_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View [Design Management](project/issues/design_management.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View [related issues](project/issues/related_issues.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Set [weight](project/issues/issue_weight.md) | ✓ (*15*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>View [confidential issues](project/issues/confidential_issues.md) | (*2*) | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Close / reopen (*19*) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Lock threads | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Manage [related issues](project/issues/related_issues.md) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Manage tracker | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Move issues (*14*) | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Set issue [time tracking](project/time_tracking.md) estimate and time spent | | ✓ | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Archive [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | +| [Issues](project/issues/index.md):<br>Delete | | | | | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View allowed and denied licenses | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View License Compliance reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>View License list | | ✓ | ✓ | ✓ | ✓ | +| [License Compliance](compliance/license_compliance/index.md):<br>Manage license policy | | | | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Assign reviewer | | ✓ | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>See list | | ✓ | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Apply code change suggestions | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Approve (*8*) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Assign | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Create (*17*) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Add labels | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Lock threads | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Manage or accept | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>[Resolve a thread](discussions/#resolve-a-thread) | | | ✓ | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Manage [merge approval rules](project/merge_requests/approvals/settings.md) (project settings) | | | | ✓ | ✓ | +| [Merge requests](project/merge_requests/index.md):<br>Delete | | | | | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Manage user-starred metrics dashboards (*6*) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | +| [Metrics dashboards](../operations/metrics/dashboards/index.md):<br>Create/edit/delete metrics dashboard annotations | | | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Pull a package | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Publish a package | | | ✓ | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Delete a package | | | | ✓ | ✓ | +| [Package registry](packages/index.md):<br>Delete a file associated with a package | | | | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>View [Error Tracking](../operations/error_tracking.md) list | | ✓ | ✓ | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>Manage [Feature Flags](../operations/feature_flags.md) | | | ✓ | ✓ | ✓ | +| [Project operations](../operations/index.md):<br>Manage [Error Tracking](../operations/error_tracking.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Reposition comments on images (posted by any user) | ✓ (*9*) | ✓ (*9*) | ✓ (*9*) | ✓ | ✓ | +| [Projects](project/index.md):<br>View [Insights](project/insights/index.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [releases](project/releases/index.md) | ✓ (*5*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [Requirements](project/requirements/index.md) | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [time tracking](project/time_tracking.md) reports | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [wiki](project/wiki/index.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create [snippets](snippets.md) | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage labels | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [project traffic statistics](../api/project_statistics.md) | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create, edit, delete [milestones](project/milestones/index.md). | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Create, edit, delete [releases](project/releases/index.md) | | | ✓ (*12*) | ✓ (*12*) | ✓ (*12*) | +| [Projects](project/index.md):<br>Create, edit [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Enable [Review Apps](../ci/review_apps/index.md) | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View project [Audit Events](../administration/audit_events.md) | | | ✓ (*10*) | ✓ | ✓ | +| [Projects](project/index.md):<br>Add [deploy keys](project/deploy_keys/index.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Add new [team members](project/members/index.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Change [project features visibility](public_access.md) level | | | | ✓ (*13*) | ✓ | +| [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Export project | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage [project access tokens](project/settings/project_access_tokens.md) (*11*) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Manage [Project Operations](../operations/index.md) | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Rename project | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Share (invite) projects with groups | | | | ✓ (*7*) | ✓ (*7*) | +| [Projects](project/index.md):<br>View 2FA status of members | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Assign project to a [compliance framework](project/settings/index.md#compliance-frameworks) | | | | | ✓ | +| [Projects](project/index.md):<br>Archive project | | | | | ✓ | +| [Projects](project/index.md):<br>Change project visibility level | | | | | ✓ | +| [Projects](project/index.md):<br>Delete project | | | | | ✓ | +| [Projects](project/index.md):<br>Disable notification emails | | | | | ✓ | +| [Projects](project/index.md):<br>Transfer project to another namespace | | | | | ✓ | +| [Projects](project/index.md): View [Usage Quotas](usage_quotas.md) page | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>View a commit status | | ✓ | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Add tags | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Create new branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Create or update commit status | | | ✓ (*4*) | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Force push to non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Push to non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Remove non-protected branches | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Rewrite or remove Git tags | | | ✓ | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Enable or disable branch protection | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Enable or disable tag protection | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Manage [push rules](project/repository/push_rules.md) | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Push to protected branches (*4*) | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Turn on or off protected branch push for developers | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Remove fork relationship | | | | | ✓ | +| [Repository](project/repository/index.md):<br>Force push to protected branches (*3*) | | | | | | +| [Repository](project/repository/index.md):<br>Remove protected branches (*3*) | | | | | | +| [Requirements Management](project/requirements/index.md):<br>Archive / reopen | | ✓ | ✓ | ✓ | ✓ | +| [Requirements Management](project/requirements/index.md):<br>Create / edit | | ✓ | ✓ | ✓ | ✓ | +| [Requirements Management](project/requirements/index.md):<br>Import / export | | ✓ | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Create issue from vulnerability finding | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Create vulnerability from vulnerability finding | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Dismiss vulnerability finding | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Resolve vulnerability | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Revert vulnerability to detected state | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>Use security dashboard | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability | | | ✓ | ✓ | ✓ | +| [Security dashboard](application_security/security_dashboard/index.md):<br>View vulnerability findings in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | +| [Terraform](infrastructure/index.md):<br>Read Terraform state | | | ✓ | ✓ | ✓ | +| [Terraform](infrastructure/index.md):<br>Manage Terraform state | | | | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Archive | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Create | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Move | | ✓ | ✓ | ✓ | ✓ | +| [Test cases](../ci/test_cases/index.md):<br>Reopen | | ✓ | ✓ | ✓ | ✓ | <!-- markdownlint-disable MD029 --> @@ -224,7 +227,7 @@ The following table lists project permissions available for each role: supported on GitLab SaaS Premium and above (excluding [trial licenses](https://about.gitlab.com/free-trial/)). 12. If the [tag is protected](#release-permissions-with-protected-tags), this depends on the access Developers and Maintainers are given. 13. A Maintainer can't change project features visibility level if - [project visibility](../public_access/public_access.md) is set to private. + [project visibility](public_access.md) is set to private. 14. Attached design files are moved together with the issue even if the user doesn't have the Developer role. 15. Guest users can only set metadata (for example, labels, assignees, or milestones) @@ -262,6 +265,7 @@ More details about the permissions for some project-level features follow. | View pipelines page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | | View pipelines tab in MR | ✓ (*3*) | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | | [View vulnerabilities in a pipeline](application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline) | | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| View and download project-level [Secure Files](../api/secure_files.md) | | | | ✓ | ✓ | ✓ | | Cancel and retry jobs | | | | ✓ | ✓ | ✓ | | Create new [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | | Delete job logs or job artifacts | | | | ✓ (*4*) | ✓ | ✓ | @@ -276,6 +280,7 @@ More details about the permissions for some project-level features follow. | Manage CI/CD settings | | | | | ✓ | ✓ | | Manage job triggers | | | | | ✓ | ✓ | | Manage project-level CI/CD variables | | | | | ✓ | ✓ | +| Manage project-level [Secure Files](../api/secure_files.md) | | | | | ✓ | ✓ | | Use [environment terminals](../ci/environments/index.md#web-terminals-deprecated) | | | | | ✓ | ✓ | | Delete pipelines | | | | | | ✓ | @@ -391,7 +396,6 @@ The following table lists group permissions available for each role: | View [Group DevOps Adoption](group/devops_adoption/index.md) | | ✓ | ✓ | ✓ | ✓ | | View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | | View [Productivity analytics](analytics/productivity_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| View Usage quota data | | ✓ | ✓ | ✓ | ✓ | | Create and edit [group wiki](project/wiki/group.md) pages | | | ✓ | ✓ | ✓ | | Create project in group | | | ✓ (3)(5) | ✓ (3) | ✓ (3) | | Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | @@ -407,7 +411,7 @@ The following table lists group permissions available for each role: | List group deploy tokens | | | | ✓ | ✓ | | Manage [group push rules](group/index.md#group-push-rules) | | | | ✓ | ✓ | | View/manage group-level Kubernetes cluster | | | | ✓ | ✓ | -| Administer project compliance frameworks | | | | | ✓ | +| Create and manage compliance frameworks | | | | | ✓ | | Create/Delete group deploy tokens | | | | | ✓ | | Change group visibility level | | | | | ✓ | | Delete group | | | | | ✓ | @@ -421,8 +425,8 @@ The following table lists group permissions available for each role: | Share (invite) groups with groups | | | | | ✓ | | View 2FA status of members | | | | | ✓ | | View [Billing](../subscriptions/gitlab_com/index.md#view-your-gitlab-saas-subscription) | | | | | ✓ (4) | -| View [Usage Quotas](usage_quotas.md) Page | | | | ✓ | ✓ (4) | -| Manage runners | | | | | ✓ | +| View group [Usage Quotas](usage_quotas.md) page | | | | | ✓ (4) | +| Manage group runners | | | | | ✓ | <!-- markdownlint-disable MD029 --> diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index f201e04183c..a86968654c7 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -88,7 +88,7 @@ The following is hidden from your user profile page (`https://gitlab.example.com - Tabs for activity, groups, contributed projects, personal projects, starred projects, snippets NOTE: -Making your user profile page private does not hide your public resources from the REST or GraphQL APIs. +Making your user profile page private does not hide all your public resources from the REST or GraphQL APIs. ### User visibility @@ -123,7 +123,7 @@ To create a new project and add its README to your profile: 1. For **Project Configuration**, ensure **Initialize repository with a README** is selected. 1. Select **Create project**. 1. Create a README file inside this project. The file can be any valid [README or index file](../project/repository/index.md#readme-and-index-files). -1. Populate the README file with [Markdown](../markdown.md). +1. Populate the README file with [Markdown](../markdown.md), or another [supported markup language](../project/repository/index.md#supported-markup-languages). GitLab displays the contents of your README below your contribution graph. @@ -150,7 +150,7 @@ To add links to other accounts: ## Show private contributions on your user profile page -In the user contribution calendar graph and recent activity list, you can see your [contribution actions](../index.md#user-contribution-events) to private projects. +In the user contribution calendar graph and recent activity list, you can see your [contribution actions](#user-contribution-events) to private projects. To show private contributions: @@ -322,6 +322,65 @@ and configure it on your local machine by using the following command: git config --global user.email <your email address> ``` +## User activity + +GitLab tracks user contribution activity. +You can follow or unfollow other users from their [user profiles](#access-your-user-profile). +To view a user's activity in a top-level Activity view: + +1. From a user's profile, select **Follow**. +1. In the GitLab menu, select **Activity**. +1. Select the **Followed users** tab. + +### User contribution events + +Each of these contribution events is tracked: + +- `approved` + - Merge request +- `closed` + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone +- `commented` on any `Noteable` record. + - Alert + - Commit + - Design + - Issue + - Merge request + - Snippet +- `created` + - Design + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone + - Project + - Wiki page +- `destroyed` + - Design + - Milestone + - Wiki page +- `expired` + - Project membership +- `joined` + - Project membership +- `left` + - Project membership +- `merged` + - Merge request +- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. + - Project +- `reopened` + - [Epic](../group/epics/index.md) + - Issue + - Merge request + - Milestone +- `updated` + - Design + - Wiki page + ## Troubleshooting ### Why do I keep getting signed out? @@ -379,6 +438,6 @@ Without the `config.extend_remember_period` flag, you would be forced to sign in - [Receive emails for sign-ins from unknown IP addresses or devices](unknown_sign_in_notification.md) - Manage applications that can [use GitLab as an OAuth provider](../../integration/oauth_provider.md#introduction-to-oauth) - Manage [personal access tokens](personal_access_tokens.md) to access your account via API and authorized applications -- Manage [SSH keys](../../ssh/index.md) to access your account via SSH +- Manage [SSH keys](../ssh.md) to access your account via SSH - Change your [syntax highlighting theme](preferences.md#syntax-highlighting-theme) - [View your active sessions](active_sessions.md) and revoke any of them if necessary diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 4c9622810b2..d0e9b427f1c 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -11,7 +11,7 @@ Stay informed about what's happening in GitLab with email notifications. You can receive updates about activity in issues, merge requests, epics, and designs. For the tool that GitLab administrators can use to send messages to users, read -[Email from GitLab](../../tools/email.md). +[Email from GitLab](../admin_area/email_from_gitlab.md). ## Who receives notifications @@ -21,11 +21,14 @@ that happen there. You might receive notifications for one of the following reasons: - You participate in an issue, merge request, epic, or design. You become a participant when you comment - or edit, or someone mentions you. + or edit, or someone mentions <sup>1</sup> you. - You've [enabled notifications in an issue, merge request, or epic](#notifications-on-issues-merge-requests-and-epics). - You've configured notifications for the [project](#change-level-of-project-notifications) or [group](#group-notifications). - You're subscribed to group or project pipeline notifications via the pipeline emails [integration](../project/integrations/overview.md). +1. GitLab doesn't send a notification when + [a comment is edited to include a user mention](../discussions/index.md#editing-a-comment-to-add-a-mention). + NOTE: Administrators can block notifications, preventing them from being sent. @@ -171,26 +174,27 @@ Users are notified of the following events: <!-- The table is sorted first by recipient, then alphabetically. --> -| Event | Sent to | Settings level | -|------------------------------|---------------------|------------------------------| -| New release | Project members | Custom notification. | -| Project moved | Project members | Any other than disabled. | -| Email changed | User | Security email, always sent. | -| Group access level changed | User | Sent when user group access level is changed. | -| New email added | User | Security email, always sent. | -| New SAML/SCIM user provisioned | User | Sent when a user is provisioned through SAML/SCIM. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276018) in GitLab 13.8 | -| New SSH key added | User | Security email, always sent. | -| New user created | User | Sent on user creation, except for OmniAuth (LDAP). | -| Password changed | User | Security email, always sent when user changes their own password. | -| Password changed by administrator | User | Security email, always sent when an administrator changes the password of another user. | -| Personal access tokens expiring soon | User | Security email, always sent. | -| Personal access tokens have been created | User | Security email, always sent. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337591) in GitLab 14.9. | -| Personal access tokens have expired | User | Security email, always sent. | -| Project access level changed | User | Sent when user project access level is changed. | -| SSH key has expired | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.12._ | -| Two-factor authentication disabled | User | Security email, always sent. | -| User added to group | User | Sent when user is added to group. | -| User added to project | User | Sent when user is added to project. | +| Event | Sent to | Settings level | +|------------------------------------------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------| +| New release | Project members | Custom notification. | +| Project moved | Project members | Any other than disabled. | +| Email changed | User | Security email, always sent. | +| Group access level changed | User | Sent when user group access level is changed. | +| New email address added | User | Security email, sent to primary email address. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337635) in GitLab 14.9._ | +| New email address added | User | Security email, sent to newly-added email address. | +| New SAML/SCIM user provisioned | User | Sent when a user is provisioned through SAML/SCIM. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276018) in GitLab 13.8._ | +| New SSH key added | User | Security email, always sent. | +| New user created | User | Sent on user creation, except for OmniAuth (LDAP). | +| Password changed | User | Security email, always sent when user changes their own password. | +| Password changed by administrator | User | Security email, always sent when an administrator changes the password of another user. | +| Personal access tokens expiring soon | User | Security email, always sent. | +| Personal access tokens have been created | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337591) in GitLab 14.9._ | +| Personal access tokens have expired | User | Security email, always sent. | +| Project access level changed | User | Sent when user project access level is changed. | +| SSH key has expired | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.12._ | +| Two-factor authentication disabled | User | Security email, always sent. | +| User added to group | User | Sent when user is added to group. | +| User added to project | User | Sent when user is added to project. | ## Notifications on issues, merge requests, and epics @@ -241,26 +245,26 @@ epics: | Event | Sent to | |------------------------|---------| -| Change milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected. | -| Change milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected. | +| Change milestone issue | Subscribers and participants mentioned. | +| Change milestone merge request | Subscribers and participants mentioned. | | Close epic | | | Close issue | | | Close merge request | | -| Due issue | Participants and Custom notification level with this event selected. | | Failed pipeline | The author of the pipeline. | | Fixed pipeline | The author of the pipeline. Enabled by default. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24309) in GitLab 13.1._ | +| Issue due | Participants and Custom notification level with this event selected. | | Merge merge request | | | Merge when pipeline succeeds | Author, Participants, Watchers, Subscribers, and Custom notification level with this event selected. Custom notification level is ignored for Author, Watchers and Subscribers. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211961) in GitLab 13.4._ | | Merge request [marked as ready](../project/merge_requests/drafts.md) | Watchers and participants. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15332) in GitLab 13.10._ | -| New comment | Participants, Watchers, Subscribers, and Custom notification level with this event selected. Also anyone mentioned by username in the comment, with notification level "Mention" or higher. | | New epic | | | New issue | | | New merge request | | +| New note | Participants, Watchers, Subscribers, and Custom notification level with this event selected. Also anyone mentioned by username in the comment, with notification level "Mention" or higher. | | Push to merge request | Participants and Custom notification level with this event selected. | | Reassign issue | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old assignee. | | Reassign merge request | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old assignee. | -| Remove milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected. | -| Remove milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected. | +| Remove milestone issue | Subscribers and participants mentioned. | +| Remove milestone merge request | Subscribers and participants mentioned. | | Reopen epic | | | Reopen issue | | | Reopen merge request | | @@ -304,7 +308,7 @@ If you no longer wish to receive any email notifications: **Disabled**. On self-managed installations, even after doing this, your instance administrator -[can still email you](../../tools/email.md). +[can still email you](../admin_area/email_from_gitlab.md). To unsubscribe, select the unsubscribe link in one of these emails. ## Email headers you can use to filter email diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 1fbbe438370..4c132094d24 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -98,8 +98,8 @@ A personal access token can perform actions based on the assigned scopes. | `read_api` | Read-only for the complete API, including all groups and projects, the Container Registry, and the Package Registry. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28944) in GitLab 12.10.) | | `read_repository` | Read-only (pull) for the repository through `git clone`. | | `write_repository` | Read-write (pull, push) for the repository through `git clone`. | -| `read_registry` | Read-only (pull) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. | -| `write_registry` | Read-write (push) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28958) in GitLab 12.10.) | +| `read_registry` | Read-only (pull) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. Available only when the Container Registry is enabled. | +| `write_registry` | Read-write (push) for [Container Registry](../packages/container_registry/index.md) images if a project is private and authorization is required. Available only when the Container Registry is enabled. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28958) in GitLab 12.10.) | | `sudo` | API actions as any user in the system (if the authenticated user is an administrator). | ## When personal access tokens expire diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 48160bb97ac..ecd6e83efa1 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -89,6 +89,20 @@ The default syntax theme is White, and you can choose among 5 different themes: Introduced in GitLab 13.6, the themes [Solarized](https://gitlab.com/gitlab-org/gitlab/-/issues/221034) and [Monokai](https://gitlab.com/gitlab-org/gitlab/-/issues/221034) also apply to the [Web IDE](../project/web_ide/index.md) and [Snippets](../snippets.md). +## Diff colors + +A diff compares the old/removed content with the new/added content (e.g. when +[reviewing a merge request](../project/merge_requests/reviews/index.md#review-a-merge-request) or in a +[Markdown inline diff](../markdown.md#inline-diff)). +Typically, the colors red and green are used for removed and added lines in diffs. +The exact colors depend on the selected [syntax highlighting theme](#syntax-highlighting-theme). +The colors may lead to difficulties in case of red–green color blindness. + +For this reason, you can customize the following colors: + +- Color for removed lines +- Color for added lines + ## Behavior The following settings allow you to customize the behavior of the GitLab layout diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index f6f31ee8f36..c897100f14e 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Container Host Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can diff --git a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md index 25e2f6ddb91..af3128e3006 100644 --- a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Container Host Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. The following steps are recommended for installing Container Host Security. diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index eeaf7e82ef4..b294859c660 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Container Network Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index 43f4ea6c326..7671ed7eb73 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: Container Network Security is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. The following steps are recommended for installing Container Network Security. diff --git a/doc/user/project/clusters/protect/index.md b/doc/user/project/clusters/protect/index.md index 3a80132fb50..6b89f7f1557 100644 --- a/doc/user/project/clusters/protect/index.md +++ b/doc/user/project/clusters/protect/index.md @@ -12,7 +12,7 @@ WARNING: The Container Network Security and Container Host Security features are in their end-of-life processes. They're [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) -for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) +in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0. GitLab makes it straightforward to protect applications deployed in [connected Kubernetes clusters](index.md). diff --git a/doc/user/project/code_intelligence.md b/doc/user/project/code_intelligence.md index f1071af7c1f..7f35caf2a68 100644 --- a/doc/user/project/code_intelligence.md +++ b/doc/user/project/code_intelligence.md @@ -48,8 +48,8 @@ After the job succeeds, code intelligence data can be viewed while browsing the ## Find references -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217392) in GitLab 13.2. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235735) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217392) in GitLab 13.2 [with a flag](../../administration/feature_flags.md) named `code_navigation_references`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/225621) in GitLab 13.3. Feature flag `code_navigation_references` removed. To find where a particular object is being used, you can see links to specific lines of code under the **References** tab: diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index fefc27063a6..e37ff560080 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -124,8 +124,8 @@ Only one CODEOWNERS pattern can match per file path. ### Organize Code Owners by putting them into sections -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12137) in GitLab 13.2 behind a feature flag, enabled by default. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42389) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12137) in GitLab 13.2 [with a flag](../../administration/feature_flags.md) named `sectional_codeowners`. Disabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42389) in GitLab 13.4. Feature flag `sectional_codeowners` removed. You can organize Code Owners by putting them into named sections. @@ -254,6 +254,11 @@ README @group @group/with-nested/subgroup # `docs/index.md` but not `docs/projects/index.md`: /docs/* @root-docs +# Include `/**` to specify Code Owners for all subdirectories +# in a directory. This rule matches `docs/projects/index.md` or +# `docs/development/index.md` +/docs/**/*.md @root-docs + # This code makes matches a `lib` directory nested anywhere in the repository: lib/ @lib-owner diff --git a/doc/user/project/deploy_keys/img/deploy_keys_v13_0.png b/doc/user/project/deploy_keys/img/deploy_keys_v13_0.png Binary files differdeleted file mode 100644 index 15e6e71803c..00000000000 --- a/doc/user/project/deploy_keys/img/deploy_keys_v13_0.png +++ /dev/null diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index b7674be2fa0..8f1da4b278a 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -69,7 +69,7 @@ The deploy keys available are listed: Prerequisites: - You must have at least the Maintainer role for the project. -- [Generate an SSH key pair](../../../ssh/index.md#generate-an-ssh-key-pair). Put the private SSH +- [Generate an SSH key pair](../../ssh.md#generate-an-ssh-key-pair). Put the private SSH key on the host that requires access to the repository. 1. On the top bar, select **Menu > Projects** and find your project. @@ -87,7 +87,7 @@ name and permissions. Prerequisites: - You must have administrator access. -- [Generate an SSH key pair](../../../ssh/index.md#generate-an-ssh-key-pair). Put the private SSH +- [Generate an SSH key pair](../../ssh.md#generate-an-ssh-key-pair). Put the private SSH key on the host that requires access to the repository. To create a public deploy key: diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 4d69209aafa..64c18ab6f3b 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -16,7 +16,9 @@ container registry images of a project without having a user and a password. Deploy tokens can be managed only by users with the Maintainer role. -Deploy tokens cannot be used with the GitLab API. +Deploy tokens can't be used with the GitLab public API. However, you can use deploy tokens with some +endpoints, such as those from the Package Registry. For details, see +[Authenticate with the registry](../../packages/package_registry/index.md#authenticate-with-the-registry). Deploy tokens are tied to the project and stay enabled even when the user who created the token is removed from the project. diff --git a/doc/user/project/img/promote_to_parent_group_workaround_v14_10.png b/doc/user/project/img/promote_to_parent_group_workaround_v14_10.png Binary files differnew file mode 100644 index 00000000000..ed4ac9ba234 --- /dev/null +++ b/doc/user/project/img/promote_to_parent_group_workaround_v14_10.png diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index 8c0d9fc422b..b2425686024 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -49,7 +49,7 @@ The importer will create any new namespaces (groups) if they don't exist or in the case the namespace is taken, the repository will be imported under the user's namespace that started the import process. -## Requirements for user-mapped contributions +## Requirements for user-mapped contributions For user contributions to be mapped, each user must complete the following before the project import: @@ -83,7 +83,7 @@ For user contributions to be mapped, each user must complete the following befor ### If you have more than one Bitbucket account -Be sure to sign in to the correct account. +Be sure to sign in to the correct account. If you've accidentally started the import process with the wrong account, follow these steps: diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 4e3642eb3bd..b6241dbbdb0 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -24,11 +24,10 @@ created as private in GitLab as well. ## Import your Bitbucket repositories -Prerequisites: +Prerequisite: - An administrator must have enabled the **Bitbucket Server** in **Admin > Settings > General > Visibility and access controls > Import sources**. -- Review the importer's [limitations](#limitations). To import your Bitbucket repositories: @@ -41,24 +40,27 @@ To import your Bitbucket repositories: 1. Select the projects to import, or import all projects. You can filter projects by name and select the namespace for which to import each project. -## Limitations +### Items that are not imported -- GitLab doesn't allow comments on arbitrary lines of code. Any out-of-bounds Bitbucket comments are - inserted as comments in the merge request. -- Bitbucket Server allows multiple threading levels. The importer collapses this into one thread and - quotes part of the original comment. -- Declined pull requests have unreachable commits. This prevents the importer from generating a - proper diff. These pull requests show up as empty changes. -- Project filtering doesn't support fuzzy search. Only starts with or full match strings are - supported. - -The following aren't imported: +The following items aren't imported: - Pull request approvals - Attachments in Markdown - Task lists - Emoji reactions +### Items that are imported but changed + +The following items are changed when they are imported: + +- GitLab doesn't allow comments on arbitrary lines of code. Any out-of-bounds Bitbucket comments are + inserted as comments in the merge request. +- Multiple threading levels are collapsed into one thread and + quotes are added as part of the original comment. +- Declined pull requests have unreachable commits. These pull requests show up as empty changes. +- Project filtering doesn't support fuzzy search. Only **starts with** or **full match** strings are + supported. + ## User assignment When issues and pull requests are importing, the importer tries to find the author's email address diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 9f1c049045c..329d91916e8 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -30,7 +30,7 @@ The following aspects of a project are imported: References to pull requests and issues are preserved (GitLab.com & 8.7+), and each imported repository maintains visibility level unless that [visibility -level is restricted](../../../public_access/public_access.md#restrict-use-of-public-or-internal-projects), +level is restricted](../../public_access.md#restrict-use-of-public-or-internal-projects), in which case it defaults to the default project visibility. The namespace is a user or group in GitLab, such as `gitlab.com/janedoe` or diff --git a/doc/user/project/import/img/gitlab_import_history_page_v14_10.png b/doc/user/project/import/img/gitlab_import_history_page_v14_10.png Binary files differnew file mode 100644 index 00000000000..c93b5ed2b27 --- /dev/null +++ b/doc/user/project/import/img/gitlab_import_history_page_v14_10.png diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 41ef15108ec..432f043f945 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -30,6 +30,30 @@ repository is too large, the import can timeout. You can also [connect your external repository to get CI/CD benefits](../../../ci/ci_cd_for_external_repos/index.md). +## Project import history + +You can view all project imports created by you. This list includes the following: + +- Source (without credentials for security reasons) +- Destination +- Status +- Error details if the import failed + +To view project import history: + +1. Sign in to GitLab. +1. On the top bar, select **New** (**{plus}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **History**. + + + +The history also includes projects created from [built-in](../working_with_projects.md#create-a-project-from-a-built-in-template) +or [custom](../working_with_projects.md#create-a-project-from-a-built-in-template) +templates. GitLab uses [import repository by URL](repo_by_url.md) +to create a new project from a template. + ## LFS authentication When importing a project that contains LFS objects, if the project has an [`.lfsconfig`](https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-config.5.ronn) diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 801c2520bda..60a4ca5c0ea 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -11,7 +11,7 @@ your codebase. You can also use projects to track issues, plan work, collaborate on code, and continuously build, test, and use built-in CI/CD to deploy your app. -Projects can be available [publicly, internally, or privately](../../public_access/public_access.md). +Projects can be available [publicly, internally, or privately](../public_access.md). GitLab does not limit the number of private projects you can create. ## Project features @@ -35,7 +35,7 @@ Projects include the following [features](https://about.gitlab.com/features/): - [Deploy tokens](deploy_tokens/index.md): Manage access to the repository and Container Registry. - [Web IDE](web_ide/index.md) - [CVE ID Requests](../application_security/cve_id_request.md): Request a CVE identifier to track a - vulnerability in your project. **(FREE SAAS)** + vulnerability in your project. **Issues and merge requests:** @@ -83,7 +83,7 @@ Projects include the following [features](https://about.gitlab.com/features/): - [Kubernetes cluster integration](../infrastructure/clusters/index.md): Connect your GitLab project with a Kubernetes cluster. - [Feature Flags](../../operations/feature_flags.md): Ship different features - by dynamically toggling functionality. **(PREMIUM)** + by dynamically toggling functionality. - [GitLab Pages](pages/index.md): Build, test, and deploy your static website. @@ -92,8 +92,8 @@ Projects include the following [features](https://about.gitlab.com/features/): - [Wiki](wiki/index.md): Document your GitLab project in an integrated Wiki. - [Snippets](../snippets.md): Store, share and collaborate on code snippets. - [Value Stream Analytics](../analytics/value_stream_analytics.md): Review your development lifecycle. -- [Insights](insights/index.md): Configure the insights that matter for your projects. **(ULTIMATE)** -- [Security Dashboard](../application_security/security_dashboard/index.md) **(ULTIMATE)** +- [Insights](insights/index.md): Configure the insights that matter for your projects. +- [Security Dashboard](../application_security/security_dashboard/index.md) - [Syntax highlighting](highlighting.md): Customize your code blocks, overriding the default language choice. - [Badges](badges.md): Add an image to the **Project information** page. @@ -102,9 +102,9 @@ Projects include the following [features](https://about.gitlab.com/features/): associated with a released version of your code. - [Package Registry](../packages/package_registry/index.md): Publish and install packages. - [Code owners](code_owners.md): Specify code owners for specific files. -- [License Compliance](../compliance/license_compliance/index.md): Approve and deny licenses for projects. **(ULTIMATE)** -- [Dependency List](../application_security/dependency_list/index.md): View project dependencies. **(ULTIMATE)** -- [Requirements](requirements/index.md): Create criteria to check your products against. **(ULTIMATE)** +- [License Compliance](../compliance/license_compliance/index.md): Approve and deny licenses for projects. +- [Dependency List](../application_security/dependency_list/index.md): View project dependencies. +- [Requirements](requirements/index.md): Create criteria to check your products against. - [Code Intelligence](code_intelligence.md): Navigate code. ## Project integrations diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md index b4d7790df1d..a10e261f10e 100644 --- a/doc/user/project/integrations/asana.md +++ b/doc/user/project/integrations/asana.md @@ -32,8 +32,8 @@ In Asana, create a Personal Access Token. Complete these steps in GitLab: -1. Go to the project you want to configure. -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Asana**. 1. Ensure that the **Active** toggle is enabled. 1. Paste the token you generated in Asana. diff --git a/doc/user/project/integrations/bugzilla.md b/doc/user/project/integrations/bugzilla.md index a54a3adc408..4a9a8d62098 100644 --- a/doc/user/project/integrations/bugzilla.md +++ b/doc/user/project/integrations/bugzilla.md @@ -14,7 +14,8 @@ You can configure Bugzilla as an To enable the Bugzilla integration in a project: -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Bugzilla**. 1. Select the checkbox under **Enable integration**. 1. Fill in the required fields: diff --git a/doc/user/project/integrations/discord_notifications.md b/doc/user/project/integrations/discord_notifications.md index ad7719f0e5b..b7e25b815fc 100644 --- a/doc/user/project/integrations/discord_notifications.md +++ b/doc/user/project/integrations/discord_notifications.md @@ -26,8 +26,9 @@ and configure it in GitLab. With the webhook URL created in the Discord channel, you can set up the Discord Notifications service in GitLab. -1. Navigate to the [Integrations page](overview.md#accessing-integrations) in your project's settings. That is, **Project > Settings > Integrations**. -1. Select the **Discord Notifications** integration to configure it. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Discord Notifications**. 1. Ensure that the **Active** toggle is enabled. 1. Check the checkboxes corresponding to the GitLab events for which you want to send notifications to Discord. 1. Paste the webhook URL that you copied from the create Discord webhook step. diff --git a/doc/user/project/integrations/emails_on_push.md b/doc/user/project/integrations/emails_on_push.md index 33c197b962e..c1c48c7fb12 100644 --- a/doc/user/project/integrations/emails_on_push.md +++ b/doc/user/project/integrations/emails_on_push.md @@ -9,17 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w By enabling this service, you receive email notifications for every change that is pushed to your project. -From the [Integrations page](overview.md#accessing-integrations) -select **Emails on push** service to activate and configure it. +To enable emails on push: -In the _Recipients_ area, provide a list of emails separated by spaces or newlines. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Emails on push**. +1. In the **Recipients** section, provide a list of emails separated by spaces or newlines. +1. Configure the following options: -The following options are available: - -- **Push events** - Email is triggered when a push event is received. -- **Tag push events** - Email is triggered when a tag is created and pushed. -- **Send from committer** - Send notifications from the committer's email address if the domain matches the domain used by your GitLab instance (such as `user@gitlab.com`). -- **Disable code diffs** - Don't include possibly sensitive code diffs in notification body. + - **Push events** - Email is triggered when a push event is received. + - **Tag push events** - Email is triggered when a tag is created and pushed. + - **Send from committer** - Send notifications from the committer's email address if the domain matches the domain used by your GitLab instance (such as `user@gitlab.com`). + - **Disable code diffs** - Don't include possibly sensitive code diffs in notification body. | Settings | Notification | | --- | --- | diff --git a/doc/user/project/integrations/ewm.md b/doc/user/project/integrations/ewm.md index bc9b2d59db3..b02f1a06e96 100644 --- a/doc/user/project/integrations/ewm.md +++ b/doc/user/project/integrations/ewm.md @@ -14,7 +14,8 @@ This IBM product was [formerly named Rational Team Concert](https://jazz.net/blo To enable the EWM integration, in a project: -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **EWM**. 1. Select the checkbox under **Enable integration**. 1. Fill in the required fields: diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index 7e8de776619..2dae02dc093 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -25,8 +25,6 @@ the [Slack App Directory](https://slack.com/apps). Clicking install takes you to the [GitLab Slack application landing page](https://gitlab.com/-/profile/slack/edit) where you can select a project to enable the GitLab Slack application for. - - ## Configuration Alternatively, you can configure the Slack application with a project's diff --git a/doc/user/project/integrations/harbor.md b/doc/user/project/integrations/harbor.md index d66e2222538..2a1b12057aa 100644 --- a/doc/user/project/integrations/harbor.md +++ b/doc/user/project/integrations/harbor.md @@ -6,6 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Harbor container registry integration **(FREE)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80999) in GitLab 14.9. + Use Harbor as the container registry for your GitLab project. [Harbor](https://goharbor.io/) is an open source registry that can help you manage artifacts across cloud native compute platforms, like Kubernetes and Docker. @@ -43,7 +45,7 @@ After the Harbor integration is activated: ## Secure your requests to the Harbor APIs For each API request through the Harbor integration, the credentials for your connection to the Harbor API use -the `username:password` combination. The following are suggestions for safe use: +the `username:password` combination. The following are suggestions for safe use: - Use TLS on the Harbor APIs you connect to. - Follow the principle of least privilege (for access on Harbor) with your credentials. diff --git a/doc/user/project/integrations/img/failed_badges.png b/doc/user/project/integrations/img/failed_badges.png Binary files differindex d44415a8687..5a1f481e54c 100644 --- a/doc/user/project/integrations/img/failed_badges.png +++ b/doc/user/project/integrations/img/failed_badges.png diff --git a/doc/user/project/integrations/img/failed_banner.png b/doc/user/project/integrations/img/failed_banner.png Binary files differindex ba40c1301d6..4384ce07873 100644 --- a/doc/user/project/integrations/img/failed_banner.png +++ b/doc/user/project/integrations/img/failed_banner.png diff --git a/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png b/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png Binary files differdeleted file mode 100644 index 57cd35c9f5d..00000000000 --- a/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png +++ /dev/null diff --git a/doc/user/project/integrations/irker.md b/doc/user/project/integrations/irker.md index 279b139bacd..b2c2aea2c2b 100644 --- a/doc/user/project/integrations/irker.md +++ b/doc/user/project/integrations/irker.md @@ -39,9 +39,8 @@ network. For more details, read ## Complete these steps in GitLab -1. On the top bar, select **Menu > Projects** and find the project you want to - configure for notifications. -1. Navigate to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **irker (IRC gateway)**. 1. Ensure that the **Active** toggle is enabled. 1. Optional. Under **Server host**, enter the server host address where `irkerd` runs. If empty, diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index f3f8d900e12..7dd4c1d1a8b 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -37,27 +37,25 @@ Display name override is not enabled by default, you need to ask your administra ## Configure GitLab to send notifications to Mattermost After the Mattermost instance has an incoming webhook set up, you can set up GitLab -to send the notifications. - -Navigate to the [Integrations page](overview.md#accessing-integrations) -and select the **Mattermost notifications** service. Select the GitLab events -you want to generate notifications for. - -For each event you select, input the Mattermost channel you want to receive the -notification. You do not need to add the hash sign (`#`). - -Then fill in the integration configuration: - -- **Webhook**: The incoming webhook URL on Mattermost, similar to - `http://mattermost.example/hooks/5xo…`. -- **Username**: Optional. The username shown in messages sent to Mattermost. - To change the bot's username, provide a value. -- **Notify only broken pipelines**: If you enable the **Pipeline** event, and you want - notifications about failed pipelines only. -- **Branches for which notifications are to be sent**: The branches to send notifications for. -- **Labels to be notified**: Optional. Labels required for the issue or merge request - to trigger a notification. Leave blank to notify for all issues and merge requests. -- **Labels to be notified behavior**: When you use the **Labels to be notified** filter, - messages are sent when an issue or merge request contains _any_ of the labels specified - in the filter. You can also choose to trigger messages only when the issue or merge request - contains _all_ the labels defined in the filter. +to send the notifications: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Mattermost notifications**. +1. Select the GitLab events to generate notifications for. For each event you select, input the Mattermost channel + to receive the notification. You do not need to add the hash sign (`#`). +1. Fill in the integration configuration: + + - **Webhook**: The incoming webhook URL on Mattermost, similar to + `http://mattermost.example/hooks/5xo…`. + - **Username**: Optional. The username shown in messages sent to Mattermost. + To change the bot's username, provide a value. + - **Notify only broken pipelines**: If you enable the **Pipeline** event, and you want + notifications about failed pipelines only. + - **Branches for which notifications are to be sent**: The branches to send notifications for. + - **Labels to be notified**: Optional. Labels required for the issue or merge request + to trigger a notification. Leave blank to notify for all issues and merge requests. + - **Labels to be notified behavior**: When you use the **Labels to be notified** filter, + messages are sent when an issue or merge request contains _any_ of the labels specified + in the filter. You can also choose to trigger messages only when the issue or merge request + contains _all_ the labels defined in the filter. diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index 2cb62b8924e..081780e6277 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -22,9 +22,9 @@ want to configure. ## Integrations listing -Click on the service links to see further configuration instructions and details. +Click on the integration links to see further configuration instructions and details. -| Service | Description | Service hooks | +| Integration | Description | Integration hooks | | --------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------- | | [Asana](asana.md) | Add commit messages as comments to Asana tasks. | **{dotted-circle}** No | | Assembla | Manage projects. | **{dotted-circle}** No | @@ -69,7 +69,7 @@ Click on the service links to see further configuration instructions and details > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17874) in GitLab 12.4. -If a single push includes changes to more than three branches or tags, services +If a single push includes changes to more than three branches or tags, integrations supported by `push_hooks` and `tag_push_hooks` events aren't executed. The number of branches or tags supported can be changed via @@ -89,12 +89,12 @@ By default, the SSL certificate for outgoing HTTP requests is verified based on an internal list of Certificate Authorities. This means the certificate cannot be self-signed. -You can turn off SSL verification in the configuration settings for [webhooks](webhooks.md#configure-a-webhook) +You can turn off SSL verification in the configuration settings for [webhooks](webhooks.md#configure-a-webhook-in-gitlab) and some integrations. ## Troubleshooting integrations -Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. Learn more about [troubleshooting service hooks](webhooks.md#troubleshoot-webhooks). +Some integrations use hooks for integration with external applications. To confirm which ones use integration hooks, see the [integrations listing](#integrations-listing) above. Learn more about [troubleshooting integration hooks](webhooks.md#troubleshoot-webhooks). ### Uninitialized repositories diff --git a/doc/user/project/integrations/pivotal_tracker.md b/doc/user/project/integrations/pivotal_tracker.md index 8b17f4afaa8..7f5414b86de 100644 --- a/doc/user/project/integrations/pivotal_tracker.md +++ b/doc/user/project/integrations/pivotal_tracker.md @@ -37,8 +37,8 @@ In Pivotal Tracker, [create an API token](https://www.pivotaltracker.com/help/ar Complete these steps in GitLab: -1. Go to the project you want to configure. -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Pivotal Tracker**. 1. Ensure that the **Active** toggle is enabled. 1. Paste the token you generated in Pivotal Tracker. diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 760b5030416..068a2810a53 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -62,9 +62,9 @@ GitLab can use these to access the resource. More information about authenticati service account can be found at Google's documentation for [Authenticating from a service account](https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account). -1. Navigate to the [Integrations page](overview.md#accessing-integrations) at - **Settings > Integrations**. -1. Click the **Prometheus** service. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Prometheus**. 1. For **API URL**, provide the domain name or IP address of your server, such as `http://prometheus.example.com/` or `http://192.0.2.1/`. 1. (Optional) In **Google IAP Audience Client ID**, provide the Client ID of the @@ -73,7 +73,7 @@ service account can be found at Google's documentation for Service Account credentials file that is authorized to access the Prometheus resource. The JSON key `token_credential_uri` is discarded to prevent [Server-side Request Forgery (SSRF)](https://www.hackerone.com/application-security/how-server-side-request-forgery-ssrf). -1. Click **Save changes**. +1. Select **Save changes**.  @@ -83,11 +83,12 @@ You can configure [Thanos](https://thanos.io/) as a drop-in replacement for Prom with GitLab. Use the domain name or IP address of the Thanos server you'd like to integrate with. -1. Navigate to the [Integrations page](overview.md#accessing-integrations). -1. Click the **Prometheus** service. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Prometheus**. 1. Provide the domain name or IP address of your server, for example `http://thanos.example.com/` or `http://192.0.2.1/`. -1. Click **Save changes**. +1. Select **Save changes**. ### Precedence with multiple Prometheus configurations diff --git a/doc/user/project/integrations/prometheus_library/cloudwatch.md b/doc/user/project/integrations/prometheus_library/cloudwatch.md index e8d611af30d..08488c33ac7 100644 --- a/doc/user/project/integrations/prometheus_library/cloudwatch.md +++ b/doc/user/project/integrations/prometheus_library/cloudwatch.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab supports automatically detecting and monitoring AWS resources, starting with the [Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/) (ELB). diff --git a/doc/user/project/integrations/prometheus_library/haproxy.md b/doc/user/project/integrations/prometheus_library/haproxy.md index 76d13d5487c..ad2cb2681b9 100644 --- a/doc/user/project/integrations/prometheus_library/haproxy.md +++ b/doc/user/project/integrations/prometheus_library/haproxy.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab has support for automatically detecting and monitoring HAProxy. This is provided by leveraging the [HAProxy Exporter](https://github.com/prometheus/haproxy_exporter), which translates HAProxy statistics into a Prometheus readable form. diff --git a/doc/user/project/integrations/prometheus_library/index.md b/doc/user/project/integrations/prometheus_library/index.md index 9bdd4945f5d..aba14e1f3e9 100644 --- a/doc/user/project/integrations/prometheus_library/index.md +++ b/doc/user/project/integrations/prometheus_library/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 33a06958e0c..9a9880a0cb6 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab has support for automatically detecting and monitoring Kubernetes metrics. diff --git a/doc/user/project/integrations/prometheus_library/nginx.md b/doc/user/project/integrations/prometheus_library/nginx.md index ecf75d7b17a..2825066b8b0 100644 --- a/doc/user/project/integrations/prometheus_library/nginx.md +++ b/doc/user/project/integrations/prometheus_library/nginx.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab has support for automatically detecting and monitoring NGINX. This is provided by leveraging the [NGINX VTS exporter](https://github.com/hnlq715/nginx-vts-exporter), which translates [VTS statistics](https://github.com/vozlt/nginx-module-vts) into a Prometheus readable form. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index e123000e0c5..6e751b907eb 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. GitLab has support for automatically detecting and monitoring the Kubernetes NGINX Ingress controller. This is provided by leveraging the built-in Prometheus metrics included with Kubernetes NGINX Ingress controller [version 0.16.0](https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md#0160) onward. diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index fda7744e847..e1eee649f0a 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346541) -for use in GitLab 14.7, and is planned for removal in GitLab 15.0. +in GitLab 14.7, and is planned for removal in GitLab 16.0. NOTE: [NGINX Ingress version 0.16](nginx_ingress.md) and above have built-in Prometheus metrics, which are different than the VTS based metrics. diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index 05d7c31a288..bcab8d05f69 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -10,7 +10,8 @@ Use [Redmine](https://www.redmine.org/) as the issue tracker. To enable the Redmine integration in a project: -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Redmine**. 1. Select the checkbox under **Enable integration**. 1. Fill in the required fields: diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index cddb72a83b2..5ad344a7d8e 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -18,7 +18,7 @@ For GitLab.com, use the [GitLab Slack app](gitlab_slack_application.md) instead. ## Configure GitLab and Slack -Slack slash command [integrations](overview.md#accessing-integrations) +Slack slash command integrations are scoped to a project. 1. In GitLab, on the top bar, select **Menu > Projects** and find your project. diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index daab24a8ab9..1e607d89e80 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -15,7 +15,8 @@ copy its URL. In GitLab: -1. Go to the [Integrations page](overview.md#accessing-integrations) in your project's settings. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Unify Circuit**. 1. Turn on the **Active** toggle. 1. Select the checkboxes corresponding to the GitLab events you want to receive in Unify Circuit. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index d37196ec114..2bf6b4bbe01 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -203,7 +203,7 @@ The `assignee` and `assignee_id` keys are deprecated and contain the first assignee only. The `escalation_status` and `escalation_policy` fields are -only available for issue types which support escalations, +only available for issue types which [support escalations](../../../operations/incident_management/paging.md#paging), such as incidents. Request header: @@ -538,6 +538,32 @@ Payload example: "iid": 1, "description": "Et voluptas corrupti assumenda temporibus. Architecto cum animi eveniet amet asperiores. Vitae numquam voluptate est natus sit et ad id.", "position": 0, + "labels": [ + { + "id": 25, + "title": "Afterpod", + "color": "#3e8068", + "project_id": null, + "created_at": "2019-06-05T14:32:20.211Z", + "updated_at": "2019-06-05T14:32:20.211Z", + "template": false, + "description": null, + "type": "GroupLabel", + "group_id": 4 + }, + { + "id": 86, + "title": "Element", + "color": "#231afe", + "project_id": 4, + "created_at": "2019-06-05T14:32:20.637Z", + "updated_at": "2019-06-05T14:32:20.637Z", + "template": false, + "description": null, + "type": "ProjectLabel", + "group_id": null + } + ], "source":{ "name":"Gitlab Test", "description":"Aut reprehenderit ut est.", diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index e823391401d..f4f5b3f545b 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -48,7 +48,7 @@ specific to a group, including: - [Group member events](webhook_events.md#group-member-events) - [Subgroup events](webhook_events.md#subgroup-events) -## Configure a webhook +## Configure a webhook in GitLab You can configure a webhook for a group or a project. @@ -60,6 +60,72 @@ You can configure a webhook for a group or a project. 1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](overview.md#ssl-verification). 1. Select **Add webhook**. +## Configure your webhook receiver endpoint + +Webhook receivers should be *fast* and *stable*. +Slow and unstable receivers may be disabled temporarily to ensure system reliability. +If you are writing your own endpoint (web server) to receive GitLab webhooks, keep in mind the following: + +- Your endpoint should send its HTTP response as fast as possible. + You should aim for sub-second response times in all circumstances. + If the response takes longer than the configured timeout, GitLab assumes the + hook failed, which can lead to retries and potentially cause duplicate + events. + To customize the timeout, see + [Webhook fails or multiple webhook requests are triggered](#webhook-fails-or-multiple-webhook-requests-are-triggered). +- Your endpoint should ALWAYS return a valid HTTP response. If not, + GitLab assumes the hook failed and retries it. + Most HTTP libraries take care of the response for you automatically but if + you are writing a low-level hook, this is important to remember. +- GitLab usually ignores the HTTP status code returned by your endpoint, + unless the [`web_hooks_disable_failed` feature flag is set](#failing-webhooks). + +Best practices for a webhook receiver: + +- Prefer to return `200` or `201` status responses. + Only return error statuses (in the `4xx` range) to + indicate that the webhook has been misconfigured. For example, if your receiver + only supports push events, it is acceptable to return `400` if sent an issue + payload, since that is an indication that the hook has been set up + incorrectly. Alternatively, it is acceptable to ignore unrecognized event + payloads. Never return `500` status responses if the event has been handled. +- Your service should be idempotent. In some circumstances (including + timeouts), the same event may be sent twice. Be prepared to handle duplicate + events. You can reduce the chances of this by ensuring that your endpoint is + reliably fast and stable. +- Keep response payloads as short as possible. Empty responses are + fine. GitLab does not examine the response body, and it is only + stored so you can examine it later in the logs. +- Limit the number and size of response headers. Only send headers that would + help you diagnose problems when examining the web hook logs. +- To support fast response times, perform I/O or computationally intensive + operations asynchronously. You may indicate that the webhook is + asynchronous by returning `201`. + +### Failing webhooks + +> - Introduced in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 14.9. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. +The feature is not ready for production use. + +If a webhook fails repeatedly, it may be disabled automatically. + +Webhooks that return response codes in the `5xx` range are understood to be failing +intermittently, and are temporarily disabled. This lasts initially +for 10 minutes. If the hook continues to fail, the back-off period is +extended on each retry, up to a maximum disabled period of 24 hours. + +Webhooks that return failure codes in the `4xx` range are understood to be +misconfigured, and these are disabled until you manually re-enable +them. These webhooks are not automatically retried. + +See [troubleshooting](#troubleshoot-webhooks) for information on +how to see if a webhook is disabled, and how to re-enable it. + ## Test a webhook You can trigger a webhook manually, to ensure it's working properly. You can also send @@ -131,47 +197,7 @@ that the request is legitimate. Push events can be filtered by branch using a branch name or wildcard pattern to limit which push events are sent to your webhook endpoint. By default, all push events are sent to your webhook endpoint. You can configure branch filtering -in the [webhook settings](#configure-a-webhook) in your project. - -## HTTP responses for your endpoint - -If you are writing your own endpoint (web server) to receive -GitLab webhooks, keep in mind the following: - -- Your endpoint should send its HTTP response as fast as possible. If the response - takes longer than the configured timeout, GitLab assumes the hook failed and retries it. - To customize the timeout, see - [Webhook fails or multiple webhook requests are triggered](#webhook-fails-or-multiple-webhook-requests-are-triggered). -- Your endpoint should ALWAYS return a valid HTTP response. If not, - GitLab assumes the hook failed and retries it. - Most HTTP libraries take care of the response for you automatically but if - you are writing a low-level hook, this is important to remember. -- GitLab usually ignores the HTTP status code returned by your endpoint, - unless the [`web_hooks_disable_failed` feature flag is set](#failing-webhooks). - -### Failing webhooks - -> - Introduced in GitLab 13.12 [with a flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/329849) in GitLab 14.9. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `web_hooks_disable_failed`. -The feature is not ready for production use. - -If a webhook fails repeatedly, it may be disabled automatically. - -Webhooks that return response codes in the `5xx` range are understood to be failing -intermittently, and are temporarily disabled. This lasts initially -for 10 minutes. If the hook continues to fail, the back-off period is -extended on each retry, up to a maximum disabled period of 24 hours. - -Webhooks that return failure codes in the `4xx` range are understood to be -misconfigured, and these are disabled until you manually re-enable -them. These webhooks are not automatically retried. - -See [troubleshooting](#troubleshoot-webhooks) for information on -how to see if a webhook is disabled, and how to re-enable it. +in the [webhook settings](#configure-a-webhook-in-gitlab) in your project. ## How image URLs are displayed in the webhook body @@ -220,7 +246,7 @@ To view the table: - **Fails to connect** if it is temporarily disabled and will retry later.  - + 1. Select **Edit** for the webhook you want to view. The table includes the following details about each request: diff --git a/doc/user/project/integrations/youtrack.md b/doc/user/project/integrations/youtrack.md index eda0874ac08..6c70a5e679b 100644 --- a/doc/user/project/integrations/youtrack.md +++ b/doc/user/project/integrations/youtrack.md @@ -14,7 +14,8 @@ You can configure YouTrack as an To enable the YouTrack integration in a project: -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **YouTrack**. 1. Select the checkbox under **Enable integration**. 1. Fill in the required fields: diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index e7bb5ad4eeb..e5dde0ed451 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -9,259 +9,265 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab 12.2. > - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab 12.4. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. +> - Design Management section in issues [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223193) in GitLab 13.2, with a feature flag named `design_management_moved`. In earlier versions, designs were displayed in a separate tab. +> - Design Management section in issues [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/223197) for new displays in GitLab 13.4. -Design Management allows you to upload design assets (including wireframes and mockups) -to GitLab issues and keep them stored in a single place, accessed by the Design -Management's page within an issue, giving product designers, product managers, and engineers a -way to collaborate on designs over a single source of truth. +With Design Management you can upload design assets (including wireframes and mockups) +to GitLab issues and keep them stored in a single place. Product designers, product managers, and +engineers can collaborate on designs with a single source of truth. -You can share mock-ups of designs with your team, or visual regressions can be +You can share mockups of designs with your team, or visual regressions can be viewed and addressed. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, see the video [Design Management (GitLab 12.2)](https://www.youtube.com/watch?v=CCMtCqdK_aM). +For a video overview, see [Design Management (GitLab 12.2)](https://www.youtube.com/watch?v=CCMtCqdK_aM). ## Requirements -Design Management requires -[Large File Storage (LFS)](../../../topics/git/lfs/index.md) -to be enabled: +- [Git Large File Storage (LFS)](../../../topics/git/lfs/index.md) must be enabled: + - On GitLab.com, LFS is already enabled. + - On self-managed instances, a GitLab administrator must + [enable LFS globally](../../../administration/lfs/index.md). + - On both GitLab.com and self-managed instances, LFS must be + [enabled for the project itself](../settings/index.md#sharing-and-permissions). + If enabled globally, LFS is enabled by default for all projects. If you have + disabled it for your project, you must enable it again. -- For GitLab.com, LFS is already enabled. -- For self-managed instances, a GitLab administrator must have - [enabled LFS globally](../../../administration/lfs/index.md). -- For both GitLab.com and self-managed instances: LFS must be enabled for the project itself. - If enabled globally, LFS is enabled by default to all projects. To enable LFS on the - project level, navigate to your project's **Settings > General**, expand **Visibility, project features, permissions** - and enable **Git Large File Storage**. + Designs are stored as LFS objects. + Image thumbnails are stored as other uploads, and are not associated with a project but rather + with a specific design model. -Design Management also requires that projects are using -[hashed storage](../../../administration/raketasks/storage.md#migrate-to-hashed-storage). -Newly created projects use hashed storage by default. A GitLab administrator -can verify the storage type of a project by going to **Admin Area > Projects** -and then selecting the project in question. A project can be identified as -hashed-stored if its *Gitaly relative path* contains `@hashed`. +- Projects must use + [hashed storage](../../../administration/raketasks/storage.md#migrate-to-hashed-storage). -If the requirements are not met, the **Designs** tab displays a message to the user. + Newly created projects use hashed storage by default. -## Supported files + A GitLab administrator can verify the storage type of a project by going to **Admin Area > Projects** + and then selecting the project in question. A project can be identified as + hashed-stored if its **Gitaly relative path** contains `@hashed`. -Files uploaded must have a file extension of either `png`, `jpg`, `jpeg`, -`gif`, `bmp`, `tiff`, `ico`, `webp`, or `svg`. +If the requirements are not met, you are notified in the **Designs** section. -Support for PDF is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/32811). +## Supported file types -## Limitations +You can upload files of the following types as designs: -- Design uploads are limited to 10 files at a time. -- From GitLab 13.1, Design filenames are limited to 255 characters. -- Design Management data - [isn't deleted when a project is destroyed](https://gitlab.com/gitlab-org/gitlab/-/issues/13429) yet. -- Design Management data [isn't deleted](https://gitlab.com/gitlab-org/gitlab/-/issues/13427) - when an issue is deleted. -- From GitLab 12.7, Design Management data [can be replicated](../../../administration/geo/replication/datatypes.md#limitations-on-replicationverification) +- BMP +- GIF +- ICO +- JPEG +- JPG +- PNG +- SVG +- TIFF +- WEBP + +Support for PDF files is tracked in [issue 32811](https://gitlab.com/gitlab-org/gitlab/-/issues/32811). + +## Known issues + +- Design Management data isn't deleted when: + - [A project is destroyed](https://gitlab.com/gitlab-org/gitlab/-/issues/13429). + - [An issue is deleted](https://gitlab.com/gitlab-org/gitlab/-/issues/13427). +- In GitLab 12.7 and later, Design Management data [can be replicated](../../../administration/geo/replication/datatypes.md#limitations-on-replicationverification) by Geo but [not verified](https://gitlab.com/gitlab-org/gitlab/-/issues/32467). -- Only the latest version of the designs can be deleted. -- Deleted designs cannot be recovered but you can see them on previous designs versions. -## GitLab-Figma plugin +## View a design -> [Introduced](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/issues/2) in GitLab 13.2. +The **Designs** section is in the issue description. + +Prerequisites: + +- You must have at least the Guest role for the project. + +To view a design: + +1. Go to an issue. +1. In the **Designs** section, select the design image you want to view. + +The design you selected opens. You can then [zoom in](#zoom-in-on-a-design) on it or +[create a comment](#add-a-comment-to-a-design). -Connect your design environment with your source code management in a seamless workflow. The GitLab-Figma plugin makes it quick and easy to collaborate in GitLab by bringing the work of product designers directly from Figma to GitLab Issues as uploaded Designs. + -To use the plugin, install it from the [Figma Directory](https://www.figma.com/community/plugin/860845891704482356) -and connect to GitLab through a personal access token. The details are explained in the [plugin documentation](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/wikis/home). +When viewing a design, you can move to other designs. To do so, either: -## The Design Management section +- In the top-right corner, select **Go to previous design** (**{angle-left}**) or **Go to next design** (**{angle-right}**). +- Press <kbd>Left</kbd> or <kbd>Right</kbd> on your keyboard. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223193) in GitLab 13.2. Designs are displayed directly in the issue description instead of a separate tab. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/223197) for new displays in GitLab 13.4. +To return to the issue view, either: -You can find to the **Design Management** section in the issue description: +- In the top-left corner, select the close icon (**{close}**). +- Press <kbd>Esc</kbd> on your keyboard. - +When a design is added, a green icon (**{plus-square}**) is displayed on the image +thumbnail. When a design has been [changed](#add-a-new-version-of-a-design) in the current version, +a blue icon (**{file-modified-solid}**) is displayed. -## Adding designs +### Zoom in on a design + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab 12.7. +> - Ability to drag a zoomed image to move it [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. + +You can explore a design in more detail by zooming in and out of the image: + +- To control the amount of zoom, select plus (`+`) and minus (`-`) + at the bottom of the image. +- To reset the zoom level, select the redo icon (**{redo}**). + +To move around the image while zoomed in, drag the image. + +## Add a design to an issue > - Drag and drop uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.9. > - New version creation on upload [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.9. > - Copy and paste uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202634) in GitLab 12.10. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. -To upload Design images, drag files from your computer and drop them in the Design Management section, -or select **click to upload** to select images from your file browser: +Prerequisites: - +- You must have at least the Developer role for the project. +- In GitLab 13.1 and later, the names of the uploaded files must be no longer than 255 characters. -You can drag and drop designs onto the dedicated drop zone to upload them. +To add a design to an issue: - +1. Go to an issue. +1. Either: + - Select **Upload designs** and then select images from your file browser. You can select up to + 10 files at once. + <!-- vale gitlab.SubstitutionWarning = NO --> + - Select **click to upload** and then select images from your file browser. You can select up to + 10 files at once. + <!-- vale gitlab.SubstitutionWarning = YES --> -You can also copy images from your file system and paste them directly on the -GitLab Design page as a new design. + - Drag a file from your file browser and drop it in the drop zone in the **Designs** section. -On macOS, you can take a screenshot and immediately copy it to the clipboard -by simultaneously pressing <kbd>Control</kbd> + <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>3</kbd>, -and then paste it as a design. +  -Copy-and-pasting has some limitations: + - Take a screenshot or copy a local image file into your clipboard, hover your cursor over the + drop zone, and press <kbd>Control</kbd> or <kbd>Cmd</kbd> + <kbd>V</kbd>. -- You can paste only one image at a time. When copy/pasting multiple files, only the first one is uploaded. -- All images are converted to `png` format under the hood, so when you want to copy/paste `gif` file, it results in broken animation. -- If you are pasting a screenshot from the clipboard, it is renamed to `design_<timestamp>.png` -- Copy/pasting designs is not supported on Internet Explorer. + When pasting images like this, keep the following in mind: -Designs with the same filename as an existing uploaded design create a new version -of the design, and replaces the previous version. Dropping a design on an -existing uploaded design creates a new version if the filenames are the same. - -### Skipped designs + - You can paste only one image at a time. When you paste multiple copied files, only the first + one is uploaded. + - If you are pasting a screenshot, the image is added as a PNG file with a generated name of: + `design_<timestamp>.png`. + - It's not supported in Internet Explorer. -Designs with the same filename as an existing uploaded design _and_ whose content has not changed are skipped. -This means that no new version of the design is created. When designs are skipped, you are made aware by a warning -message on the Issue. +## Add a new version of a design -## Viewing designs +As discussion on a design continues, you might want to upload a new version of a design. -Images on the Design Management page can be enlarged by selecting them. -You can navigate through designs by selecting the navigation buttons on the -top-right corner or with <kbd>Left</kbd>/<kbd>Right</kbd> keyboard buttons. +Prerequisites: -The number of discussions on a design — if any — is listed to the right -of the design filename. Selecting this number enlarges the design, -similar to clicking or tapping anywhere else in the design. -When a design is added or modified, an icon is displayed on the item -to help summarize changes between versions. +- You must have at least the Developer role for the project. -| Indicator | Example | -| --------- | ------- | -| Discussions |  | -| Modified (in the selected version) |  | -| Added (in the selected version) |  | +To do so, [add a design](#add-a-design-to-an-issue) with the same filename. -### Exploring designs by zooming +To browse all the design versions, use the dropdown list at the top of the **Designs** section. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab 12.7. -> - Ability to drag a zoomed image to move it [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. - -Designs can be explored in greater detail by zooming in and out of the image. -Control the amount of zoom with the `+` and `-` buttons at the bottom of the image. -While zoomed, you can still [start new discussions](#starting-discussions-on-designs) on the image, and see any existing ones. -While zoomed in, you can drag the image to move around it. +### Skipped designs - +When you upload an image with the same filename as an existing uploaded design _and_ that is the +same, it's skipped. This means that no new version of the design is created. +When designs are skipped, a warning message is displayed. -## Deleting designs +## Archive a design > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in GitLab 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/220964) the button from "Delete" to "Archive" in GitLab 13.3. -There are two ways to delete designs: manually delete them -individually, or select a few of them to delete at once, -as shown below. +You can archive individual designs or select a few of them to archive at once. -To delete a single design, select it to view it enlarged, -then select the trash icon on the top right corner and confirm -the deletion by selecting **Delete** in the window: +Prerequisites: - +- You must have at least the Developer role for the project. -To delete multiple designs at once, on the design's list view, -first select the designs you want to delete: +To archive a single design: - +1. Select the design to view it enlarged. +1. In the top right corner, select **Archive design** (**{archive}**). +1. Select **Archive designs**. -Select **Delete selected** to confirm the deletion: +To archive multiple designs at once: - +1. Select the checkboxes on the designs you want to archive. +1. Select **Archive selected**. NOTE: -Only the latest version of the designs can be deleted. -Deleted designs are not permanently lost; they can be -viewed by browsing previous versions. +Only the latest version of the designs can be archived. +Archived designs are not permanently lost. You can browse +[previous versions](#add-a-new-version-of-a-design). -## Reordering designs +## Reorder designs > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34382) in GitLab 13.3. You can change the order of designs by dragging them to a new position. -## Starting discussions on designs - -> - Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) adjusting a pin's position in GitLab 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) from GitLab Premium to GitLab Free in 13.0. +## Add a comment to a design -When a design is uploaded, you can start a discussion by selecting -the image on the exact location you would like the discussion to be focused on. -A pin is added to the image, identifying the discussion's location. +> Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab 12.8. - +You can start [discussions](../../discussions/index.md) on uploaded designs. To do so: -You can adjust a pin's position by dragging it around the image. This is useful -for when your design layout has changed between revisions, or if you need to move an -existing pin to add a new one in its place. +<!-- vale gitlab.SubstitutionWarning = NO --> +1. Go to an issue. +1. Select the design. +1. Click or tap the image. A pin is created in that spot, identifying the discussion's location. +1. Enter your message. +1. Select **Comment**. +<!-- vale gitlab.SubstitutionWarning = YES --> -Different discussions have different pin numbers: +You can adjust a pin's position by dragging it around the image. You can use this when your design's +layout has changed, or when you want to move a pin to add a new one in its place. - +New discussion threads get different pin numbers, which you can use to refer to them. In GitLab 12.5 and later, new discussions are output to the issue activity, so that everyone involved can participate in the discussion. -## Resolve Design threads +## Resolve a discussion thread on a design > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13049) in GitLab 13.1. -Discussion threads can be resolved on Designs. - -There are two ways to resolve/unresolve a Design thread: - -1. You can mark a thread as resolved or unresolved by selecting the checkmark icon for **Resolve thread** in the top-right corner of the first comment of the discussion: +When you're done discussing part of a design, you can resolve the discussion thread. -  +To mark a thread as resolved or unresolved, either: -1. Design threads can also be resolved or unresolved in their threads by using a checkbox. - When replying to a comment, you can select or clear a checkbox to resolve or unresolve - the thread after publishing: +- In the top-right corner of the first comment of the discussion, select **Resolve thread** or **Unresolve thread** (**{check-circle}**). +- Add a new comment to the thread and select or clear the **Resolve thread** checkbox. -  +Resolving a discussion thread also marks any pending [to-do items](../../todos.md) related to notes +inside the thread as done. Only to-do items for the user triggering the action are affected. -Resolving a discussion thread also marks any pending to-do items related to notes -inside the thread as done. This is applicable only for to-do items owned by the user triggering the action. +Your resolved comment pins disappear from the design to free up space for new discussions. +To revisit a resolved discussion, expand **Resolved Comments** below the visible threads. -Your resolved comment pins disappear from the Design to free up space for new discussions. -However, if you need to revisit or find a resolved discussion, all of your resolved threads are -available in the **Resolved Comment** area at the bottom of the right sidebar. - -## Add to-do items for designs +## Add a to-do item for a design > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198439) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245074) in GitLab 13.5. -Add a to-do item for a design by selecting **Add a to do** on the design sidebar: - - +To add a [to-do item](../../todos.md) for a design, select **Add a to do** on the design sidebar. -## Referring to designs in Markdown +## Refer to a design in Markdown > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217160) in GitLab 13.1. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258662) in GitLab 13.5. -We support referring to designs in [Markdown](../../markdown.md), which is available -throughout the application, including in merge request and issue descriptions, in discussions and comments, and in wiki pages. +To refer to a design in a [Markdown](../../markdown.md) text box in GitLab, for example, in +a comment or description, paste its URL. It's then displayed as a short reference. -Full URL references are supported. For example, if we refer to a design -somewhere with: +For example, if you refer to a design somewhere with: ```markdown -See https://gitlab.com/your-group/your-project/-/issues/123/designs/homescreen.png +See https://gitlab.com/gitlab-org/gitlab/-/issues/13195/designs/Group_view.png. ``` -This is rendered as: +It's rendered as: -> See [#123[homescreen.png]](https://gitlab.com/your-group/your-project/-/issues/123/designs/homescreen.png) +> See [#13195[Group_view.png]](https://gitlab.com/gitlab-org/gitlab/-/issues/13195/designs/Group_view.png). ## Design activity records @@ -272,3 +278,15 @@ User activity events on designs (creation, deletion, and updates) are tracked by displayed on the [user profile](../../profile/index.md#access-your-user-profile), [group](../../group/index.md#view-group-activity), and [project](../working_with_projects.md#view-project-activity) activity pages. + +## GitLab-Figma plugin + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/issues/2) in GitLab 13.2. + +You can use the GitLab-Figma plugin to upload your designs from Figma directly to your issues +in GitLab. + +To use the plugin in Figma, install it from the [Figma Directory](https://www.figma.com/community/plugin/860845891704482356) +and connect to GitLab through a personal access token. + +For more information, see the [plugin documentation](https://gitlab.com/gitlab-org/gitlab-figma-plugin/-/wikis/home). diff --git a/doc/user/project/issues/img/adding_note_to_design_1.png b/doc/user/project/issues/img/adding_note_to_design_1.png Binary files differdeleted file mode 100644 index 3c25fcb1241..00000000000 --- a/doc/user/project/issues/img/adding_note_to_design_1.png +++ /dev/null diff --git a/doc/user/project/issues/img/adding_note_to_design_2.png b/doc/user/project/issues/img/adding_note_to_design_2.png Binary files differdeleted file mode 100644 index c418a0364c0..00000000000 --- a/doc/user/project/issues/img/adding_note_to_design_2.png +++ /dev/null diff --git a/doc/user/project/issues/img/confirm_design_deletion_v12_4.png b/doc/user/project/issues/img/confirm_design_deletion_v12_4.png Binary files differdeleted file mode 100644 index 5631b6ec98e..00000000000 --- a/doc/user/project/issues/img/confirm_design_deletion_v12_4.png +++ /dev/null diff --git a/doc/user/project/issues/img/delete_multiple_designs_v12_4.png b/doc/user/project/issues/img/delete_multiple_designs_v12_4.png Binary files differdeleted file mode 100644 index 40d449d5b39..00000000000 --- a/doc/user/project/issues/img/delete_multiple_designs_v12_4.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_added_v12_3.png b/doc/user/project/issues/img/design_added_v12_3.png Binary files differdeleted file mode 100644 index 92aa953db8e..00000000000 --- a/doc/user/project/issues/img/design_added_v12_3.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_comments_v12_3.png b/doc/user/project/issues/img/design_comments_v12_3.png Binary files differdeleted file mode 100644 index c01a8bb7ba7..00000000000 --- a/doc/user/project/issues/img/design_comments_v12_3.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_drag_and_drop_uploads_v13_2.png b/doc/user/project/issues/img/design_drag_and_drop_uploads_v13_2.png Binary files differindex 4ab5e184905..0225505e829 100644 --- a/doc/user/project/issues/img/design_drag_and_drop_uploads_v13_2.png +++ b/doc/user/project/issues/img/design_drag_and_drop_uploads_v13_2.png diff --git a/doc/user/project/issues/img/design_management_upload_v13.3.png b/doc/user/project/issues/img/design_management_upload_v13.3.png Binary files differdeleted file mode 100644 index f7b3f79fb22..00000000000 --- a/doc/user/project/issues/img/design_management_upload_v13.3.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_management_v13_2.png b/doc/user/project/issues/img/design_management_v13_2.png Binary files differdeleted file mode 100644 index 3da11c92514..00000000000 --- a/doc/user/project/issues/img/design_management_v13_2.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_management_v14_10.png b/doc/user/project/issues/img/design_management_v14_10.png Binary files differnew file mode 100644 index 00000000000..a10be15eafd --- /dev/null +++ b/doc/user/project/issues/img/design_management_v14_10.png diff --git a/doc/user/project/issues/img/design_modified_v12_3.png b/doc/user/project/issues/img/design_modified_v12_3.png Binary files differdeleted file mode 100644 index 01b752fa531..00000000000 --- a/doc/user/project/issues/img/design_modified_v12_3.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_todo_button_v13_5.png b/doc/user/project/issues/img/design_todo_button_v13_5.png Binary files differdeleted file mode 100644 index 970161a6097..00000000000 --- a/doc/user/project/issues/img/design_todo_button_v13_5.png +++ /dev/null diff --git a/doc/user/project/issues/img/design_zooming_v12_7.png b/doc/user/project/issues/img/design_zooming_v12_7.png Binary files differdeleted file mode 100644 index 4966af06e41..00000000000 --- a/doc/user/project/issues/img/design_zooming_v12_7.png +++ /dev/null diff --git a/doc/user/project/issues/img/resolve_design-discussion_checkbox_v13_1.png b/doc/user/project/issues/img/resolve_design-discussion_checkbox_v13_1.png Binary files differdeleted file mode 100644 index 8791419b919..00000000000 --- a/doc/user/project/issues/img/resolve_design-discussion_checkbox_v13_1.png +++ /dev/null diff --git a/doc/user/project/issues/img/resolve_design-discussion_icon_v13_1.png b/doc/user/project/issues/img/resolve_design-discussion_icon_v13_1.png Binary files differdeleted file mode 100644 index fc1fff321ba..00000000000 --- a/doc/user/project/issues/img/resolve_design-discussion_icon_v13_1.png +++ /dev/null diff --git a/doc/user/project/issues/img/select_designs_v12_4.png b/doc/user/project/issues/img/select_designs_v12_4.png Binary files differdeleted file mode 100644 index fe1c55a4ae2..00000000000 --- a/doc/user/project/issues/img/select_designs_v12_4.png +++ /dev/null diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md deleted file mode 100644 index e9f3f4be1c3..00000000000 --- a/doc/user/project/issues/issue_data_and_actions.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-02-24' ---- - -This file was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2022-02-24>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 58591129d97..4508ef30ac5 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -342,7 +342,7 @@ To do it: issues.each do |issue| if issue.state != "closed" && issue.moved_to.nil? - Issues::MoveService.new(project, admin_user).execute(issue, target_project) + Issues::MoveService.new(project: project, current_user: admin_user).execute(issue, target_project) else puts "issue with id: #{issue.id} and title: #{issue.title} was not moved" end diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index 9311ef590df..95a7e9387e8 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -6,20 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Sorting and ordering issue lists **(FREE)** -You can sort a list of issues several ways, including by: - -- [Blocking issues](#sorting-by-blocking-issues) -- [Created date](#sorting-by-created-date) -- [Due date](#sorting-by-due-date) -- [Label priority](#sorting-by-label-priority) -- [Last updated](#sorting-by-last-updated) -- [Manual sorting](#manual-sorting) -- [Milestone due date](#sorting-by-milestone-due-date) -- [Popularity](#sorting-by-popularity) -- [Priority](#sorting-by-priority) -- [Title](#sorting-by-title) -- [Weight](#sorting-by-weight) - +You can sort a list of issues several ways. The available sorting options can change based on the context of the list. ## Sorting by blocking issues **(PREMIUM)** @@ -51,10 +38,10 @@ For more information, see [issue 14523](https://gitlab.com/gitlab-org/gitlab/-/i To learn how to change label priority, see [Label priority](../labels.md#set-label-priority). -## Sorting by last updated +## Sorting by updated date -When you sort by **Last updated**, the issue list changes to sort by the time of a last -update. Issues changed the most recently are first. +When you sort by **Updated date**, the issue list changes to sort by the time of a last +update. Issues changed the most recently are shown first. ## Manual sorting diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 18197cd860f..2cc23b14857 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -253,6 +253,35 @@ with the old labels are assigned to the new group label. The new group label has the same ID as the previous project label. +## Promote a subgroup label to the parent group + +It's not possible to directly promote a group label to the parent group. +To achieve this, use the following workaround. + +Prerequisites: + +- There must be a group that contains subgroups ("parent group"). +- There must be a subgroup in the parent group, that has a label you want to promote. +- You must have at least the Reporter role for both groups. + +To "promote" the label to the parent group: + +1. In the parent group, [create a label](#create-a-group-label) with the same name as the original + one. We recommend making it a different color so you don't mistake the two while you're doing this. +1. In the subgroup, [view its labels](#view-group-labels). You should see the two labels and where + they come from: + +  + +1. Next to the subgroup label (the old one), select **Issues**, **Merge requests**, or **Epics**. +1. Add the new label to issues, merge requests, and epics that have the old label. + To do it faster, use [bulk editing](issues/managing_issues.md#bulk-edit-issues-from-a-group). +1. In the subgroup or the parent group, [delete the label](#delete-a-group-label) that belongs to + the lower-level group. + +You should now have a label in the parent group that is named the same as the old one, and added +to the same issues, MRs, and epics. + ## Generate default project labels If a project or its parent group has no labels, you can generate a default set of project @@ -420,6 +449,22 @@ The labels higher in the list get higher priority. To learn what happens when you sort by priority or label priority, see [Sorting and ordering issue lists](issues/sorting_issue_lists.md). +## Real-time changes to labels + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241538) in GitLab 14.10 with a [feature flag](../../administration/feature_flags.md) named `realtime_labels`, disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an +administrator to [enable the feature flag](../../administration/feature_flags.md) named `realtime_labels`. +On GitLab.com, this feature is unavailable. + +Changed labels are immediately visible to other users, without refreshing the page, on the following: + +- Epics +- Incidents +- Issues +- Merge requests + ## Troubleshooting ### Some label titles end with `_duplicate<number>` diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index c3d5dca0675..ff4677eddde 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Authentication and Authorization +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -49,7 +49,7 @@ flowchart RL [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. Add users to a project so they become members and have permission -to perform actions. +to perform actions. The Owner [role](../../permissions.md#project-members-permissions) can only be added at the group level. Prerequisite: @@ -145,7 +145,10 @@ In this example: - **Administrator** is the [Owner](../../permissions.md) and member of all groups. They have inherited their role from the **demo** group. -If a user is a direct member of a project, the expiration date can be updated. If membership is inherited from a parent group, the expiration date can be updated only from the parent group itself. +If a user is a: + +- Direct member of a project, the **Expiration** and **Max role** fields can be updated directly on the project. +- Inherited member from a parent group, the **Expiration** and **Max role** fields must be updated on the parent group. ## Remove a member from a project diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 0a7fbc9ee95..0ede9310393 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -146,7 +146,7 @@ You can also enforce merge request approval settings: - At the [instance level](../../../admin_area/merge_requests_approvals.md), which apply to all groups on an instance and, therefore, all projects. -- On a [top-level group](../../../group/index.md#group-approval-rules), which apply to all subgroups +- On a [top-level group](../../../group/index.md#group-approval-settings), which apply to all subgroups and projects. If the settings are inherited by a group or project, they cannot be changed in the group or project diff --git a/doc/user/project/merge_requests/fast_forward_merge.md b/doc/user/project/merge_requests/fast_forward_merge.md index dc13b270f17..77162aa0b83 100644 --- a/doc/user/project/merge_requests/fast_forward_merge.md +++ b/doc/user/project/merge_requests/fast_forward_merge.md @@ -13,8 +13,6 @@ merge commits. In such cases, the fast-forward merge is the perfect candidate. With fast-forward merge requests, you can retain a linear Git history and a way to accept merge requests without creating merge commits. -## Overview - When the fast-forward merge ([`--ff-only`](https://git-scm.com/docs/git-merge#git-merge---ff-only)) setting is enabled, no merge commits are created and all merges are fast-forwarded, @@ -22,6 +20,11 @@ which means that merging is only allowed if the branch can be fast-forwarded. When a fast-forward merge is not possible, the user is given the option to rebase. +NOTE: +Projects using the fast-forward merge strategy can't filter merge requests +[by deployment date](../../search/index.md#filtering-merge-requests-by-environment-or-deployment-date), +because no merge commit is created. + ## Enabling fast-forward merges 1. On the top bar, select **Menu > Projects** and find your project. diff --git a/doc/user/project/merge_requests/img/attention_request_list_v14_10.png b/doc/user/project/merge_requests/img/attention_request_list_v14_10.png Binary files differnew file mode 100644 index 00000000000..00427a0aa40 --- /dev/null +++ b/doc/user/project/merge_requests/img/attention_request_list_v14_10.png diff --git a/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png b/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png Binary files differnew file mode 100644 index 00000000000..174cf01dbb0 --- /dev/null +++ b/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png diff --git a/doc/user/project/merge_requests/img/ff_merge_rebase_v14_9.png b/doc/user/project/merge_requests/img/ff_merge_rebase_v14_9.png Binary files differindex f4330549a57..17ce42e7a69 100644 --- a/doc/user/project/merge_requests/img/ff_merge_rebase_v14_9.png +++ b/doc/user/project/merge_requests/img/ff_merge_rebase_v14_9.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 9872bd2e936..a3b9fb52f0d 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -70,6 +70,53 @@ change and whether you need access to a development environment: - [Push changes from the command line](../../../gitlab-basics/start-using-git.md), if you are familiar with Git and the command line. +## Request attention to a merge request + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343528) in GitLab 14.10 [with a flag](../../../administration/feature_flags.md) named `mr_attention_requests`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `mr_attention_requests`. +On GitLab.com, this feature is dependent on the enablement status of the feature flag. Refer to the [enablement issue](https://gitlab.com/gitlab-org/gitlab/-/issues/343528) for details. + +To tell a merge request's assignee or reviewer that their attention is +needed on a merge request, you can request their attention. If an assignee or a +reviewer has their attention requested on a merge request, the **Attention request** +icon (**{attention}**) is displayed as a solid icon (**{attention-solid}**) on +the merge request list page: + + + +To view a list of merge requests that need your attention: + +1. On the top bar, select **Merge requests** (**{merge-request}**). +1. Select **Attention requests**. + +To request attention from another user, use the `/attention @user` +[quick action](../quick_actions.md) or: + +1. Go to the merge request. +1. On the right sidebar, identify the user you want to request attention from. +1. Next to the user's name, select **Request attention** (**{attention}**), and the appearance + of the icon changes: + +  + +### Remove an attention request + +If your attention was requested as an assignee or reviewer, it's removed when you: + +- Manually remove the attention request by selecting **Remove attention request** (**{attention-solid}**). +- Approve the merge request. +- Add a new user as an assignee or reviewer. +- Request the attention of a different assignee or reviewer. +- Remove yourself (or are removed by someone else) as an assignee or reviewer. +- Merge or close the merge request. + +If you are both the assignee and a reviewer on a merge request, you receive +only one attention request, which is synced across both duties. If the +attention request is removed from you, either as an assignee or a reviewer, +it is removed from both your duties. + ## Close a merge request If you decide to permanently stop work on a merge request, diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 280ae07b401..512faae82a9 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -112,7 +112,13 @@ This example shows reviewers and approval rules in a merge request sidebar: ### Request a new review -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) in GitLab 14.10. + +WARNING: +This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) +in GitLab 14.10, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) in GitLab 15.0. +Use [attention requests](../index.md#request-attention-to-a-merge-request) instead. After a reviewer completes their [merge request reviews](../../../discussions/index.md), the author of the merge request can request a new review from the reviewer: diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 9868f2619ba..8e6794bcfa7 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -108,6 +108,8 @@ For example, to customize the commit message to output **Addresses user_1's review**, set the custom text to `Addresses %{username}'s review`. +For merge requests created from forks, GitLab uses the template defined in target project. + NOTE: Custom commit messages for each applied suggestion is introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index a952c0550bc..76a67487881 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -54,6 +54,9 @@ External status checks have the following states: Support for adding a `failed` state is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338827). +If something changes outside of GitLab, you can [set the status of an external status check](../../../api/status_checks.md#set-status-of-an-external-status-check) +using the API. You don't need to wait for a merge request webhook payload to be sent first. + ## View the status checks on a project Within each project's settings, you can see a list of status checks added to the project: diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index d7177208a6e..9f1e5ae7046 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -28,7 +28,7 @@ between pipeline completion and the visualization loading on the page. For the coverage analysis to work, you have to provide a properly formatted [Cobertura XML](https://cobertura.github.io/cobertura/) report to -[`artifacts:reports:cobertura`](../../../ci/yaml/artifacts_reports.md#artifactsreportscobertura). +[`artifacts:reports:cobertura`](../../../ci/yaml/artifacts_reports.md#artifactsreportscobertura-deprecated). This format was originally developed for Java, but most coverage analysis frameworks for other languages have plugins to add support for it, like: @@ -156,7 +156,9 @@ test: - npx nyc --reporter cobertura mocha artifacts: reports: - cobertura: coverage/cobertura-coverage.xml + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml ``` ### Java and Kotlin examples @@ -324,18 +326,13 @@ run tests: The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Go uses: - [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests. -- [`gocover-cobertura`](https://github.com/t-yuki/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. +- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. -This example assumes that [Go modules](https://go.dev/ref/mod) are being used. -Using Go modules causes paths within the coverage profile to be prefixed with your -project's module identifier, which can be found in the `go.mod` file. This -prefix must be removed for GitLab to parse the Cobertura XML file correctly. You can use the following `sed` command to remove the prefix: - -```shell -sed -i 's;filename=\"<YOUR_MODULE_ID>/;filename=\";g' coverage.xml -``` - -Replace the `gitlab.com/my-group/my-project` placeholder in the following example with your own module identifier to make it work. +This example assumes that [Go modules](https://go.dev/ref/mod) +are being used. Please note that the `-covermode count` option does not work with the `-race` flag. +If you want to generate code coverage while also using the `-race` flag, you must switch to +`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover) +for more details. ```yaml run tests: @@ -343,9 +340,9 @@ run tests: image: golang:1.17 script: - go install - - go test . -coverprofile=coverage.txt -covermode count - - go run github.com/t-yuki/gocover-cobertura < coverage.txt > coverage.xml - - sed -i 's;filename=\"gitlab.com/my-group/my-project/;filename=\";g' coverage.xml + - go test ./... -coverprofile=coverage.txt -covermode count + - go get github.com/boumenot/gocover-cobertura + - go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml artifacts: reports: cobertura: coverage.xml diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md index 3491346f7d9..5433e02b210 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -1,7 +1,7 @@ --- type: concepts -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 4d8919090a2..d970c0f9ef4 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -1,7 +1,7 @@ --- disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pages/getting_started_part_three.html' -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index f09aea3b02a..cb22a200514 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -1,14 +1,14 @@ --- type: reference description: "Automatic Let's Encrypt SSL certificates for GitLab Pages." -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Pages integration with Let's Encrypt **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28996) in GitLab 12.1. For versions earlier than GitLab 12.1, see the [manual Let's Encrypt instructions](../lets_encrypt_for_gitlab_pages.md). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28996) in GitLab 12.1. The GitLab Pages integration with Let's Encrypt (LE) allows you to use LE certificates for your Pages website with custom domains diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md index 21f2dd51f70..0c848a24dec 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -1,7 +1,7 @@ --- type: concepts -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index 25382778b1d..510f9332e7b 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -1,7 +1,7 @@ --- type: reference, howto -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index b43af2f0efe..71ed3134c1e 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -1,7 +1,7 @@ --- type: reference, howto -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index e0c10e27ec3..f08afc924f3 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index cee10675a62..b32d71a4887 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 5773dd1f2c0..54b843945cd 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 82b1a824f7a..af49522efe2 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -1,7 +1,7 @@ --- description: 'Learn how to use GitLab Pages to deploy a static website at no additional cost.' -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index e4bc58854ef..f274338c2fd 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md deleted file mode 100644 index 7779f87b459..00000000000 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'custom_domains_ssl_tls_certification/lets_encrypt_integration.md' -remove_date: '2022-03-14' ---- - -This file was moved to [another location](custom_domains_ssl_tls_certification/lets_encrypt_integration.md). - -<!-- This redirect file can be deleted after <2022-03-14>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/pages/pages_access_control.md b/doc/user/project/pages/pages_access_control.md index 002b234f561..9b747e04973 100644 --- a/doc/user/project/pages/pages_access_control.md +++ b/doc/user/project/pages/pages_access_control.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index cdae1d2f837..1db404f4888 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 292530e6c9c..06396b5cd62 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -147,8 +147,8 @@ Deploy keys are not available in the **Allowed to merge** dropdown list. ## Allow force push on a protected branch -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15611) in GitLab 13.10 behind a disabled feature flag. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323431) in GitLab 14.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15611) in GitLab 13.10 [with a flag](../../administration/feature_flags.md) named `allow_force_push_to_protected_branches`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/323431) in GitLab 14.0. Feature flag `allow_force_push_to_protected_branches` removed. You can allow [force pushes](../../topics/git/git_rebase.md#force-push) to protected branches. diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index 20dd37578fd..d04f79d11c7 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -35,7 +35,7 @@ You can use push options to skip a CI/CD pipeline, or pass CI/CD variables. | Push option | Description | Introduced in version | | ------------------------------ | ------------------------------------------------------------------------------------------- |---------------------- | -| `ci.skip` | Do not create a CI pipeline for the latest push. | [11.7](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15643) | +| `ci.skip` | Do not create a CI pipeline for the latest push. Only skips branch pipelines and not [merge request pipelines](../../ci/pipelines/merge_request_pipelines.md). | [11.7](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15643) | | `ci.variable="<name>=<value>"` | Provide [CI/CD variables](../../ci/variables/index.md) to be used in a CI pipeline, if one is created due to the push. | [12.6](https://gitlab.com/gitlab-org/gitlab/-/issues/27983) | An example of using `ci.skip`: diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index ae42d90af06..b73b381ffcd 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -55,6 +55,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself. | | `/assign_reviewer @user1 @user2` or `/reviewer @user1 @user2` or `/request_review @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users as reviewers. | | `/assign_reviewer me` or `/reviewer me` or `/request_review me` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself as a reviewer. | +| `/attention @user1` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | [Request attention](merge_requests/index.md#request-attention-to-a-merge-request) to a merge request from a user. | | `/award :emoji:` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Toggle emoji award. | | `/child_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Add child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7330) in GitLab 12.0). | | `/clear_health_status` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Clear [health status](issues/managing_issues.md#health-status) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213814) in GitLab 14.7). | diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index ddeef65a6b5..0e6c98457c7 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -32,7 +32,7 @@ To fork an existing project in GitLab: It must be unique in the namespace. 1. Optional. Add a **Project description**. 1. Select the **Visibility level** for your fork. For more information about - visibility levels, read [Project and group visibility](../../../public_access/public_access.md). + visibility levels, read [Project and group visibility](../../public_access.md). 1. Select **Fork project**. GitLab creates your fork, and redirects you to the new fork's page. diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index cd0c6679d8d..39b57f89ceb 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -25,8 +25,14 @@ GitLab. ## Cleaner diffs -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6589) in GitLab 14.5 [with a flag](../../../../administration/feature_flags.md) named `jupyter_clean_diffs`. Enabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6589) in GitLab 14.5 as an [Alpha](../../../../policy/alpha-beta-support.md#alpha-features) release [with a flag](../../../../administration/feature_flags.md) named `jupyter_clean_diffs`. Enabled by default. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75500) in GitLab 14.9. Feature flag `jupyter_clean_diffs` removed. +> - [Reintroduced toggle](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85079) in GitLab 15.0 [with a flag](../../../../administration/feature_flags.md) named `ipynb_semantic_diff`. Enabled by default. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `ipynb_semantic_diff`. +On GitLab.com, this feature is available. +This feature is ready for production use. When commits include changes to Jupyter Notebook files, GitLab: @@ -37,6 +43,10 @@ Code suggestions are not available on diffs and merge requests for `.ipynb` file  +This feature is an [Alpha](../../../../policy/alpha-beta-support.md#alpha-features) release, +and might lead to performance degradation. On self-managed GitLab, if unexpected issues +arise, disable the feature. + ## Jupyter Git integration Jupyter can be configured as an OAuth application with repository access, acting diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index c9fa30e39a8..e1017e78437 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -7,9 +7,13 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.htm # Repository mirroring **(FREE)** -You can _mirror_ a repository to and from external sources. You can select which -repository serves as the source, and modify which parts of the repository are copied. -Branches, tags, and commits can be mirrored. +You can _mirror_ a repository to and from external sources. You can select which repository +serves as the source. Branches, tags, and commits can be mirrored. + +NOTE: +SCP-style URLs are **not** supported. However, the work for implementing SCP-style URLs is tracked +in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/18993). +Subscribe to the issue to follow its progress. Several mirroring methods exist: @@ -23,11 +27,7 @@ Mirror a repository when: copy of your project at its previous home, configure your GitLab repository as a [push mirror](push.md). Changes you make to your GitLab repository are copied to the old location. -- Your GitLab project is private, but some components can be shared publicly. - Configure your primary repository as a [push mirror](push.md) and push the portions - you want to make public. With this configuration, you can open-source specific - projects, contribute back to the open-source community, and protect the sensitive - parts of your project. +- Your GitLab instance is private, but you want to open-source some projects. - You migrated to GitLab, but the canonical version of your project is somewhere else. Configure your GitLab repository as a [pull mirror](pull.md) of the other project. Your GitLab repository pulls copies of the commits, tags, and branches of project. diff --git a/doc/user/project/repository/repository_mirroring.md b/doc/user/project/repository/repository_mirroring.md deleted file mode 100644 index 8fbe5aec6a3..00000000000 --- a/doc/user/project/repository/repository_mirroring.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'mirror/index.md' -remove_date: '2022-03-22' ---- - -This document was moved to [another location](mirror/index.md). - -<!-- This redirect file can be deleted after <2022-03-22>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 4165c1828cc..6eed1717507 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -117,12 +117,19 @@ There are multiple ways to create a branch from the GitLab web interface. ### Create a new branch from an issue +> The **Create merge request** button [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/349566) to open the merge request creation form in GitLab 14.8. + If your development workflow requires an issue for every merge request, you can create a branch directly from the issue to speed the process up. The new branch, and later its merge request, are marked as related to this issue. Once merged, the merge request closes the issue. You can see a **Create merge request** dropdown below the issue description. +NOTE: +In GitLab 14.8 and later, selecting **Create merge request** +[redirects to the merge request creation form](https://gitlab.com/gitlab-org/gitlab/-/issues/349566) +instead of immediately creating the merge request. + The **Create merge request** button doesn't display if: - A branch with the same name already exists. diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index c9cddad1a91..0259ff6ce30 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -88,7 +88,7 @@ If you have the correct version, you can proceed to configure Git. Configure Git to use your key for signing: ```shell -signingkey = $( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) +signingkey=$( gpgsm --list-secret-keys | egrep '(key usage|ID)' | grep -B 1 digitalSignature | awk '/ID/ {print $2}' ) git config --global user.signingkey $signingkey git config --global gpg.format x509 ``` diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 6abe9c08d28..8b80c494e2f 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -127,7 +127,7 @@ To search for a requirement: You can also sort the requirements list by: - Created date -- Last updated +- Updated date ## Allow requirements to be satisfied from a CI job diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index 90732ba4fe2..19c3218137f 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -241,7 +241,8 @@ The configuration options are the same as for configuring ##### Microsoft Graph -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214900) in GitLab 13.11. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214900) in GitLab 13.11. +> - Alternative Azure deployments [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5978) in GitLab 14.9. Service Desk can be configured to read Microsoft Exchange Online mailboxes with the Microsoft Graph API instead of IMAP. Follow the [documentation in the incoming email section for setting up an OAuth2 application for Microsoft Graph](../../administration/incoming_email.md#microsoft-graph). @@ -263,6 +264,22 @@ Graph API instead of IMAP. Follow the [documentation in the incoming email secti } ``` +For Microsoft Cloud for US Government or [other Azure deployments](https://docs.microsoft.com/en-us/graph/deployments), configure the `azure_ad_endpoint` and `graph_endpoint` settings. + +- Example for Microsoft Cloud for US Government: + +```ruby + gitlab_rails['service_desk_email_inbox_options'] = { + 'azure_ad_endpoint': 'https://login.microsoftonline.us', + 'graph_endpoint': 'https://graph.microsoft.us', + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional + } +} +``` + The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. #### Configuring a custom email address suffix @@ -330,20 +347,6 @@ Note that: - The project's visibility (private, internal, public) does not affect Service Desk. - The path to the project, including its group or namespace, is shown in emails. -#### Issues created on someone's behalf - -To allow third party applications and ticketing systems to interface with Service Desk, -when the email contains the `Reply-To` email header, this email address is used as the address of the -issue author. - -Because the `Reply-To` header can be set to arbitrary values, do not blindly trust that an issue -created on behalf of `someone@example.com` was indeed created by the real owner of such email address. - -For example, an email with headers `To: support@example.com` and `Reply-To:someone@example.com` -creates an issue with the following note: - -> Created (…) by `support@example.com` (reply to: `someone@example.com`) (…) - #### Privacy considerations Service Desk issues are confidential, but the project owner can diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index ae3decb8079..30261ed5082 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -72,13 +72,14 @@ The following items are exported: The following items are **not** exported: +- [Child pipeline history](https://gitlab.com/gitlab-org/gitlab/-/issues/221088) - Build traces and artifacts - Container registry images - CI/CD variables - Pipeline triggers - Webhooks - Any encrypted tokens -- Merge Request Approvers +- Merge Request Approvers and [the number of required approvals](https://gitlab.com/gitlab-org/gitlab/-/issues/221088) - Repository size limits - Deploy keys allowed to push to protected branches @@ -127,7 +128,7 @@ The following items are imported but changed slightly: associated with such merge requests are created in a project during the import/export. Thus, the number of branches in the exported project might be bigger than in the original project. - If use of the `Internal` visibility level - [is restricted](../../../public_access/public_access.md#restrict-use-of-public-or-internal-projects), + [is restricted](../../public_access.md#restrict-use-of-public-or-internal-projects), all imported projects are given `Private` visibility. Deploy keys aren't imported. To use deploy keys, you must enable them in your imported project and update protected branches. @@ -154,9 +155,9 @@ The default is `0` (unlimited). Imported users can be mapped by their public email addresses on self-managed instances, if an administrator (not an owner) does the import. -- Public email addresses are not set by default. Users must -[set it in their profiles](../../profile/index.md#set-your-public-email) -for mapping to work correctly. +- The project must be exported by a project or group member with the Owner role. +- Public email addresses are not set by default. Users must [set it in their profiles](../../profile/index.md#set-your-public-email) + for mapping to work correctly. - For contributions to be mapped correctly, users must be an existing member of the namespace, or they can be added as a member of the project. Otherwise, a supplementary comment is left to mention that the original author and the MRs, notes, or issues that are owned by the importer. - Imported users are set as [direct members](../members/index.md) @@ -237,7 +238,7 @@ and the exports between them are compatible. ### Project fails to import due to mismatch -If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners) +If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners-for-a-project) does not match between the exported project, and the project import, the project fails to import. Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either: @@ -306,7 +307,7 @@ reduce the repository size for another import attempt: #### Workaround option 2 NOTE: -This workaround requires access to the rails console, which isn't available to end-users on GitLab.com. +This workaround does not account for LFS objects. Rather than attempting to push all changes at once, this workaround: @@ -383,3 +384,17 @@ s = Gitlab::ImportExport::Saver.new(exportable: p, shared:p.import_export_shared s.send(:compress_and_save) s.send(:save_upload) ``` + +### Import using the REST API fails when using a group access token + +[Group access tokens](../../group/settings/group_access_tokens.md) +don't work for project or group import operations. When a group access token initiates an import, +the import fails with this message: + +```plaintext +Error adding importer user to Project members. +Validation failed: User project bots cannot be added to other groups / projects +``` + +To use [Import REST APIs](../../../api/project_import_export.md), +pass regular user account credentials such as [personal access tokens](../../profile/personal_access_tokens.md). diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 342b8d80bcf..31cda756a78 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -89,15 +89,11 @@ read-only view to discourage this behavior. Compliance framework pipelines allow group owners to define a compliance pipeline in a separate repository that gets executed in place of the local project's `gitlab-ci.yml` file. As part of this pipeline, an -`include` statement can reference the local project's `gitlab-ci.yml` file. This way, the two CI -files are merged together any time the pipeline runs. Jobs and variables defined in the compliance +`include` statement can reference the local project's `gitlab-ci.yml` file. This way, the compliance +pipeline jobs can run alongside the project-specific jobs any time the pipeline runs. +Jobs and variables defined in the compliance pipeline can't be changed by variables in the local project's `gitlab-ci.yml` file. -When used to enforce scan execution, this feature has some overlap with [scan execution policies](../../application_security/policies/scan-execution-policies.md), -as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). -For details on the similarities and differences between these features, see -[Enforce scan execution](../../application_security/#enforce-scan-execution). - When you set up the compliance framework, use the **Compliance pipeline configuration** box to link the compliance framework to specific CI/CD configuration. Use the `path/file.y[a]ml@group-name/project-name` format. For example: @@ -185,6 +181,11 @@ include: # Execute individual project's configuration (if project contains .git ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch ``` +When used to enforce scan execution, this feature has some overlap with [scan execution policies](../../application_security/policies/scan-execution-policies.md), +as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). +For details on the similarities and differences between these features, see +[Enforce scan execution](../../application_security/#enforce-scan-execution). + ##### Ensure compliance jobs are always run Compliance pipelines use GitLab CI/CD to give you an incredible amount of flexibility @@ -242,7 +243,7 @@ documentation, access permissions, and more. To do so from your project, go to **Settings** > **General**, and expand the **Visibility, project features, permissions** section. -You can now change the [Project visibility](../../../public_access/public_access.md). +You can now change the [Project visibility](../../public_access.md). If you set **Project Visibility** to public, you can limit access to some features to **Only Project Members**. In addition, you can select the option to [Allow users to request access](../members/index.md#request-access-to-a-project). diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index a78226ac2f8..b66913b7223 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -12,17 +12,18 @@ type: reference, howto > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235765) in GitLab 13.5. > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5. Default prefix added. -You can use a project access token to authenticate: +Project access tokens are similar to passwords, except you can [limit access to resources](#scopes-for-a-project-access-token), +select a limited role, and provide an expiry date. -- With the [GitLab API](../../../api/index.md#personalprojectgroup-access-tokens). -- With Git, when using HTTP Basic Authentication. +Use a project access token to authenticate: -After you configure a project access token, you don't need a password when you authenticate. -Instead, you can enter any non-blank value. +- With the [GitLab API](../../../api/index.md#personalprojectgroup-access-tokens). +- With Git, when using HTTP Basic Authentication, use: + - Any non-blank value as a username. + - The project access token as the password. Project access tokens are similar to [group access tokens](../../group/settings/group_access_tokens.md) -and [personal access tokens](../../profile/personal_access_tokens.md), except they are -associated with a project rather than a group or user. +and [personal access tokens](../../profile/personal_access_tokens.md). In self-managed instances, project access tokens are subject to the same [maximum lifetime limits](../../admin_area/settings/account_and_limit_settings.md#limit-the-lifetime-of-personal-access-tokens) as personal access tokens if the limit is set. @@ -35,6 +36,8 @@ You can use project access tokens: - Consider [disabling project access tokens](#enable-or-disable-project-access-token-creation) to lower potential abuse. +You cannot use project access tokens to create other access tokens. + Project access tokens inherit the [default prefix setting](../../admin_area/settings/account_and_limit_settings.md#personal-access-token-prefix) configured for personal access tokens. diff --git a/doc/user/project/static_site_editor/index.md b/doc/user/project/static_site_editor/index.md index 7c74a625b92..220623d0372 100644 --- a/doc/user/project/static_site_editor/index.md +++ b/doc/user/project/static_site_editor/index.md @@ -17,7 +17,7 @@ description: "The static site editor enables users to edit content on static web WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77246) -for use in GitLab 14.7, and is planned for +in GitLab 14.7, and is planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7351) in GitLab 14.10. Users should instead use the [Web Editor](../repository/web_editor.md) or [Web IDE](../web_ide/index.md). [Removal instructions](#remove-the-static-site-editor) for existing projects are included on this page. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index ff8a076465d..8f9486633d5 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -222,7 +222,7 @@ To edit Markdown files in the Web IDE: 1. Go to your repository, and navigate to the Markdown page you want to edit. 1. Select **Open in Web IDE**, and GitLab loads the page in a tab in the editor. -1. Make your changes to the file. GitLab supports [GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown). +1. Make your changes to the file. GitLab supports [GitLab Flavored Markdown (GLFM)](../../markdown.md). 1. When your changes are complete, select **Commit** in the left sidebar. 1. Add a commit message, select the branch you want to commit to, and select **Commit**. diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 33f6c32d334..7d155ea9b06 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -327,13 +327,7 @@ to disable the wiki but toggle it on (in blue). > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5643) in GitLab 14.0. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345398) switching between editing experiences in GitLab 14.7 [with a flag](../../../administration/feature_flags.md) named `wiki_switch_between_content_editor_raw_markdown`. Enabled by default. - -FLAG: -On self-managed GitLab, by default this feature is available. -To hide the feature, ask an administrator to -[disable the feature flag](../../../administration/feature_flags.md) named -`wiki_switch_between_content_editor_raw_markdown`. -On GitLab.com, this feature is available. +> - Switching between editing experiences generally available in GitLab 14.10. [Feature flag `wiki_switch_between_content_editor_raw_markdown`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83760) removed. GitLab version 14.0 introduces a WYSIWYG editing experience for GitLab Flavored Markdown in Wikis through the [Content Editor](../../../development/fe_guide/content_editor.md). diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 19e77d18aca..03530b59e9b 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -76,13 +76,13 @@ To create a project in GitLab: - [custom template](#create-a-project-from-a-custom-template). - [HIPAA audit protocol template](#create-a-project-from-the-hipaa-audit-protocol-template). - [Import a project](../../user/project/import/index.md) - from a different repository. Contact your GitLab administrator if this option is not available. + from a different repository. Contact your GitLab administrator if this option is not available. - [Connect an external repository to GitLab CI/CD](../../ci/ci_cd_for_external_repos/index.md). -- For a list of words that you cannot use as project names, see -[reserved project and group names](../../user/reserved_names.md). -- For a list of characters that you cannot use in project and group names, see -[limitations on project and group names](../../user/reserved_names.md#limitations-on-project-and-group-names). +- For a list of words that you cannot use as project names, see + [reserved project and group names](../../user/reserved_names.md). +- For a list of characters that you cannot use in project and group names, see + [limitations on project and group names](../../user/reserved_names.md#limitations-on-project-and-group-names). ## Create a blank project @@ -99,12 +99,12 @@ To create a blank project: - In the **Project description (optional)** field, enter the description of your project's dashboard. - In the **Project target (optional)** field, select your project's deployment target. This information helps GitLab better understand its users and their deployment requirements. - - To modify the project's [viewing and access rights](../../public_access/public_access.md) for - users, change the **Visibility Level**. + - To modify the project's [viewing and access rights](../public_access.md) for + users, change the **Visibility Level**. - To create README file so that the Git repository is initialized, has a default branch, and can be cloned, select **Initialize repository with a README**. - To analyze the source code in the project for known security vulnerabilities, - select **Enable Static Application Security Testing (SAST)**. + select **Enable Static Application Security Testing (SAST)**. 1. Select **Create project**. ## Create a project from a built-in template @@ -132,13 +132,13 @@ To create a project from a built-in template: slug as the URL path to the project. To change the slug, first enter the project name, then change the slug. - In the **Project description (optional)** field, enter the description of your project's dashboard. - - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, - change the **Visibility Level**. + - To modify the project's [viewing and access rights](../public_access.md) for users, + change the **Visibility Level**. 1. Select **Create project**. ## Create a project from a custom template **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in GitLab 11.2. Custom project templates are available at: @@ -158,8 +158,8 @@ Custom project templates are available at: slug as the URL path to the project. To change the slug, first enter the project name, then change the slug. - The description of your project's dashboard in the **Project description (optional)** field. - - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, - change the **Visibility Level**. + - To modify the project's [viewing and access rights](../public_access.md) for users, + change the **Visibility Level**. 1. Select **Create project**. ## Create a project from the HIPAA Audit Protocol template **(ULTIMATE)** @@ -184,8 +184,8 @@ To create a project from the HIPAA Audit Protocol template: slug as the URL path to the project. To change the slug, first enter the project name, then change the slug. - In the **Project description (optional)** field, enter the description of your project's dashboard. - - To modify the project's [viewing and access rights](../../public_access/public_access.md) for users, - change the **Visibility Level**. + - To modify the project's [viewing and access rights](../public_access.md) for users, + change the **Visibility Level**. 1. Select **Create project**. ## Create a new project with Git push @@ -206,8 +206,8 @@ used or renamed project, use the [UI](#create-a-project) or the [Projects API](. Prerequisites: -- To push with SSH, you must have [an SSH key](../../ssh/index.md) that is -[added to your GitLab account](../../ssh/index.md#add-an-ssh-key-to-your-gitlab-account). +- To push with SSH, you must have [an SSH key](../ssh.md) that is + [added to your GitLab account](../ssh.md#add-an-ssh-key-to-your-gitlab-account). - You must have permission to add new projects to a namespace. To check if you have permission: 1. On the top bar, select **Menu > Projects**. @@ -250,7 +250,7 @@ remote: The private project namespace/myproject was created. To view your new project, go to `https://gitlab.example.com/namespace/myproject`. Your project's visibility is set to **Private** by default. To change project visibility, adjust your -[project's settings](../../public_access/public_access.md#change-project-visibility). +[project's settings](../public_access.md#change-project-visibility). ## Star a project @@ -299,7 +299,7 @@ To delete a project: > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/351556) in GitLab 14.9. [Feature flag `project_owners_list_project_pending_deletion`](https://gitlab.com/gitlab-org/gitlab/-/issues/351556) removed. When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-deletion), -projects within that group are not deleted immediately, but only after a delay. +projects within that group are not deleted immediately, but only after a delay. To view a list of all projects that are pending deletion: @@ -409,9 +409,9 @@ To disable fetching: 1. Disable checksum queries in `GONOSUMDB`. - If the module name or its prefix is in `GOPRIVATE` or `GONOPROXY`, Go does not query module -proxies. + proxies. - If the module name or its prefix is in `GONOPRIVATE` or `GONOSUMDB`, Go does not query -Checksum databases. + Checksum databases. ### Fetch Go modules from Geo secondary sites diff --git a/doc/user/public_access.md b/doc/user/public_access.md new file mode 100644 index 00000000000..cca753a2830 --- /dev/null +++ b/doc/user/public_access.md @@ -0,0 +1,98 @@ +--- +stage: Manage +group: Workspace +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Project and group visibility **(FREE)** + +GitLab allows users with the Owner role to set a project's or group's visibility as: + +- **Public** +- **Internal** +- **Private** + +These visibility levels affect who can see the project in the public access directory (`/public` +for your GitLab instance). For example, <https://gitlab.com/public>. +You can control the visibility of individual features with +[project feature settings](permissions.md#project-features). + +## Public projects and groups + +Public projects can be cloned **without any** authentication over HTTPS. + +They are listed in the public access directory (`/public`) for all users. + +**Any signed-in user** has the Guest role on the repository. + +NOTE: +By default, `/public` is visible to unauthenticated users. However, if the +[**Public** visibility level](admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) +is restricted, `/public` is visible only to signed-in users. + +## Internal projects and groups + +Internal projects can be cloned by any signed-in user except +[external users](permissions.md#external-users). + +They are also listed in the public access directory (`/public`), but only for signed-in users. + +Any signed-in users except [external users](permissions.md#external-users) have the +Guest role on the repository. + +NOTE: +From July 2019, the `Internal` visibility setting is disabled for new projects, groups, +and snippets on GitLab.com. Existing projects, groups, and snippets using the `Internal` +visibility setting keep this setting. You can read more about the change in the +[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12388). + +## Private projects and groups + +Private projects can only be cloned and viewed by project members (except for guests). + +They appear in the public access directory (`/public`) for project members only. + +## Change project visibility + +Prerequisite: + +- You must have the Owner role for a project. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility, project features, permissions**. +1. Change **Project visibility** to either **Private**, **Internal**, or **Public**. +1. Select **Save changes**. + +## Change group visibility + +Prerequisite: + +- You must have the Owner role for a group. + +1. On the top bar, select **Menu > Groups** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Naming, visibility**. +1. Under **Visibility level** select either **Private**, **Internal**, or **Public**. +1. Select **Save changes**. + +## Restrict use of public or internal projects **(FREE SELF)** + +You can restrict the use of visibility levels for users when they create a project or a snippet. +This is useful to prevent users from publicly exposing their repositories by accident. The +restricted visibility settings do not apply to administrators. + +For details, see [Restricted visibility levels](admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 33ecf252e43..45c5f53e33c 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -19,7 +19,7 @@ under the `TOP_LEVEL_ROUTES`, `PROJECT_WILDCARD_ROUTES` and `GROUP_ROUTES` lists ## Limitations on project and group names -- Special characters are not permitted at the start or end of project or group names. They are permitted in any other location of the name. +- Special characters are not permitted at the start or end of project or group names. They are permitted in any other location of the name. - Project or group names cannot end in `.git` or `.atom`. - Project or group names can only contain letters, digits, emojis, "_", ".", "+", dashes, or spaces. - Paths can only contain letters, digits, "_", "-", and "." diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index cb272b3feed..5435a9d027c 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -26,8 +26,8 @@ when searching in: - Comments - Code - Commits -- Wiki (except [group wikis](../project/wiki/group.md)) - Users +- Wiki (except [group wikis](../project/wiki/group.md)) The Advanced Search can be useful in various scenarios: @@ -46,72 +46,9 @@ The Advanced Search can be useful in various scenarios: may be connected to each other, so your developers need to instantly search throughout the GitLab instance and find the code they search for. -## Use the Advanced Search syntax - -Elasticsearch has data for the default branch only. That means that if you go -to the repository tree and switch the branch from the default to something else, -then the **Code** tab in the search result page is served by the basic -search even if Elasticsearch is enabled. - -The Advanced Search syntax supports fuzzy or exact search queries with prefixes, -boolean operators, and much more. Use the search as before and GitLab shows -you matching code from each project you have access to. - - - -Full details can be found in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax), but -here's a quick guide: - -- Searches look for all the words in a query, in any order - for example: searching - issues for [`display bug`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=display+bug&group_id=9970&project_id=278964) and [`bug display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+Display&group_id=9970&project_id=278964) return the same results. -- To find the exact phrase (stemming still applies), use double quotes: [`"display bug"`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%22display+bug%22&group_id=9970&project_id=278964) -- To find bugs not mentioning display, use `-`: [`bug -display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+-display&group_id=9970&project_id=278964) -- To find a bug in display or banner, use `|`: [`bug display | banner`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+display+%7C+banner&group_id=9970&project_id=278964) -- To group terms together, use parentheses: [`bug | (display +banner)`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) -- To match a partial word, use `*`. In this example, I want to find bugs with any 500 errors. : [`bug error 50*`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+error+50*&group_id=9970&project_id=278964) -- To use one of symbols above literally, escape the symbol with a preceding `\`: [`argument \-last`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=argument+%5C-last&group_id=9970&project_id=278964) - -## Syntax search filters - -Advanced Search also supports the use of filters. The available filters are: - -- `filename`: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. -- `path`: Filters by path. You can use the glob (`*`) operator for fuzzy matching. -- `extension`: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. -- `blob`: Filters by Git `object ID`. Exact match only. - -To use them, add them to your keyword in the format `<filter_name>:<value>` without -any spaces between the colon (`:`) and the value. When no keyword is provided, an asterisk (`*`) is used as the keyword. - -Examples: - -- Finding a file with any content named `search_results.rb`: [`* filename:search_results.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=*+filename%3Asearch_results.rb&group_id=9970&project_id=278964) -- The leading asterisk (`*`) can be ignored in the case above: [`filename:search_results.rb`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=filename%3Asearch_results.rb) -- Finding a file named `found_blob_spec.rb` with the text `CHANGELOG` inside of it: [`CHANGELOG filename:found_blob_spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=CHANGELOG+filename%3Afound_blob_spec.rb&group_id=9970&project_id=278964) -- Finding the text `EpicLinks` inside files with the `.rb` extension: [`EpicLinks extension:rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=EpicLinks+extension%3Arb&group_id=9970&project_id=278964) -- Finding any file with the `.yaml` extension: [`extension:yaml`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=extension%3Ayaml&group_id=9970&project_id=278964) -- Finding the text `Sidekiq` in a file, when that file is in a path that includes `elastic`: [`Sidekiq path:elastic`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=Sidekiq+path%3Aelastic&group_id=9970&project_id=278964) -- Finding any file in a path that includes `elasticsearch`: [`path:elasticsearch`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=path%3Aelasticsearch&group_id=9970&project_id=278964) -- Finding the files represented by the Git object ID `998707b421c89bd9a3063333f9f728ef3e43d101`: [`* blob:998707b421c89bd9a3063333f9f728ef3e43d101`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=*+blob%3A998707b421c89bd9a3063333f9f728ef3e43d101&group_id=9970) -- Syntax filters can be combined for complex filtering. Finding any file starting with `search` containing `eventHub` and with the `.js` extension: [`eventHub filename:search* extension:js`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=eventHub+filename%3Asearch*+extension%3Ajs&group_id=9970&project_id=278964) - -### Excluding filters - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31684) in GitLab 13.3. - -Filters can be inverted to **filter out** results from the result set, by prefixing the filter name with a `-` (hyphen) character, such as: - -- `-filename` -- `-path` -- `-extension` -- `-blob` - -Examples: +## Advanced Search syntax -- Finding `rails` in all files but `Gemfile.lock`: [`rails -filename:Gemfile.lock`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=rails+-filename%3AGemfile.lock&group_id=9970&project_id=278964) -- Finding `success` in all files excluding `.po|pot` files: [`success -filename:*.po*`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=success+-filename%3A*.po*&group_id=9970&project_id=278964) -- Finding `import` excluding minified JavaScript (`.min.js`) files: [`import -extension:min.js`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=import+-extension%3Amin.js&group_id=9970&project_id=278964) -- Finding `docs` for all files outside the `docs/` folder: [`docs -path:docs/`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=docs+-path%3Adocs%2F&group_id=9970&project_id=278964) +See the documentation on [Advanced Search syntax](global_search/advanced_search_syntax.md). ## Search by issue or merge request ID @@ -139,6 +76,7 @@ its performance: | Commits | `global_search_commits_tab` | When enabled, the global search includes commits as part of the search. | | Issues | `global_search_issues_tab` | When enabled, the global search includes issues as part of the search. | | Merge Requests | `global_search_merge_requests_tab` | When enabled, the global search includes merge requests as part of the search. | +| Users | `global_search_users_tab` | When enabled, the global search includes users as part of the search. | | Wiki | `global_search_wiki_tab` | When enabled, the global search includes wiki as part of the search. [Group wikis](../project/wiki/group.md) are not included. | ## Global Search validation diff --git a/doc/user/search/global_search/advanced_search_syntax.md b/doc/user/search/global_search/advanced_search_syntax.md new file mode 100644 index 00000000000..962aa00eea8 --- /dev/null +++ b/doc/user/search/global_search/advanced_search_syntax.md @@ -0,0 +1,49 @@ +--- +stage: Enablement +group: Global Search +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference +--- + +# Advanced Search syntax **(PREMIUM)** + +With [Advanced Search](../advanced_search.md), you can perform a thorough +search through your entire GitLab instance. + +The Advanced Search syntax supports fuzzy or exact search queries with prefixes, +boolean operators, and much more. Advanced Search uses +[Elasticsearch's syntax](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#simple-query-string-syntax). + +WARNING: +Advanced Search searches projects' default branches only. + +See query examples on the tables below and their respective expected output. +The examples link to a search on GitLab.com to help you visualize the output. + +## General search + +| Query example | Expected output | +|---|---| +[`“display bug”`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%22display+bug%22&group_id=9970&project_id=278964) | Returns the **exact phrase** _display bug_ (stemming still applies). | +[`bug -display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+-display&group_id=9970&project_id=278964) | Results include _bug_, and **exclude** _display_. | +[<code>bug | display</code>](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+banner&group_id=9970&project_id=278964) | Results include _bug_ **or** _display_. | +[<code>bug | (display +banner)</code>](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) | Results include _bug_ **or** _display_ **and** _banner_. | +| [`bug error 50*`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+error+50*&group_id=9970&project_id=278964) | `*` finds **partial matches**. Results include _bug_, _error_, and the partial _50_ (looking for any 500 errors, for example). | +| [`bug \-display`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=argument+%5C-last&group_id=9970&project_id=278964) | `\` **scapes symbols**. Results include _bug_ **and** _-display_. | + +## Code Search + +| Query example | Expected output | Notes | +|---|---|---| +| [`filename:*spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=filename%3A*spec.rb&group_id=9970&project_id=278964) | Returns the specified filename. | Use `*` for fuzzy matching. | +| [`path:spec/controllers/`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=path%3Aspec%2Fcontrollers%2F&snippets=) | Returns the specified path location of the repository. | Use `*` for fuzzy matching. | +| [`extension:js`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=extension%3Ajs&snippets=) | Returns the specified file extension. | **Do not** include a leading dot. This only works with exact matches for the extension. | +| [`blob:998707b421c89b*`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=blob%3A998707b421c89b*&group_id=9970) | Returns the specified Git object ID. | This only works with exact matches. | + +## Excluding filters + +Filters can also be inverted to filter out results from the result set by prefixing the filter name with a `-` (hyphen) character. + +| Query example | Expected output | +|---|---| +| [`rails -filename:gemfile.lock`](https://gitlab.com/search?group_id=9970&project_id=278964&repository_ref=&scope=blobs&search=rails+-filename%3Agemfile.lock&snippets=) | Results include _`rails`_ in all files except the _`gemfile.lock`_ file. | diff --git a/doc/user/search/img/advanced_search_v13.10.png b/doc/user/search/img/advanced_search_v13.10.png Binary files differdeleted file mode 100644 index 39cd54fea75..00000000000 --- a/doc/user/search/img/advanced_search_v13.10.png +++ /dev/null diff --git a/doc/user/search/img/code_search_git_blame_v14_9.png b/doc/user/search/img/code_search_git_blame_v14_9.png Binary files differindex 33d4e77e3f5..eb8d14de4a4 100644 --- a/doc/user/search/img/code_search_git_blame_v14_9.png +++ b/doc/user/search/img/code_search_git_blame_v14_9.png diff --git a/doc/user/search/img/sort_projects.png b/doc/user/search/img/sort_projects.png Binary files differdeleted file mode 100644 index 9bf2770b299..00000000000 --- a/doc/user/search/img/sort_projects.png +++ /dev/null diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 44327af380e..de5f469498e 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -14,8 +14,12 @@ The numbers indicate how many issues, merge requests, and to-do items are assign  -- **{issues}** **Issues**: The open issues assigned to you. -- **{merge-request-open}** **Merge requests**: The [merge requests](../project/merge_requests/index.md) assigned to you. +- **{issues}** **Issues**: Issues assigned to you. +- **{merge-request-open}** **Merge requests**: Open [merge requests](../project/merge_requests/index.md). + Select the icon to show a dropdown list of merge request filters: + - [Attention requests](../project/merge_requests/index.md#request-attention-to-a-merge-request) (**{attention-solid}**) for you. + - [Review requests](../project/merge_requests/reviews/index.md) for you. + - Merge requests assigned to you. - **{todo-done}** **To-do items**: The [to-do items](../todos.md) assigned to you. You can search through **Open**, **Closed**, or **All** issues. @@ -37,6 +41,8 @@ in the search field in the upper right corner: > - Filtering by iterations was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6. > - Filtering by iterations was moved from GitLab Ultimate to GitLab Premium in 13.9. > - Filtering by type was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322755) in GitLab 13.10 [with a flag](../../administration/feature_flags.md) named `vue_issues_list`. Disabled by default. +> - Filtering by type was [enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/322755) in GitLab 14.10. +> - Filtering by attention request was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343528) in GitLab 14.10 [with a flag](../../administration/feature_flags.md) named `mr_attention_requests`. Disabled by default. Follow these steps to filter the **Issues** and **Merge requests** list pages in projects and groups: @@ -44,6 +50,7 @@ groups: 1. Select **Search or filter results...**. 1. In the dropdown list that appears, select the attribute you wish to filter by: - Assignee + - [Attention requests](../project/merge_requests/index.md#request-attention-to-a-merge-request) - Author - Confidential - [Epic and child Epic](../group/epics/index.md) (available only for the group the Epic was created, not for [higher group levels](https://gitlab.com/gitlab-org/gitlab/-/issues/233729)). @@ -53,12 +60,6 @@ groups: - My-reaction - Release - Type - - FLAG: - On self-managed GitLab, by default filtering by type is not available. - To make it available per group, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `vue_issues_list`. - On GitLab.com, this feature is not available. - - Weight - Search for this text 1. Select or type the operator to use for filtering the attribute. The following operators are @@ -146,6 +147,10 @@ you can type (or select from the dropdown list) the following: - Deployed-before - Deployed-after +NOTE: +Projects using a [fast-forward merge method](../project/merge_requests/fast_forward_merge.md) +do not return results, as this method does not create a merge commit. + When filtering by an environment, a dropdown list presents all environments that you can choose from: @@ -207,10 +212,14 @@ You can also look for the projects you [starred](../project/working_with_project You can **Explore** all public and internal projects available in GitLab.com, from which you can filter by visibility, through **Trending**, best rated with **Most stars**, or **All** of them. -You can also sort them by **Name**, **Last created**, **Oldest created**, **Last updated**, -**Oldest updated**, **Owner**, and choose to hide or show **archived projects**: +You can also sort them by: + +- Name +- Created date +- Updated date +- Owner - +You can also choose to hide or show archived projects. ## Groups @@ -221,7 +230,7 @@ On the field **Filter by name**, type the group name you want to find, and GitLa filters them for you as you type. You can also **Explore** all public and internal groups available in GitLab.com, -and sort them by **Last created**, **Oldest created**, **Last updated**, or **Oldest updated**. +and sort them by **Name**, **Last created**, **Oldest created**, or **Updated date**. ## Issue boards @@ -246,7 +255,7 @@ In the search bar, you can view autocomplete suggestions for: - Recently viewed issues (try and type some word from the title of a recently viewed issue) - Recently viewed merge requests (try and type some word from the title of a recently viewed merge request) - Recently viewed epics (try and type some word from the title of a recently viewed epic) -- [GitLab Flavored Markdown](../markdown.md#gitlab-specific-references) (GFM) for issues in a project (try and type a GFM reference for an issue) +- [GitLab Flavored Markdown](../markdown.md#gitlab-specific-references) (GLFM) for issues in a project (try and type a GLFM reference for an issue) ## Basic search diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index e807f251da1..e5285d63cf4 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -101,6 +101,12 @@ These shortcuts are available when viewing issues and [merge requests](project/m | <kbd>b</kbd> | Copy source branch name (merge requests only). | | <kbd>.</kbd> | Open the [Web IDE](project/web_ide/index.md). | +Merge requests additionally support the following shortcuts: + +| macOS shortcut | Windows shortcut | Description | +|---------------------------------|---------------------|-------------| +| <kbd>Command</kbd> + <kbd>p</kbd> | <kbd>Control</kbd> + <kbd>p</kbd> | Search for, and then jump to a file for review. | + ### Project files These shortcuts are available when browsing the files in a project (go to diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 1750e2c8d10..cab18b221c1 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -22,7 +22,7 @@ using the [GitLab Workflow VS Code extension](project/repository/vscode.md). GitLab provides two types of snippets: - **Personal snippets**: Created independent of any project. - You can set a [visibility level](../public_access/public_access.md) + You can set a [visibility level](public_access.md) for your snippet: public, internal, or private. - **Project snippets**: Always related to a specific project. Project snippets can be visible publicly or to only group members. diff --git a/doc/user/ssh.md b/doc/user/ssh.md new file mode 100644 index 00000000000..54d4722ee2b --- /dev/null +++ b/doc/user/ssh.md @@ -0,0 +1,497 @@ +--- +stage: Manage +group: Authentication and Authorization +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: howto, reference +--- + +# Use SSH keys to communicate with GitLab **(FREE)** + +Git is a distributed version control system, which means you can work locally, +then share or *push* your changes to a server. In this case, the server you push to is GitLab. + +GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +## Prerequisites + +To use SSH to communicate with GitLab, you need: + +- The OpenSSH client, which comes pre-installed on GNU/Linux, macOS, and Windows 10. +- SSH version 6.5 or later. Earlier versions used an MD5 signature, which is not secure. + +To view the version of SSH installed on your system, run `ssh -V`. + +## Supported SSH key types + +To communicate with GitLab, you can use the following SSH key types: + +- [ED25519](#ed25519-ssh-keys) +- [ED25519_SK](#ed25519_sk-ssh-keys) (Available in GitLab 14.8 and later.) +- [ECDSA_SK](#ecdsa_sk-ssh-keys) (Available in GitLab 14.8 and later.) +- [RSA](#rsa-ssh-keys) +- DSA ([Deprecated](https://about.gitlab.com/releases/2018/06/22/gitlab-11-0-released/#support-for-dsa-ssh-keys) in GitLab 11.0.) +- ECDSA (As noted in [Practical Cryptography With Go](https://leanpub.com/gocrypto/read#leanpub-auto-ecdsa), the security issues related to DSA also apply to ECDSA.) + +Administrators can [restrict which keys are permitted and their minimum lengths](../security/ssh_keys_restrictions.md). + +### ED25519 SSH keys + +The book [Practical Cryptography With Go](https://leanpub.com/gocrypto/read#leanpub-auto-chapter-5-digital-signatures) +suggests that [ED25519](https://ed25519.cr.yp.to/) keys are more secure and performant than RSA keys. + +OpenSSH 6.5 introduced ED25519 SSH keys in 2014 and they should be available on most +operating systems. + +### ED25519_SK SSH keys + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. + +To use ED25519_SK SSH keys on GitLab, your local client and GitLab server +must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. + +### ECDSA_SK SSH keys + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934) in GitLab 14.8. + +To use ECDSA_SK SSH keys on GitLab, your local client and GitLab server +must have [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) or later installed. + +### RSA SSH keys + +Available documentation suggests that ED25519 is more secure than RSA. + +If you use an RSA key, the US National Institute of Science and Technology in +[Publication 800-57 Part 3 (PDF)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-57Pt3r1.pdf) +recommends a key size of at least 2048 bits. The default key size depends on your version of `ssh-keygen`. +Review the `man` page for your installed `ssh-keygen` command for details. + +## See if you have an existing SSH key pair + +Before you create a key pair, see if a key pair already exists. + +1. On Windows, Linux, or macOS, go to your home directory. +1. Go to the `.ssh/` subdirectory. If the `.ssh/` subdirectory doesn't exist, + you are either not in the home directory, or you haven't used `ssh` before. + In the latter case, you need to [generate an SSH key pair](#generate-an-ssh-key-pair). +1. See if a file with one of the following formats exists: + + | Algorithm | Public key | Private key | + | --------- | ---------- | ----------- | + | ED25519 (preferred) | `id_ed25519.pub` | `id_ed25519` | + | ED25519_SK | `id_ed25519_sk.pub` | `id_ed25519_sk` | + | ECDSA_SK | `id_ecdsa_sk.pub` | `id_ecdsa_sk` | + | RSA (at least 2048-bit key size) | `id_rsa.pub` | `id_rsa` | + | DSA (deprecated) | `id_dsa.pub` | `id_dsa` | + | ECDSA | `id_ecdsa.pub` | `id_ecdsa` | + +## Generate an SSH key pair + +If you do not have an existing SSH key pair, generate a new one. + +1. Open a terminal. +1. Type `ssh-keygen -t` followed by the key type and an optional comment. + This comment is included in the `.pub` file that's created. + You may want to use an email address for the comment. + + For example, for ED25519: + + ```shell + ssh-keygen -t ed25519 -C "<comment>" + ``` + + For 2048-bit RSA: + + ```shell + ssh-keygen -t rsa -b 2048 -C "<comment>" + ``` + +1. Press Enter. Output similar to the following is displayed: + + ```plaintext + Generating public/private ed25519 key pair. + Enter file in which to save the key (/home/user/.ssh/id_ed25519): + ``` + +1. Accept the suggested filename and directory, unless you are generating a [deploy key](project/deploy_keys/index.md) + or want to save in a specific directory where you store other keys. + + You can also dedicate the SSH key pair to a [specific host](#configure-ssh-to-point-to-a-different-directory). + +1. Specify a [passphrase](https://www.ssh.com/academy/ssh/passphrase): + + ```plaintext + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + ``` + +1. A confirmation is displayed, including information about where your files are stored. + +A public and private key are generated. +[Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account) and keep +the private key secure. + +### Configure SSH to point to a different directory + +If you did not save your SSH key pair in the default directory, +configure your SSH client to point to the directory where the private key is stored. + +1. Open a terminal and run this command: + + ```shell + eval $(ssh-agent -s) + ssh-add <directory to private SSH key> + ``` + +1. Save these settings in the `~/.ssh/config` file. For example: + + ```conf + # GitLab.com + Host gitlab.com + PreferredAuthentications publickey + IdentityFile ~/.ssh/gitlab_com_rsa + + # Private GitLab instance + Host gitlab.company.com + PreferredAuthentications publickey + IdentityFile ~/.ssh/example_com_rsa + ``` + + For more information on these settings, see the [`man ssh_config`](https://man.openbsd.org/ssh_config) page in the SSH configuration manual. + +Public SSH keys must be unique to GitLab because they bind to your account. +Your SSH key is the only identifier you have when you push code with SSH. +It must uniquely map to a single user. + +### Update your SSH key passphrase + +You can update the passphrase for your SSH key. + +1. Open a terminal and run this command: + + ```shell + ssh-keygen -p -f /path/to/ssh_key + ``` + +1. At the prompts, type the passphrase and press Enter. + +### Upgrade your RSA key pair to a more secure format + +If your version of OpenSSH is between 6.5 and 7.8, +you can save your private RSA SSH keys in a more secure +OpenSSH format. + +1. Open a terminal and run this command: + + ```shell + ssh-keygen -o -f ~/.ssh/id_rsa + ``` + + Alternatively, you can generate a new RSA key with the more secure encryption format with + the following command: + + ```shell + ssh-keygen -o -t rsa -b 4096 -C "<comment>" + ``` + +## Generate an SSH key pair for a FIDO/U2F hardware security key + +To generate ED25519_SK or ECDSA_SK SSH keys, you must use OpenSSH 8.2 or later. + +1. Insert a hardware security key into your computer. +1. Open a terminal. +1. Type `ssh-keygen -t` followed by the key type and an optional comment. + This comment is included in the `.pub` file that's created. + You may want to use an email address for the comment. + + For example, for ED25519_SK: + + ```shell + ssh-keygen -t ed25519-sk -C "<comment>" + ``` + + For ECDSA_SK: + + ```shell + ssh-keygen -t ecdsa-sk -C "<comment>" + ``` + + If your security key supports FIDO2 resident keys, you can enable this when + creating your SSH key: + + ```shell + ssh-keygen -t ed25519-sk -O resident -C "<comment>" + ``` + + `-O resident` indicates that the key should be stored on the FIDO authenticator itself. + Resident key is easier to import to a new computer because it can be loaded directly + from the security key by [`ssh-add -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-add.1#K) + or [`ssh-keygen -K`](https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/ssh-keygen#K). + +1. Select Enter. Output similar to the following is displayed: + + ```plaintext + Generating public/private ed25519-sk key pair. + You may need to touch your authenticator to authorize key generation. + ``` + +1. Touch the button on the hardware security key. + +1. Accept the suggested filename and directory: + + ```plaintext + Enter file in which to save the key (/home/user/.ssh/id_ed25519_sk): + ``` + +1. Specify a [passphrase](https://www.ssh.com/academy/ssh/passphrase): + + ```plaintext + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + ``` + +1. A confirmation is displayed, including information about where your files are stored. + +A public and private key are generated. +[Add the public SSH key to your GitLab account](#add-an-ssh-key-to-your-gitlab-account). + +## Add an SSH key to your GitLab account + +To use SSH with GitLab, copy your public key to your GitLab account. + +1. Copy the contents of your public key file. You can do this manually or use a script. + For example, to copy an ED25519 key to the clipboard: + + **macOS:** + + ```shell + tr -d '\n' < ~/.ssh/id_ed25519.pub | pbcopy + ``` + + **Linux** (requires the `xclip` package): + + ```shell + xclip -sel clip < ~/.ssh/id_ed25519.pub + ``` + + **Git Bash on Windows:** + + ```shell + cat ~/.ssh/id_ed25519.pub | clip + ``` + + Replace `id_ed25519.pub` with your filename. For example, use `id_rsa.pub` for RSA. + +1. Sign in to GitLab. +1. On the top bar, in the top right corner, select your avatar. +1. Select **Preferences**. +1. On the left sidebar, select **SSH Keys**. +1. In the **Key** box, paste the contents of your public key. + If you manually copied the key, make sure you copy the entire key, + which starts with `ssh-rsa`, `ssh-dss`, `ecdsa-sha2-nistp256`, `ecdsa-sha2-nistp384`, `ecdsa-sha2-nistp521`, + `ssh-ed25519`, `sk-ecdsa-sha2-nistp256@openssh.com`, or `sk-ssh-ed25519@openssh.com`, and may end with a comment. +1. In the **Title** box, type a description, like `Work Laptop` or + `Home Workstation`. +1. Optional. In the **Expires at** box, select an expiration date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36243) in GitLab 12.9.) + In: + - GitLab 13.12 and earlier, the expiration date is informational only. It doesn't prevent + you from using the key. Administrators can view expiration dates and use them for + guidance when [deleting keys](admin_area/credentials_inventory.md#delete-a-users-ssh-key). + - GitLab 14.0 and later, the expiration date is enforced. Administrators can + [allow expired keys to be used](admin_area/settings/account_and_limit_settings.md#allow-expired-ssh-keys-to-be-used-deprecated). + - GitLab checks all SSH keys at 02:00 AM UTC every day. It emails an expiration notice for all SSH keys that expire on the current date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) + - GitLab checks all SSH keys at 01:00 AM UTC every day. It emails an expiration notice for all SSH keys that are scheduled to expire seven days from now. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.11.) +1. Select **Add key**. + +## Verify that you can connect + +Verify that your SSH key was added correctly. + +1. For GitLab.com, to ensure you're connecting to the correct server, confirm the + [SSH host keys fingerprints](gitlab_com/index.md#ssh-host-keys-fingerprints). +1. Open a terminal and run this command, replacing `gitlab.example.com` with your GitLab instance URL: + + ```shell + ssh -T git@gitlab.example.com + ``` + +1. If this is the first time you connect, you should verify the + authenticity of the GitLab host. If you see a message like: + + ```plaintext + The authenticity of host 'gitlab.example.com (35.231.145.151)' can't be established. + ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw. + Are you sure you want to continue connecting (yes/no)? yes + Warning: Permanently added 'gitlab.example.com' (ECDSA) to the list of known hosts. + ``` + + Type `yes` and press Enter. + +1. Run the `ssh -T git@gitlab.example.com` command again. You should receive a _Welcome to GitLab, `@username`!_ message. + +If the welcome message doesn't appear, you can troubleshoot by running `ssh` +in verbose mode: + +```shell +ssh -Tvvv git@gitlab.example.com +``` + +## Use different keys for different repositories + +You can use a different key for each repository. + +Open a terminal and run this command: + +```shell +git config core.sshCommand "ssh -o IdentitiesOnly=yes -i ~/.ssh/private-key-filename-for-this-repository -F /dev/null" +``` + +This command does not use the SSH Agent and requires Git 2.10 or later. For more information +on `ssh` command options, see the `man` pages for both `ssh` and `ssh_config`. + +## Use different accounts on a single GitLab instance + +You can use multiple accounts to connect to a single instance of GitLab. +You can do this by using the command in the [previous topic](#use-different-keys-for-different-repositories). +However, even if you set `IdentitiesOnly` to `yes`, you cannot sign in if an `IdentityFile` exists +outside of a `Host` block. + +Instead, you can assign aliases to hosts in the `~.ssh/config` file. + +- For the `Host`, use an alias like `user_1.gitlab.com` and + `user_2.gitlab.com`. Advanced configurations + are more difficult to maintain, and these strings are easier to + understand when you use tools like `git remote`. +- For the `IdentityFile`, use the path the private key. + +```conf +# User1 Account Identity +Host <user_1.gitlab.com> + Hostname gitlab.com + PreferredAuthentications publickey + IdentityFile ~/.ssh/<example_ssh_key1> + +# User2 Account Identity +Host <user_2.gitlab.com> + Hostname gitlab.com + PreferredAuthentications publickey + IdentityFile ~/.ssh/<example_ssh_key2> +``` + +Now, to clone a repository for `user_1`, use `user_1.gitlab.com` in the `git clone` command: + +```shell +git clone git@<user_1.gitlab.com>:gitlab-org/gitlab.git +``` + +To update a previously-cloned repository that is aliased as `origin`: + +```shell +git remote set-url origin git@<user_1.gitlab.com>:gitlab-org/gitlab.git +``` + +NOTE: +Private and public keys contain sensitive data. Ensure the permissions +on the files make them readable to you but not accessible to others. + +## Configure two-factor authentication (2FA) + +You can set up two-factor authentication (2FA) for +[Git over SSH](../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations). We recommend using +[ED25519_SK](#ed25519_sk-ssh-keys) or [ECDSA_SK](#ecdsa_sk-ssh-keys) SSH keys. + +## Use EGit on Eclipse + +If you are using [EGit](https://www.eclipse.org/egit/), you can [add your SSH key to Eclipse](https://wiki.eclipse.org/EGit/User_Guide#Eclipse_SSH_Configuration). + +## Use SSH on Microsoft Windows + +If you're running Windows 10, you can either use the [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install) +with [WSL 2](https://docs.microsoft.com/en-us/windows/wsl/install#update-to-wsl-2) which +has both `git` and `ssh` preinstalled, or install [Git for Windows](https://gitforwindows.org) to +use SSH through PowerShell. + +The SSH key generated in WSL is not directly available for Git for Windows, and vice versa, +as both have a different home directory: + +- WSL: `/home/<user>` +- Git for Windows: `C:\Users\<user>` + +You can either copy over the `.ssh/` directory to use the same key, or generate a key in each environment. + +Alternative tools include: + +- [Cygwin](https://www.cygwin.com) +- [PuttyGen](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html) + +## Overriding SSH settings on the GitLab server + +GitLab integrates with the system-installed SSH daemon and designates a user +(typically named `git`) through which all access requests are handled. Users +who connect to the GitLab server over SSH are identified by their SSH key instead +of their username. + +SSH *client* operations performed on the GitLab server are executed as this +user. You can modify this SSH configuration. For example, you can specify +a private SSH key for this user to use for authentication requests. However, this practice +is **not supported** and is strongly discouraged as it presents significant +security risks. + +GitLab checks for this condition, and directs you +to this section if your server is configured this way. For example: + +```shell +$ gitlab-rake gitlab:check + +Git user has default SSH configuration? ... no + Try fixing it: + mkdir ~/gitlab-check-backup-1504540051 + sudo mv /var/lib/git/.ssh/id_rsa ~/gitlab-check-backup-1504540051 + sudo mv /var/lib/git/.ssh/id_rsa.pub ~/gitlab-check-backup-1504540051 + For more information see: + doc/user/ssh.md#overriding-ssh-settings-on-the-gitlab-server + Please fix the error above and rerun the checks. +``` + +Remove the custom configuration as soon as you can. These customizations +are **explicitly not supported** and may stop working at any time. + +## Troubleshooting + +### Password prompt with `git clone` + +When you run `git clone`, you may be prompted for a password, like `git@gitlab.example.com's password:`. +This indicates that something is wrong with your SSH setup. + +- Ensure that you generated your SSH key pair correctly and added the public SSH + key to your GitLab profile. +- Try to manually register your private SSH key by using `ssh-agent`. +- Try to debug the connection by running `ssh -Tv git@example.com`. + Replace `example.com` with your GitLab URL. + +### `Could not resolve hostname` error + +You may receive the following error when [verifying that you can connect](#verify-that-you-can-connect): + +```shell +ssh: Could not resolve hostname gitlab.example.com: nodename nor servname provided, or not known +``` + +If you receive this error, restart your terminal and try the command again. + +### `Key enrollment failed: invalid format` error + +You may receive the following error when [generating an SSH key pair for a FIDO/U2F hardware security key](#generate-an-ssh-key-pair-for-a-fidou2f-hardware-security-key): + +```shell +Key enrollment failed: invalid format +``` + +You can troubleshoot this by trying the following: + +- Run the `ssh-keygen` command using `sudo`. +- Verify your IDO/U2F hardware security key supports + the key type provided. +- Verify the version of OpenSSH is 8.2 or greater by + running `ssh -v`. diff --git a/doc/user/todos.md b/doc/user/todos.md index ec316d8ebe4..5cea619c830 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -119,7 +119,7 @@ Actions that dismiss to-do items include: - Closing the issue or merge request - Adding or removing a label - Commenting on the issue -- Resolving a [design discussion thread](project/issues/design_management.md#resolve-design-threads) +- Resolving a [design discussion thread](project/issues/design_management.md#resolve-a-discussion-thread-on-a-design) If someone else closes, merges, or takes action on an issue, merge request, or epic, your to-do item remains pending. |
