diff options
Diffstat (limited to 'doc')
671 files changed, 13636 insertions, 11208 deletions
diff --git a/doc/.vale/gitlab/Acronyms.yml b/doc/.vale/gitlab/Acronyms.yml index 122cac6f8a1..b75f81dd075 100644 --- a/doc/.vale/gitlab/Acronyms.yml +++ b/doc/.vale/gitlab/Acronyms.yml @@ -17,6 +17,7 @@ exceptions: - AJAX - ANSI - API + - APM - ARM - ARN - ASCII @@ -24,6 +25,7 @@ exceptions: - BSD - CAS - CDN + - CIDR - CLI - CNA - CNAME @@ -41,11 +43,13 @@ exceptions: - DNS - DOM - DSA + - DSL - DVCS - ECDSA - ECS - EFS - EKS + - ELB - EOL - EXIF - FAQ @@ -59,6 +63,7 @@ exceptions: - GDK - GDPR - GET + - GID - GIF - GKE - GNU @@ -94,6 +99,7 @@ exceptions: - LESS - LFS - LRU + - LTM - LTS - MIME - MIT @@ -115,6 +121,8 @@ exceptions: - PEM - PEP - PGP + - PID + - PKCS - PHP - PNG - POSIX @@ -124,6 +132,7 @@ exceptions: - RAM - RBAC - RDP + - RDS - REST - RFC - RHEL @@ -135,6 +144,7 @@ exceptions: - RVM - SAAS - SAML + - SAN - SAST - SATA - SCIM @@ -168,7 +178,9 @@ exceptions: - TODO - TOML - TTL + - UID - UDP + - UID - UNIX - URI - URL diff --git a/doc/.vale/gitlab/UnclearAntecedent.yml b/doc/.vale/gitlab/UnclearAntecedent.yml index 863bbd4e109..5f238598d9f 100644 --- a/doc/.vale/gitlab/UnclearAntecedent.yml +++ b/doc/.vale/gitlab/UnclearAntecedent.yml @@ -1,13 +1,13 @@ --- -# Suggestion: gitlab.UnclearAntecedent +# Warning: gitlab.UnclearAntecedent # # Checks for words that need a noun for clarity. # -# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +# For a list of all options, see https://docs.errata.ai/vale/styles extends: existence message: "'%s' is not precise. Try rewriting with a specific subject and verb." link: https://docs.gitlab.com/ee/development/documentation/styleguide/word_list.html#this-these-that-those -level: suggestion +level: warning ignorecase: false tokens: - 'That is' diff --git a/doc/.vale/vale-json.tmpl b/doc/.vale/vale-json.tmpl new file mode 100644 index 00000000000..ed3c3259df3 --- /dev/null +++ b/doc/.vale/vale-json.tmpl @@ -0,0 +1,58 @@ +{{- /* Modify Vale's output https://docs.errata.ai/vale/cli#--output */ -}} + +{{- /* Keep track of our various counts */ -}} + +{{- $e := 0 -}} +{{- $w := 0 -}} +{{- $s := 0 -}} +{{- $f := 0 -}} + +{{- /* Range over the linted files */ -}} + +{{- range .Files}} + +{{- $f = add1 $f -}} +{{- $path := .Path -}} + +{{- /* Range over the file's alerts */ -}} +[ + +{{- range $idx, $a := .Alerts -}} + +{{- $error := "" -}} +{{- if eq .Severity "error" -}} + {{- $error = .Severity -}} + {{- $e = add1 $e -}} +{{- else if eq .Severity "warning" -}} + {{- $error = .Severity -}} + {{- $w = add1 $w -}} +{{- else -}} + {{- $error = .Severity -}} + {{- $s = add1 $s -}} +{{- end}} + +{{- /* Variables setup */ -}} + +{{- $path = $path -}} +{{- $loc := printf "%d" .Line -}} +{{- $check := printf "%s" .Check -}} +{{- $message := printf "%s" .Message -}} +{{- $link := printf "%s" .Link -}} +{{- if $idx -}},{{- end -}} + +{{- /* Output */ -}} + + { + "description": "{{ $message }}", + "fingerprint": "{{ $path }}-{{ $loc }}", + "severity": "{{ $error }}", + "location": { + "path": "{{ $path }}", + "lines": { + "begin": {{ $loc }} + } + } + } +{{end -}} +{{end -}} +] diff --git a/doc/README.md b/doc/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 3ff5fb2635d..572c341f2b2 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -137,8 +137,6 @@ Project event queries are limited to a maximum of 30 days. ### Instance events **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in GitLab 9.3. - Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md deleted file mode 100644 index 5ab8653dc35..00000000000 --- a/doc/administration/auth/README.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -redirect_to: 'index.md' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index a072cc73c43..959c5218554 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -31,7 +31,7 @@ providers: - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** -- [Shibboleth](../../integration/shibboleth.md) +- [Shibboleth](../../integration/saml.md) - [Smartcard](smartcard.md) **(PREMIUM SELF)** - [Twitter](../../integration/twitter.md) @@ -45,7 +45,7 @@ For more information, see the links shown on this page for each external provide | Capability | SaaS | Self-Managed | |-------------------------------------------------|-----------------------------------------|------------------------------------| -| **User Provisioning** | SCIM<br>JIT Provisioning | LDAP Sync | +| **User Provisioning** | SCIM<br>Just-In-Time (JIT) Provisioning | LDAP Sync | | **User Detail Updating** (not group management) | Not Available | LDAP Sync | | **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | | **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 137f35986ac..e5af8e8256a 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -15,7 +15,7 @@ LDAP service that can be configured with GitLab for authentication and group syn Secure LDAP requires a slightly different configuration than standard LDAP servers. The steps below cover: -- Configuring the Secure LDAP Client in the Google Admin console. +- Configuring the Secure LDAP Client in the Google administrator console. - Required GitLab configuration. ## Configuring Google LDAP client diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 1992b450338..92815f10b92 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# General LDAP setup **(FREE SELF)** +# Integrate LDAP with GitLab **(FREE SELF)** GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) to support user authentication. @@ -39,7 +39,9 @@ the LDAP server, or share email addresses. ### User deletion Users deleted from the LDAP server are immediately blocked from signing in -to GitLab. However, there's an LDAP check cache time of one hour (which is +to GitLab and [no longer consumes a +license](../../../user/admin_area/moderate_users.md). +However, there's an LDAP check cache time of one hour (which is [configurable](#adjust-ldap-user-sync-schedule) for GitLab Premium users). This means users already signed-in or who are using Git over SSH can access GitLab for up to one hour. Manually block the user in the GitLab Admin Area @@ -70,15 +72,15 @@ LDAP email address, and then sign into GitLab by using their LDAP credentials. LDAP service that can be configured with GitLab for authentication and group sync. See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. -## Configuration +## Configure LDAP -To enable LDAP integration you must add your LDAP server settings in -`/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus -GitLab and installations from source respectively. +To configure LDAP integration, add your LDAP server settings in: -There is a Rake task to check LDAP configuration. After configuring LDAP -using the documentation below, see [LDAP check Rake task](../../raketasks/check.md#ldap-check) -for information on the LDAP check Rake task. +- `/etc/gitlab/gitlab.rb` for Omnibus GitLab instances. +- `/home/git/gitlab/config/gitlab.yml` for source install instances. + +After configuring LDAP, to test the configuration, use the +[LDAP check Rake task](../../raketasks/check.md#ldap-check). NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP @@ -90,9 +92,9 @@ with `start_tls` and `ssl` was replaced with `simple_tls`. LDAP users must have a set email address, regardless of whether or not it's used to sign in. -### Example Configurations +### Example Omnibus GitLab configuration -**Omnibus Configuration** +This example shows configuration for Omnibus GitLab instances: ```ruby gitlab_rails['ldap_enabled'] = true @@ -139,7 +141,9 @@ gitlab_rails['ldap_servers'] = { } ``` -**Source Configuration** +### Example source install configuration + +This example shows configuration for source install instances: ```yaml production: @@ -155,6 +159,8 @@ production: ### Basic configuration settings +These configuration settings are available: + | Setting | Description | Required | Examples | |--------------------|-------------|----------|----------| | `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | @@ -183,6 +189,8 @@ Some examples of the `user_filter` field syntax: ### SSL configuration settings +These SSL configuration settings are available: + | Setting | Description | Required | Examples | |---------------|-------------|----------|----------| | `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | @@ -193,69 +201,72 @@ Some examples of the `user_filter` field syntax: ### Attribute configuration settings -LDAP attributes that GitLab uses to create an account for the LDAP user. The specified -attribute can either be the attribute name as a string (for example, `'mail'`), or an -array of attribute names to try in order (for example, `['mail', 'email']`). -The user's LDAP sign-in is the attribute specified as `uid` above. +GitLab uses these LDAP attributes to create an account for the LDAP user. The specified +attribute can be either: + +- The attribute name as a string. For example, `'mail'`. +- An array of attribute names to try in order. For example, `['mail', 'email']`. + +The user's LDAP sign in is the LDAP attribute [specified as `uid`](#basic-configuration-settings). | Setting | Description | Required | Examples | |--------------|-------------|----------|----------| -| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | +| `username` | Used in paths for the user's own projects (for example, `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (for example, `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | | `email` | LDAP attribute for user email. | **{dotted-circle}** No | `['mail', 'email', 'userPrincipalName']` | | `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | **{dotted-circle}** No | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | | `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'givenName'` | | `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'sn'` | -### LDAP Sync configuration settings **(PREMIUM SELF)** +### LDAP sync configuration settings **(PREMIUM SELF)** + +These LDAP sync configuration settings are available: | Setting | Description | Required | Examples | |-------------------|-------------|----------|----------| | `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` | -| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | -| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | +| `admin_group` | The CN of a group containing GitLab administrators. Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | +| `external_groups` | An array of CNs of groups containing users that should be considered external. Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | | `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | ### Set up LDAP user filter -If you want to limit all GitLab access to a subset of the LDAP users on your -LDAP server, the first step should be to narrow the configured `base`. However, -it's sometimes necessary to further filter users. In this case, you can set -up an LDAP user filter. The filter must comply with -[RFC 4515](https://tools.ietf.org/search/rfc4515). +To limit all GitLab access to a subset of the LDAP users on your LDAP server, first narrow the +configured `base`. However, to further filter users if +necessary, you can set up an LDAP user filter. The filter must comply with [RFC 4515](https://tools.ietf.org/search/rfc4515). -**Omnibus configuration** +- Example user filter for Omnibus GitLab instances: -```ruby -gitlab_rails['ldap_servers'] = { -'main' => { - # snip... - 'user_filter' => '(employeeType=developer)' + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'user_filter' => '(employeeType=developer)' + } } -} -``` + ``` -**Source configuration** +- Example user filter for source install instances: -```yaml -production: - ldap: - servers: - main: - # snip... - user_filter: '(employeeType=developer)' -``` + ```yaml + production: + ldap: + servers: + main: + # snip... + user_filter: '(employeeType=developer)' + ``` -If you want to limit access to the nested members of an Active Directory -group, use the following syntax: +To limit access to the nested members of an Active Directory group, use the following syntax: ```plaintext (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` -For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following -[Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. +For more information about `LDAP_MATCHING_RULE_IN_CHAIN` filters, see +[Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax). + Support for nested members in the user filter shouldn't be confused with -[group sync nested groups support](#supported-ldap-group-typesattributes). **(PREMIUM SELF)** +[group sync nested groups](#supported-ldap-group-typesattributes) support. GitLab does not support the custom filter syntax used by OmniAuth LDAP. @@ -451,7 +462,7 @@ If initially your LDAP configuration looked like: ### TLS server authentication -There are two encryption methods, `simple_tls` and `start_tls`. +`simple_tls` and `start_tls` are the two available encryption methods. For either encryption method, if setting `verify_certificates: false`, TLS encryption is established with the LDAP server before any LDAP-protocol data is @@ -463,9 +474,9 @@ exchanged but no validation of the LDAP server's SSL certificate is performed. Not implemented by `Net::LDAP`. -You should disable anonymous LDAP authentication and enable simple or SASL -authentication. The TLS client authentication setting in your LDAP server cannot -be mandatory and clients cannot be authenticated with the TLS protocol. +You should disable anonymous LDAP authentication and enable simple or Simple Authentication +and Security Layer (SASL) authentication. The TLS client authentication setting in your LDAP server +cannot be mandatory and clients cannot be authenticated with the TLS protocol. ## Multiple LDAP servers **(PREMIUM SELF)** @@ -474,7 +485,7 @@ connects to. To add another LDAP server: -1. Duplicate the settings under [the main configuration](#configuration). +1. Duplicate the settings under [the main configuration](#configure-ldap). 1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. @@ -526,7 +537,7 @@ The process executes the following access checks: - Ensure the user is still present in LDAP. - If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This is checked only if + blocked/disabled state). This check is performed only if `active_directory: true` is set in the LDAP configuration. In Active Directory, a user is marked as disabled/blocked if the user @@ -702,7 +713,7 @@ When enabled, the following applies: To enable it, you must: -1. [Enable LDAP](#configuration) +1. [Configure LDAP](#configure-ldap). 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. @@ -716,7 +727,7 @@ The values shown are in cron format. If needed, you can use a WARNING: Do not start the sync process too frequently as this -could lead to multiple syncs running concurrently. This is primarily a concern +could lead to multiple syncs running concurrently. This concern is primarily for installations with a large number of LDAP users. Review the [LDAP group sync benchmark metrics](#benchmarks) to see how your installation compares before proceeding. @@ -850,7 +861,7 @@ LDAP group links each: - Subsequent syncs (checking membership, no writes) took 15 minutes These metrics are meant to provide a baseline and performance may vary based on -any number of factors. This was an extreme benchmark and most instances don't +any number of factors. This benchmark was extreme and most instances don't have near this many users or groups. Disk speed, database performance, network and LDAP server response time affects these metrics. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 1952e8afa97..4757725d0bd 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -55,9 +55,8 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server #### Query LDAP **(PREMIUM SELF)** The following allows you to perform a search in LDAP using the rails console. -Depending on what you're trying to do, it may make more sense to query [a -user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap) directly, or -even [use `ldapsearch`](#ldapsearch) instead. +Depending on what you're trying to do, it may make more sense to query [a user](#query-a-user-in-ldap) +or [a group](#query-a-group-in-ldap) directly, or even [use `ldapsearch`](#ldapsearch) instead. ```ruby adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') @@ -90,7 +89,7 @@ established but GitLab doesn't show you LDAP users in the output, one of the following is most likely true: - The `bind_dn` user doesn't have enough permissions to traverse the user tree. -- The user(s) don't fall under the [configured `base`](index.md#configuration). +- The user(s) don't fall under the [configured `base`](index.md#configure-ldap). - The [configured `user_filter`](index.md#set-up-ldap-user-filter) blocks access to the user(s). In this case, you con confirm which of the above is true using @@ -102,7 +101,7 @@ In this case, you con confirm which of the above is true using A user can have trouble signing in for any number of reasons. To get started, here are some questions to ask yourself: -- Does the user fall under the [configured `base`](index.md#configuration) in +- Does the user fall under the [configured `base`](index.md#configure-ldap) in LDAP? The user must fall under this `base` to sign in. - Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter)? If one is not configured, this question can be ignored. If it is, then the @@ -355,11 +354,10 @@ things to check to debug the situation. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. -- You've waited an hour or [the configured - interval](index.md#adjust-ldap-group-sync-schedule) for the group to - sync. To speed up the process, either go to the GitLab group **Group information > Members** - and press **Sync now** (sync one group) or [run the group sync Rake - task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). +- You've waited an hour or [the configured interval](index.md#adjust-ldap-group-sync-schedule) for + the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** + and press **Sync now** (sync one group) or [run the group sync Rake task](../../raketasks/ldap.md#run-a-group-sync) + (sync all groups). If all of the above looks good, jump in to a little more advanced debugging in the rails console. @@ -371,8 +369,8 @@ the rails console. 1. Look through the output of the sync. See [example log output](#example-console-output-after-a-group-sync) for how to read the output. -1. If you still aren't able to see why the user isn't being added, [query the - LDAP group directly](#query-a-group-in-ldap) to see what members are listed. +1. If you still aren't able to see why the user isn't being added, [query the LDAP group directly](#query-a-group-in-ldap) + to see what members are listed. 1. Is the user's DN or UID in one of the lists from the above output? One of the DNs or UIDs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, the user does not appear to be in the LDAP group. @@ -387,7 +385,7 @@ the following are true: - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP - credentials. GitLab only grants this administrator access to the users whose + credentials. GitLab only grants the Administrator role to the users whose accounts are already connected to LDAP. If all the above are true and the users are still not getting access, [run a manual @@ -398,8 +396,8 @@ GitLab syncs the `admin_group`. #### Sync all groups NOTE: -To sync all groups manually when debugging is unnecessary, [use the Rake -task](../../raketasks/ldap.md#run-a-group-sync) instead. +To sync all groups manually when debugging is unnecessary, +[use the Rake task](../../raketasks/ldap.md#run-a-group-sync) instead. The output from a manual [group sync](index.md#group-sync) can show you what happens when GitLab syncs its LDAP group memberships against LDAP. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 6a037e75f54..12729f2050b 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -228,8 +228,7 @@ Azure B2C [offers two ways of defining the business logic for logging in a user] While cumbersome to configure, custom policies are required because standard Azure B2C user flows [do not send the OpenID `email` claim](https://github.com/MicrosoftDocs/azure-docs/issues/16566). In -other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` -parameters](../../integration/omniauth.md#initial-omniauth-configuration). +other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#initial-omniauth-configuration). With a standard Azure B2C policy, GitLab cannot create a new account or link to an existing one with an email address. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 7e2699d5eb3..d79837776b2 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -126,7 +126,7 @@ more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/ gitlab_rails['smartcard_client_certificate_required_port'] = 3444 ``` - NOTE: **Note** + NOTE: Assign a value to at least one of the following variables: `gitlab_rails['smartcard_client_certificate_required_host']` or `gitlab_rails['smartcard_client_certificate_required_port']`. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 6afaff73396..226710a8911 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -104,7 +104,7 @@ In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. See also the [user documentation](../../user/clusters/agent/index.md#troubleshooting) for troubleshooting problems with individual agents. -### KAS logs - GitOps: failed to get project info +### KAS logs - GitOps: failed to get project information If you get the following error message: diff --git a/doc/administration/configure.md b/doc/administration/configure.md index d3e37b4a0ee..822acc1a74e 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -7,10 +7,44 @@ type: reference # Configure your GitLab installation **(FREE SELF)** -Customize and configure your self-managed GitLab installation. +Customize and configure your self-managed GitLab installation. Here are some quick links to get you started: - [Authentication](auth/index.md) - [Configuration](../user/admin_area/index.md) - [Repository storage](repository_storage_paths.md) - [Geo](geo/index.md) - [Packages](packages/index.md) + +The following tables are intended to guide you to choose the right combination of capabilties based on your requirements. It is common to want the most +available, quickly recoverable, highly performant and fully data resilient solution. However, there are tradeoffs. + +The tables lists features on the left and provides their capabilities to the right along with known trade-offs. + +## Gitaly Capabilities + +| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|-|--------------|----------------|-----------------|-------------|-----------------| +|Gitaly Cluster | Very high - tolerant of node failures | RTO for a single node of 10s with no manual intervention | Data is stored on multiple nodes | Good - While writes may take slightly longer due to voting, read distribution improves read speeds | **Trade-off** - Slight decrease in write speed for redundant, strongly-consistent storage solution. **Risks** - [Does not currently support snapshot backups](gitaly/index.md#snapshot-backup-and-recovery-limitations), GitLab backup task can be slow for large data sets | +|Gitaly Shards | Single storage location is a single point of failure | Would need to restore only shards which failed | Single point of failure | Good - can allocate repositories to shards to spread load | **Trade-off** - Need to manually configure repositories into different shards to balance loads / storage space **Risks** - Single point of failure relies on recovery process when single-node failure occurs | +|Gitaly + NFS | Single storage location is a single point of failure | Single node failure requires restoration from backup | Single point of failure | Average - NFS is not ideally suited to large quantities of small reads / writes which can have a detrimental impact on performance | **Trade-off** - Easy and familiar administration though NFS is not ideally suited to Git demands **Risks** - Many instances of NFS compatibility issues which provide very poor customer experiences | + +## Geo Capabilities + +If your availabity needs to span multiple zones or multiple locations, please read about [Geo](geo/index.md). + +| | Availability | Recoverability | Data Resiliency | Performance | Risks/Trade-offs| +|-|--------------|----------------|-----------------|-------------|-----------------| +|Geo| Depends on the architecture of the Geo site. It is possible to deploy secondaries in single and multiple node configurations. | Eventually consistent. Recovery point depends on replication lag, which depends on a number of factors such as network speeds. Geo supports failover from a primary to secondary site using manual commands that are scriptable. | Geo currently replicates 100% of planned data types and verifies 50%. See [limitations table](geo/replication/datatypes.md#limitations-on-replicationverification) for more detail. | Improves read/clone times for users of a secondary. | Geo is not intended to replace other backup/restore solutions. Because of replication lag and the possibility of replicating bad data from a primary, we recommend that customers also take regular backups of their primary site and test the restore process. | + +## Scenarios for failure modes and available mitigation paths + +The following table outlines failure modes and mitigation paths for the product offerings detailed in the tables above. Note - Gitaly Cluster install assumes an odd number replication factor of 3 or greater + +| Gitaly Mode | Loss of Single Gitaly Node | Application / Data Corruption | Regional Outage (Loss of Instance) | Notes | +| ----------- | -------------------------- | ----------------------------- | ---------------------------------- | ----- | +| Single Gitaly Node | Downtime - Must restore from backup | Downtime - Must restore from Backup | Downtime - Must wait for outage to end | | +| Single Gitaly Node + Geo Secondary | Downtime - Must restore from backup, can perform a manual failover to secondary | Downtime - Must restore from Backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Sharded Gitaly Install | Partial Downtime - Only repos on impacted node affected, must restore from backup | Partial Downtime - Only repos on impacted node affected, must restore from backup | Downtime - Must wait for outage to end | | +| Sharded Gitaly Install + Geo Secondary | Partial Downtime - Only repos on impacted node affected, must restore from backup, could perform manual failover to secondary for impacted repos | Partial Downtime - Only repos on impacted node affected, must restore from backup, errors could have propagated to secondary | Manual intervention - failover to Geo secondary | | +| Gitaly Cluster Install* | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Downtime - Must wait for outage to end | Snapshot backups for Gitaly Cluster nodes not supported at this time | +| Gitaly Cluster Install* + Geo Secondary | No Downtime - will swap repository primary to another node after 10 seconds | N/A - All writes are voted on by multiple Gitaly Cluster nodes | Manual intervention - failover to Geo secondary | Snapshot backups for Gitaly Cluster nodes not supported at this time | diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 057abce0ed5..3af80363916 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -21,6 +21,7 @@ You can use the following environment variables to override certain values: |--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| | `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | | `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `EXTERNAL_URL` | string | Specify the external URL at the [time of installation](https://docs.gitlab.com/omnibus/settings/configuration.html#specifying-the-external-url-at-the-time-of-installation). | | `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | | `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | | `EXTERNAL_VALIDATION_SERVICE_TOKEN` | string | The `X-Gitlab-Token` for authentication with an [external CI/CD pipeline validation service](external_pipeline_validation.md). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 738cf591210..a4ed287cc3b 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -76,7 +76,8 @@ required number of seconds. "email": { "type": "string" }, "created_at": { "type": ["string", "null"], "format": "date-time" }, "current_sign_in_ip": { "type": ["string", "null"] }, - "last_sign_in_ip": { "type": ["string", "null"] } + "last_sign_in_ip": { "type": ["string", "null"] }, + "sign_in_count": { "type": "integer" } } }, "pipeline": { diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 575fa9eb229..f2067e7a2d1 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -1,8 +1,7 @@ --- -stage: none -group: Development -info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" -type: reference +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" --- diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index f73c961f541..e041f1e11c6 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -7,16 +7,15 @@ type: reference # File hooks **(FREE SELF)** -> - Introduced in GitLab 10.6. -> - Until GitLab 12.8, the feature name was Plugins. +> Renamed feature from Plugins to File hooks in GitLab 12.8. With custom file hooks, GitLab administrators can introduce custom integrations without modifying the GitLab source code. -NOTE: -Instead of writing and supporting your own file hook you can make changes -directly to the GitLab source code and contribute back upstream. This way we can -ensure functionality is preserved across versions and covered by tests. +A file hook runs on each event. You can filter events or projects +in a file hook's code, and create many file hooks as you need. Each file hook is +triggered by GitLab asynchronously in case of an event. For a list of events +see the [system hooks](../system_hooks/system_hooks.md) documentation. NOTE: File hooks must be configured on the file system of the GitLab server. Only GitLab @@ -24,10 +23,9 @@ server administrators can complete these tasks. Explore [system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have file system access. -A file hook runs on each event. You can filter events or projects -in a file hook's code, and create many file hooks as you need. Each file hook is -triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks](../system_hooks/system_hooks.md) documentation. +Instead of writing and supporting your own file hook, you can also make changes +directly to the GitLab source code and contribute back upstream. In this way, we can +ensure functionality is preserved across versions and covered by tests. ## Setup @@ -67,7 +65,7 @@ message is logged to: - `log/file_hook.log` in a source installation. NOTE: -Before 14.0 release, the file name was `plugin.log` +In GitLab 13.12 and earlier, the filename was `plugin.log` ## Creating file hooks diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index a7a64701cbd..6312ed669ae 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -162,6 +162,9 @@ be disabled on the **primary** site: ## Finish replicating and verifying all data +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + 1. If you are manually replicating any data not managed by Geo, trigger the final replication process now. 1. On the **primary** node: @@ -192,12 +195,13 @@ At this point, your **secondary** node contains an up-to-date copy of everything ## Promote the **secondary** node -Finally, follow the [Disaster Recovery docs](index.md) to promote the -**secondary** node to a **primary** node. This process causes a brief outage on the **secondary** node, and users may need to log in again. +After the replication is finished, [promote the **secondary** node to a **primary** node](index.md). This process causes a brief outage on the **secondary** node, and users may need to log in again. If you follow the steps correctly, the old primary Geo site should still be disabled and user traffic should go to the newly-promoted site instead. -Once it is completed, the maintenance window is over! Your new **primary** node, now -begin to diverge from the old one. If problems do arise at this point, failing +When the promotion is completed, the maintenance window is over, and your new **primary** node now +begins to diverge from the old one. If problems do arise at this point, failing back to the old **primary** node [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. -Don't forget to remove the broadcast message after failover is complete. +Don't forget to remove the broadcast message after the failover is complete. + +Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-node-to-be-a-secondary-node). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 4255fba83f6..3eb7bc2a8e0 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -63,6 +63,9 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + On the **secondary** node: 1. On the top bar, select **Menu > Admin**. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index 18923da1056..d4782144df8 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -51,6 +51,9 @@ Before following any of those steps, make sure you have `root` access to the **secondary** to promote it, since there isn't provided an automated way to promote a Geo replica and perform a failover. +NOTE: +GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 3f38436429a..e8e87f92481 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -198,12 +198,12 @@ successfully, you must replicate their data using some other means. |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| -|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | -|[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | -|[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. | -|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries currently use the same ES cluster as the primary. | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | No | Not planned because they are ephemeral and sensitive information. They can be regenerated on demand. | #### Limitation of verification for files in Object Storage diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index c6b1078ddf0..a4c2f156216 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -114,6 +114,13 @@ The following are GitLab upgrade validation tests we performed. The following are PostgreSQL upgrade validation tests we performed. +### September 2021 + +[Verify Geo installation with PostgreSQL 13](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6131): + +- Description: With PostgreSQL 13 available as an opt-in version in GitLab 14.1, we tested fresh installations of GitLab with Geo when PostgreSQL 13 is enabled. +- Outcome: Successfully built an environment with Geo and PostgreSQL 13 using [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) and performed Geo QA tests against the environment without failures. + ### September 2020 [Verify PostgreSQL 12 upgrade for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5454): diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 7db210d31f4..87b1aa7fc44 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -199,7 +199,7 @@ then make the following modifications: ## `application_role` already enables this. You only need this line if ## you selectively enable individual services that depend on Rails, like - ## `puma`, `sidekiq`, `geo-logcursor`, etc. + ## `puma`, `sidekiq`, `geo-logcursor`, and so on. gitlab_rails['enable'] = true ## diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index 1f799b30125..3a10d3bad58 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -73,7 +73,7 @@ GitLab does not currently support the case where both: ## Third-party replication services When using Amazon S3, you can use -[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to +[Cross-Region Replication (CRR)](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to have automatic replication between the bucket used by the **primary** site and the bucket used by **secondary** sites. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 966902a3d74..df893298f85 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -26,7 +26,7 @@ from [owasp.org](https://owasp.org/). - Geo streams almost all data held by a GitLab instance between sites. This includes full database replication, most files (user-uploaded attachments, - etc) and repository + wiki data. In a typical configuration, this will + and so on) and repository + wiki data. In a typical configuration, this will happen across the public Internet, and be TLS-encrypted. - PostgreSQL replication is TLS-encrypted. - See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2948) diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 7b82d742bd5..b7370d32056 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -83,7 +83,7 @@ Checking Geo ... Finished #### Sync status Rake task Current sync information can be found manually by running this Rake task on any -**secondary** app node: +node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** site: ```shell sudo gitlab-rake geo:status @@ -292,9 +292,8 @@ be set on the **primary** database. In GitLab 9.4, we have made this setting default to 1. You may need to increase this value if you have more **secondary** nodes. -Be sure to restart PostgreSQL for this to take -effect. See the [PostgreSQL replication -setup](../setup/database.md#postgresql-replication) guide for more details. +Be sure to restart PostgreSQL for this to take effect. See the +[PostgreSQL replication setup](../setup/database.md#postgresql-replication) guide for more details. ### Message: `FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist`? @@ -430,7 +429,7 @@ their resync may take a long time and cause significant load on your Geo nodes, storage and network systems. If you get the error `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file -of a repository on the secondary Geo node's filesystem: +of a repository on the secondary Geo node's file system: ```json { @@ -803,7 +802,7 @@ get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::Invalid get_ctl_options': invalid option: --force (OptionParser::InvalidOption) ``` -This can happen with XFS or filesystems that list files in lexical order, because the +This can happen with XFS or file systems that list files in lexical order, because the load order of the Omnibus command files can be different than expected, and a global function would get redefined. More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). @@ -923,6 +922,14 @@ To resolve this issue: If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the `/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites will stop getting updated. After 10 minutes, the status will become `Unhealthy`. + +Geo secondary sites will continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. + +This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issues/292983). + ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 84193e6baac..1b22a5f0991 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -13,6 +13,8 @@ for updating Geo nodes. ## Updating to 14.1, 14.2, 14.3 +### Multi-arch images + We found an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/336013) where the Container Registry replication wasn't fully working if you used multi-arch images. In case of a multi-arch image, only the primary architecture (for example `amd64`) would be replicated to the secondary node. This has been [fixed in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67624) and was backported to 14.2 and 14.1, but manual steps are required to force a re-sync. You can check if you are affected by running: @@ -46,18 +48,28 @@ Otherwise, on all your **secondary** nodes, in a [Rails console](../../operation If you are running a version prior to 14.1 and are using Geo and multi-arch containers in your Container Registry, we recommend [upgrading](updating_the_geo_sites.md) to at least GitLab 14.1. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 14.0/14.1 +### Primary sites can not be removed from the UI + We found an issue where [Primary sites can not be removed from the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/338231). This bug only exists in the UI and does not block the removal of Primary sites using any other method. -### If you have already updated to an affected version and need to remove your Primary site +If you are running an affected version and need to remove your Primary site, you can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node). -You can manually remove the Primary site by using the [Geo Nodes API](../../../api/geo_nodes.md#delete-a-geo-node). +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). ## Updating to GitLab 13.12 +### Secondary nodes re-download all LFS files upon update + We found an issue where [secondary nodes re-download all LFS files](https://gitlab.com/gitlab-org/gitlab/-/issues/334550) upon update. This bug: - Only applies to Geo secondary sites that have replicated LFS objects. @@ -68,7 +80,7 @@ We found an issue where [secondary nodes re-download all LFS files](https://gitl If you don't have many LFS objects or can stand a bit of churn, then it is safe to let the secondary sites re-download LFS objects. If you do have many LFS objects, or many Geo secondary sites, or limited bandwidth, or a combination of them all, then we recommend you skip GitLab 13.12.0 through 13.12.6 and update to GitLab 13.12.7 or newer. -### If you have already updated to an affected version, and the re-sync is ongoing +#### If you have already updated to an affected version, and the re-sync is ongoing You can manually migrate the legacy sync state to the new state column by running the following command in a [Rails console](../../operations/rails_console.md). It should take under a minute: @@ -76,15 +88,31 @@ You can manually migrate the legacy sync state to the new state column by runnin Geo::LfsObjectRegistry.where(state: 0, success: true).update_all(state: 2) ``` +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.11 We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + +## Updating to GitLab 13.10 + +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.9 +### Error during zero-downtime update: "cannot drop column asset_proxy_whitelist" + We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) that will prevent upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary -to perform the following additional steps for the zero-downtime upgrade: +to perform the following additional steps for the zero-downtime update: 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, execute the following queries using the PostgreSQL console (or `sudo gitlab-psql`) @@ -118,6 +146,10 @@ DETAIL: trigger trigger_0d588df444c8 on table application_settings depends on co To work around this bug, follow the previous steps to complete the update. More details are available [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160). +### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode + +GitLab 13.9 through GitLab 14.3 are affected by a bug in which enabling [GitLab Maintenance Mode](../../maintenance_mode/index.md) will cause Geo secondary site statuses to appear to stop updating and become unhealthy. For more information, see [Troubleshooting - Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). + ## Updating to GitLab 13.7 We've detected an issue with the `FetchRemove` call used by Geo secondaries. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index c0d5a45d8d1..2455aecafea 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -54,7 +54,7 @@ Get started: You may need to import projects from external sources like GitHub, Bitbucket, or another instance of GitLab. Many external sources can be imported into GitLab. - Review the [GitLab projects documentation](../user/project/index.md#project-integrations). -- Consider [repository mirroring](../user/project/repository/repository_mirroring.md)—an [alternative to project migrations](../ci/ci_cd_for_external_repos/index.md). +- Consider [repository mirroring](../user/project/repository/mirror/index.md)—an [alternative to project migrations](../ci/ci_cd_for_external_repos/index.md). - Check out our [migration index](../user/project/import/index.md) for documentation on common migration paths. - Schedule your project exports with our [import/export API](../api/project_import_export.md#schedule-an-export). @@ -128,7 +128,7 @@ The routine differs, depending on whether you deployed with Omnibus or the Helm When you backing up an Omnibus (single node) GitLab server, you can use a single Rake task. -Learn about [backing up Omnibus or Helm variations](../raketasks/backup_restore.md#back-up-gitlab). +Learn about [backing up Omnibus or Helm variations](../raketasks/backup_restore.md). This process backs up your entire instance, but does not back up the configuration files. Ensure those are backed up separately. Keep your configuration files and backup archives in a separate location to ensure the encryption keys are not kept with the encrypted data. @@ -214,7 +214,7 @@ If you use GitLab SaaS, you have several channels with which to get support and To get assistance for GitLab SaaS: -- Access [GitLab Docs](../README.md) for self-service support. +- Access [GitLab Docs](../index.md) for self-service support. - Join the [GitLab Forum](https://forum.gitlab.com/) for community support. - Gather [your subscription information](https://about.gitlab.com/support/#for-self-managed-users) before submitting a ticket. - Submit a support ticket for: diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md index c7ecaa020e0..f79b9555c10 100644 --- a/doc/administration/gitaly/faq.md +++ b/doc/administration/gitaly/faq.md @@ -35,7 +35,7 @@ For more information, see: ## Are there instructions for migrating to Gitaly Cluster? -Yes! For more information, see [Migrate to Gitaly Cluster](index.md#migrate-to-gitaly-cluster). +Yes! For more information, see [Migrating to Gitaly Cluster](index.md#migrating-to-gitaly-cluster). ## What are some repository storage recommendations? diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 7a7aac884ed..c689530e12c 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -10,14 +10,22 @@ type: reference [Gitaly](https://gitlab.com/gitlab-org/gitaly) provides high-level RPC access to Git repositories. It is used by GitLab to read and write Git data. +Gitaly is present in every GitLab installation and coordinates Git repository +storage and retrieval. Gitaly can be: + +- A simple background service operating on a single instance Omnibus GitLab (all of + GitLab on one machine). +- Separated onto its own instance and configured in a full cluster configuration, + depending on scaling and availability requirements. + Gitaly implements a client-server architecture: - A Gitaly server is any node that runs Gitaly itself. -- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. These - include, but are not limited to: - - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). - - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). - - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). +- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. Gitaly clients are also known as _Gitaly consumers_ and include: + - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab) + - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) + - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) + - [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) Gitaly manages only Git repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. @@ -35,9 +43,49 @@ repository storage is either: - Read requests are distributed between multiple Gitaly nodes, which can improve performance. - Write requests are broadcast to repository replicas. -WARNING: -Engineering support for NFS for Git repositories is deprecated. Read the -[deprecation notice](#nfs-deprecation-notice). +## Guidance regarding Gitaly Cluster + +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please review existing technical limitations and considerations prior to deploying Gitaly Cluster. + +- [Known issues](#known-issues) +- [Snapshot limitations](#snapshot-backup-and-recovery-limitations). + +Please also review the [configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best set-up for you. Finally, refer to the following guidance: + +- If you have not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the + service you are using. NFS is supported in 14.x releases. +- If you have not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options - a sharded Gitaly instance or Gitaly Cluster. +- If you have migrated to Gitaly Cluster and the limitations and tradeoffs are not suitable for your environment, your options are: + 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) back to your NFS solution + 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) to NFS solution or to a sharded Gitaly instance. + +Reach out to your Technical Account Manager or customer support if you have any questions. + +### Known issues + +The following table outlines current known issues impacting the use of Gitaly Cluster. For +the current status of these issues, please refer to the referenced issues and epics. + +| Issue | Summary | How to avoid | +|:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| +| Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | +| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage that do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | Don't directly change repositories on any Gitaly Cluster node at this time. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | +| Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you need to restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | + +### Snapshot backup and recovery limitations + +Gitaly Cluster does not support snapshot backups because these can cause issues where the Praefect +database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication +metadata of Gitaly disk information during a restore, we recommend using the +[official backup and restore Rake tasks](../../raketasks/backup_restore.md). If you are unable to use this method, please contact customer support for restoration help. + +To track progress on work on a solution for manually re-synchronizing the Praefect database with +disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). + +### What to do if you are on Gitaly Cluster experiencing an issue or limitation + +Please contact customer support for immediate help in restoration or recovery. ## Gitaly @@ -156,54 +204,6 @@ WARNING: If complete cluster failure occurs, disaster recovery plans should be executed. These can affect the RPO and RTO discussed above. -### Architecture and configuration recommendations - -The following table provides recommendations based on your -requirements. Users means concurrent users actively performing -simultaneous Git Operations. - -Gitaly services are present in every GitLab installation and always coordinates Git repository storage and -retrieval. Gitaly can be as simple as a single background service when operating on a single instance Omnibus -GitLab (All of GitLab on one machine). Gitaly can be separated into it's own instance and it can be configured in -a full cluster configuration depending on scaling and availability requirements. - -The GitLab Reference Architectures provide guidance for what Gitaly configuration is advisable at each of the scales. -The Gitaly configuration is noted by the architecture diagrams and the table of required resources. - -| User Scaling | Reference Architecture To Use | GitLab Instance Configuration | Gitaly Configuration | Git Repository Storage | Instances Dedicated to Gitaly Services | -| ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------- | ------------------------------- | ---------------------------------- | -------------------------------------- | -| Up to 1000 Users | [1K](../reference_architectures/1k_users.md) | Single Instance for all of GitLab | Already Integrated as a Service | Local Disk | 0 | -| Up to 2999 Users | [2K](../reference_architectures/2k_users.md) | Horizontally Scaled GitLab Instance (Non-HA) | Single Gitaly Server | Local Disk of Gitaly Instance | 1 | -| 3000 Users and Over | [3K](../reference_architectures/1k_users.md) | Horizontally Scaled GitLab Instance with HA | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | -| RTO/RPO Requirements for AZ Failure Tolerance Regardless of User Scale | [3K (with downscaling)](../reference_architectures/3k_users.md) | Custom (1) | Gitaly Cluster | Local Disk of Each Gitaly Instance | 8 | - -1. If you feel that you need AZ Failure Tolerance for user scaling lower than 3K, please contact Customer Success - to discuss your RTO and RPO needs and what options exist to meet those objectives. - -WARNING: -At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) -exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. -We will adjust the date for NFS support removal if this applies to you. - -### Known issues impacting Gitaly Cluster - -The following table outlines current known issues impacting the use of Gitaly Cluster. For -the most up to date status of these issues, please refer to the referenced issues / epics. - -| Issue | Summary | -| Gitaly Cluster + Geo can cause database inconsistencies | There are some conditions during Geo replication that can cause database inconsistencies with Gitaly Cluster. These have been identified and are being resolved by updating Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485). | -| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage which do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | - -### Snapshot backup and recovery limitations - -Gitaly Cluster does not support snapshot backups because these can cause issues where the -Praefect database becomes out of sync with the disk storage. Because of how Praefect rebuilds -the replication metadata of Gitaly disk information during a restore, we recommend using the -[official backup and restore Rake tasks](../../raketasks/backup_restore.md). - -To track progress on work on a solution for manually re-synchronizing the Praefect database -with disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). - ### Virtual storage Virtual storage makes it viable to have a single repository storage in GitLab to simplify repository @@ -232,9 +232,7 @@ As with normal Gitaly storages, virtual storages can be sharded. ### Moving beyond NFS -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. +Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable starting GitLab 15.0. Please see our [statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for more details. [Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) is not well suited to Git workloads which are CPU and IOPS sensitive. @@ -355,13 +353,9 @@ For configuration information, see [Configure replication factor](praefect.md#co For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). -## Migrate to Gitaly Cluster - -We recommend you migrate to Gitaly Cluster if your -[requirements recommend](#architecture-and-configuration-recommendations) Gitaly Cluster. +## Migrating to Gitaly Cluster -Whether migrating to Gitaly Cluster because of [NFS support deprecation](index.md#nfs-deprecation-notice) -or to move from single Gitaly nodes, the basic process involves: +Please see [current guidance on Gitaly Cluster](#guidance-regarding-gitaly-cluster). The basic process for migrating to Gitaly Cluster involves: 1. Create the required storage. Refer to [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). @@ -371,9 +365,20 @@ or to move from single Gitaly nodes, the basic process involves: automatic migration but the moves can be scheduled with the GitLab API. WARNING: -At present, some [known database inconsistency issues](#known-issues-impacting-gitaly-cluster) -exist in Gitaly Cluster. It is our recommendation that for now, you remain on your current service. -We will adjust the date for NFS support removal if this applies to you. +Some [known database inconsistency issues](#known-issues) exist in Gitaly Cluster. We recommend you +remain on your current service for now. + +### Migrate off Gitaly Cluster + +If you have repositories stored on a Gitaly Cluster, but you'd like to migrate +them back to direct Gitaly storage: + +1. Create and configure a new + [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). +1. [Move the repositories](../operations/moving_repositories.md#move-repositories) + to the newly created storage. There are different possibilities to move them + by shard or by group, this gives you the opportunity to spread them over + multiple Gitaly servers. ## Monitor Gitaly and Gitaly Cluster @@ -615,20 +620,4 @@ The second facet presents the only real solution. For this, we developed ## NFS deprecation notice Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Additional information: - -- [Recommended NFS mount options and known issues with Gitaly and NFS](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). -- [GitLab statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs). - -GitLab recommends: - -- Creating a [Gitaly Cluster](#gitaly-cluster) as soon as possible. -- [Moving your repositories](#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly - Cluster. - -We welcome your feedback on this process. You can: - -- Raise a support ticket. -- [Comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). +unavailable from GitLab 15.0. For further information, please see our [NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation) documentation. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index eb666f1caf4..ce5fb1aaf88 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -265,8 +265,8 @@ praefect['database_direct_dbname'] = 'praefect_production' #praefect['database_direct_sslrootcert'] = '...' ``` -We recommend using PgBouncer with `session` pool mode instead. You can use the [bundled -PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it +We recommend using PgBouncer with `session` pool mode instead. You can use the +[bundled PgBouncer](../postgresql/pgbouncer.md) or use an external PgBouncer and [configure it manually](https://www.pgbouncer.org/config.html). The following example uses the bundled PgBouncer and sets up two separate connection pools, @@ -429,7 +429,7 @@ On the **Praefect** node: WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -475,8 +475,8 @@ On the **Praefect** node: 1. [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2013) in GitLab 13.1 and later, enable [distribution of reads](index.md#distributed-reads). -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure @@ -499,16 +499,16 @@ On the **Praefect** node: running reconfigure automatically when running commands such as `apt-get update`. This way any additional configuration changes can be done and then reconfigure can be run manually. -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure ``` 1. To ensure that Praefect [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart - Praefect](../restart_gitlab.md#omnibus-gitlab-restart): + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), + [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart praefect @@ -695,8 +695,8 @@ Particular attention should be shown to: was set in the [previous section](#praefect). This document uses `gitaly-1`, `gitaly-2`, and `gitaly-3` as Gitaly storage names. -For more information on Gitaly server configuration, see our [Gitaly -documentation](configure_gitaly.md#configure-gitaly-servers). +For more information on Gitaly server configuration, see our +[Gitaly documentation](configure_gitaly.md#configure-gitaly-servers). 1. SSH into the **Gitaly** node and login as root: @@ -803,16 +803,16 @@ documentation](configure_gitaly.md#configure-gitaly-servers). }) ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure ``` 1. To ensure that Gitaly [has updated its Prometheus listen - address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart - Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), + [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart gitaly @@ -893,7 +893,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](index.md#migrate-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](index.md#migrating-to-gitaly-cluster) first. ```ruby @@ -1044,8 +1044,8 @@ To get started quickly: grafana['disable_login_form'] = false ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and + [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure @@ -1309,12 +1309,7 @@ To minimize data loss in GitLab 13.0 to 14.0, Gitaly Cluster: new primary. If the failed primary contained unreplicated writes, [data loss can occur](#check-for-data-loss). > - Removed in GitLab 14.1. Instead, repositories [become unavailable](#unavailable-repositories). -In GitLab 13.0 to 14.0, when Gitaly Cluster switches to a new primary, repositories enter -read-only mode if they are out of date. This can happen after failing over to an outdated -secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict -with the unreplicated writes on other nodes. - -When Gitaly Cluster switches to a new primary In GitLab 13.0 to 14.0, repositories enter +When Gitaly Cluster switches to a new primary in GitLab 13.0 to 14.0, repositories enter read-only mode if they are out of date. This can happen after failing over to an outdated secondary. Read-only mode eases data recovery efforts by preventing writes that may conflict with the unreplicated writes on other nodes. @@ -1596,3 +1591,26 @@ because of: - An in-flight RPC call targeting the repository. If this occurs, run `remove-repository` again. + +### Manually list untracked repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3926) in GitLab 14.4. + +The `list-untracked-repositories` Praefect sub-command lists repositories of the Gitaly Cluster that both: + +- Exist for at least one Gitaly storage. +- Aren't tracked in the Praefect database. + +The command outputs: + +- Result to `STDOUT` and the command's logs. +- Errors to `STDERR`. + +Each entry is a complete JSON string with a newline at the end (configurable using the +`-delimiter` flag). For example: + +```shell +sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml list-untracked-repositories +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567890.git"} +{"virtual_storage":"default","storage":"gitaly-1","relative_path":"@hashed/ab/cd/abcd123456789012345678901234567890123456789012345678901234567891.git"} +``` diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1b53a0994f5..1f90ebb7565 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -243,7 +243,7 @@ To mirror-push branches and tags only, and avoid attempting to mirror-push prote git push origin +refs/heads/*:refs/heads/* +refs/tags/*:refs/tags/* ``` -Any other namespaces that the admin wants to push can be included there as well via additional patterns. +Any other namespaces that the administrator wants to push can be included there as well via additional patterns. ### Command line tools cannot connect to Gitaly @@ -365,6 +365,15 @@ To determine the current primary Gitaly node for a specific Praefect node: curl localhost:9652/metrics | grep gitaly_praefect_primaries` ``` +### Check that repositories are in sync + +Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that +a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums) +that checksums the repository on all Gitaly nodes. + +The [Praefect dataloss](praefect.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot +be relied to detect sync problems in this scenario. + ### Relation does not exist errors By default Praefect database tables are created automatically by `gitlab-ctl reconfigure` task. @@ -393,7 +402,7 @@ $ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config praefect sql-migrate: OK (applied 21 migrations) ``` -### Requests fail with 'repo scoped: invalid Repository' errors +### Requests fail with 'repository scoped: invalid Repository' errors This indicates that the virtual storage name used in the [Praefect configuration](praefect.md#praefect) does not match the storage name used in diff --git a/doc/administration/index.md b/doc/administration/index.md index 9412994edb7..ee17edc35fc 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -41,7 +41,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Configuring GitLab -- [Adjust your instance's timezone](timezone.md): Customize the default time zone of GitLab. +- [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab. - [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/index.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 24ffee088f3..a2729e60545 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -88,6 +88,29 @@ requests per user. For more information, read - **Default rate limit**: Disabled by default. +### Files API + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the `files_api_throttling` flag](../administration/feature_flags.md). On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +The feature is not ready for production use. + +This setting limits the request rate on the Packages API per user or IP address. For more information, read +[Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md). + +- **Default rate limit**: Disabled by default. + +### Deprecated API endpoints + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68645) in GitLab 14.4. + +This setting limits the request rate on deprecated API endpoints per user or IP address. For more information, read +[Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md). + +- **Default rate limit**: Disabled by default. + ### Import/Export > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. @@ -212,7 +235,7 @@ Activity history for projects and individuals' profiles was limited to one year > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14939) in GitLab 12.7. -There is a limit when embedding metrics in GFM for performance reasons. +There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for performance reasons. - **Max limit**: 100 embeds. @@ -240,10 +263,10 @@ Set the limit to `0` to disable it. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. -The [minimum wait time between pull refreshes](../user/project/repository/repository_mirroring.md) +The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) defaults to 300 seconds (5 minutes). For example, by default a pull refresh will only run once in a given 300 second period regardless of how many times you try to trigger it. -This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/repository_mirroring.md#how-it-works). +This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). To change this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -511,7 +534,11 @@ Plan.default.actual_limits.update!(pages_file_entries: 100) ### Number of registered runners per scope -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. Disabled by default. +> - Enabled on GitLab.com in GitLab 14.3. +> - Enabled on self-managed in GitLab 14.4. +> - Feature flag `ci_runner_limits` removed in GitLab 14.4. You can still use `ci_runner_limits_override` + to remove limits for a given scope. The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, GitLab checks these limits against runners that have been active in the last 3 months. @@ -749,7 +776,7 @@ than the specified limit, hooks won't be executed. More information can be found in these docs: -- [Webhooks push events](../user/project/integrations/webhooks.md#push-events) +- [Webhooks push events](../user/project/integrations/webhook_events.md#push-events) - [Project services push hooks limit](../user/project/integrations/overview.md#push-hooks-limit) ### Activities diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index b166bb32aa1..62897651166 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -12,7 +12,7 @@ If you run a medium-sized self-managed instance (50+ users) of a free version of [either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), you qualify for a free Instance Review. -1. Sign in as a user with administrator [permissions](../user/permissions.md). +1. Sign in as a user with Administrator [role](../user/permissions.md). 1. In the top menu, click your user icon, and select **Get a free instance review**: diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 882580b35b6..45b94781adc 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Web terminals **(FREE)** -With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), +With the introduction of the [Kubernetes integration](../../user/infrastructure/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to [web terminals](../../ci/environments/index.md#web-terminals) for environments. @@ -50,8 +50,8 @@ detail below. ## Enabling and disabling terminal support NOTE: -AWS Elastic Load Balancers (ELBs) do not support web sockets. -If you want web terminals to work, use AWS Application Load Balancers (ALBs). +AWS Classic Load Balancers (CLBs) do not support web sockets. +If you want web terminals to work, use AWS Network Load Balancers (NLBs). Read [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) for more information. diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 7a880c81843..855910fec6a 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -15,8 +15,8 @@ in the cached text would still refer to the old URL. To avoid this problem, the administrator can invalidate the existing cache by increasing the `local_markdown_version` setting in application settings. This can -be done by [changing the application settings through -the API](../api/settings.md#change-application-settings): +be done by changing the application settings +[through the API](../api/settings.md#change-application-settings): ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number>" diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index b4c16e007cc..46a9ee11679 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -446,10 +446,10 @@ List the artifacts for a single project, sorted by artifact size. The output inc - on-disk location of the artifact ```ruby -p = Project.find_by_id(:project ID) +p = Project.find_by_id(<project_id>) arts = Ci::JobArtifact.where(project: p) -list = arts.order('sort DESC').limit(50).each do |art| +list = arts.order(size: :desc).limit(50).each do |art| puts "Job ID: #{art.job_id} - Size: #{art.size}b - Type: #{art.file_type} - Created: #{art.created_at} - File loc: #{art.file}" end ``` diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 64d9248cb16..f2748305c24 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -146,9 +146,9 @@ a background job archives the job log. The log is moved to `/var/opt/gitlab/gitl by default, or to object storage if configured. In a [scaled-out architecture](reference_architectures/index.md) with Rails and Sidekiq running on more than one -server, these two locations on the filesystem have to be shared using NFS. +server, these two locations on the file system have to be shared using NFS. -To eliminate both filesystem requirements: +To eliminate both file system requirements: - [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. - Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 682352d8f59..d2f220e3795 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -2,18 +2,15 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- # GitLab Git Large File Storage (LFS) Administration **(FREE SELF)** -> - Git LFS is supported in GitLab starting with version 8.2. -> - Support for object storage, such as AWS S3, was introduced in 10.0. -> - LFS is enabled in GitLab self-managed instances by default. - Documentation about how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). +LFS is enabled in GitLab self-managed instances by default. + ## Requirements - Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 or later. @@ -346,8 +343,6 @@ git lfs version ## Known limitations -- Support for removing unreferenced LFS objects was added in 8.14 onward. -- LFS authentications via SSH was added with GitLab 8.12. - Only compatible with the Git LFS client versions 1.1.0 and later, or 1.0.2. - The storage statistics count each LFS object for every project linking to it. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 990287e3907..a9fd698a525 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1069,6 +1069,14 @@ For Omnibus GitLab installations, Praefect logs are in `/var/log/gitlab/praefect GitLab also tracks [Prometheus metrics for Praefect](gitaly/#monitor-gitaly-cluster). +## Backup log + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63832) in GitLab 14.1. + +For Omnibus installations, the backup log is located at `/var/log/gitlab/gitlab-rails/backup_json.log`. + +This log is populated when a [GitLab backup is created](../raketasks/backup_restore.md). You can use this log to understand how the backup process performed. + ## Performance bar stats > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48149) in GitLab 13.7. @@ -1082,7 +1090,7 @@ Performance bar statistics (currently only duration of SQL queries) are recorded in that file. For example: ```json -{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"type": "sql"} +{"severity":"INFO","time":"2020-12-04T09:29:44.592Z","correlation_id":"33680b1490ccd35981b03639c406a697","filename":"app/models/ci/pipeline.rb","method_path":"app/models/ci/pipeline.rb:each_with_object","request_id":"rYHomD0VJS4","duration_ms":26.889,"count":2,"query_type": "active-record"} ``` These statistics are logged on .com only, disabled on self-deployments. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 39ee357cc2f..d5bcd132665 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -75,7 +75,7 @@ An error is displayed when a user tries to perform a write operation that isn't NOTE: In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197). -### Admin functions +### Administrator functions Systems administrators can edit the application settings. This allows them to disable Maintenance Mode after it's been enabled. @@ -111,18 +111,18 @@ For most JSON requests, POST, PUT, PATCH, and DELETE are blocked, and the API re |HTTP request | Allowed routes | Notes | |:----:|:--------------------------------------:|:----:| -| POST | `/admin/application_settings/general` | To allow updating application settings in the admin UI | +| POST | `/admin/application_settings/general` | To allow updating application settings in the administrator UI | | PUT | `/api/v4/application/settings` | To allow updating application settings with the API | | POST | `/users/sign_in` | To allow users to log in. | | POST | `/users/sign_out`| To allow users to log out. | | POST | `/oauth/token` | To allow users to log in to a Geo secondary for the first time. | -| POST | `/admin/session`, `/admin/session/destroy` | To allow [Admin mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | +| POST | `/admin/session`, `/admin/session/destroy` | To allow [Administrator mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | | POST | Paths ending with `/compare`| Git revision routes. | | POST | `.git/git-upload-pack` | To allow Git pull/clone. | | POST | `/api/v4/internal` | [internal API routes](../../development/internal_api.md) | -| POST | `/admin/sidekiq` | To allow management of background jobs in the admin UI | -| POST | `/admin/geo` | To allow updating Geo Nodes in the admin UI | -| POST | `/api/v4/geo_replication`| To allow certain Geo-specific admin UI actions on secondary sites | +| POST | `/admin/sidekiq` | To allow management of background jobs in the Admin UI | +| POST | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | +| POST | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | ### GraphQL API diff --git a/doc/administration/monitoring/performance/img/performance_bar_v14_0.png b/doc/administration/monitoring/performance/img/performance_bar_v14_0.png Binary files differdeleted file mode 100644 index 42261ddd720..00000000000 --- a/doc/administration/monitoring/performance/img/performance_bar_v14_0.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/img/performance_bar_v14_4.png b/doc/administration/monitoring/performance/img/performance_bar_v14_4.png Binary files differnew file mode 100644 index 00000000000..388d628c7c4 --- /dev/null +++ b/doc/administration/monitoring/performance/img/performance_bar_v14_4.png diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index ef4db93d5fc..0befd9eac5b 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -8,11 +8,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. > - The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. +> - The **Flamegraph** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30275) in GitLab 14.4. You can display the performance bar to see statistics for the performance of a GitLab UI page. For example: - + ## Available information @@ -64,6 +65,11 @@ From left to right, the performance bar displays: can be added by its full URL (authenticated as the current user), or by the value of its `X-Request-Id` header. - **Download**: a link to download the raw JSON used to generate the Performance Bar reports. +- **Flamegraph** with mode: a link to generate a [flamegraph](../../../development/profiling.md#speedscope-flamegraphs) + of the current URL with the selected [Stackprof mode](https://github.com/tmm1/stackprof#sampling): + - The **Wall** mode samples every *interval* of the time on a clock on a wall. The interval is set to `10100` microseconds. + - The **CPU** mode samples every *interval* of CPU activity. The interval is set to `10100` microseconds. + - The **Object** mode samples every *interval*. The interval is set to `100` allocations. - **Request Selector**: a select box displayed on the right-hand side of the Performance Bar which enables you to view these metrics for any requests made while the current page was open. Only the first two requests per unique URL are captured. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index c36d2b0f7a4..1d275010556 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -119,12 +119,15 @@ The following metrics are available: | `action_cable_pool_largest_size` | Gauge | 13.4 | Largest number of worker threads observed so far in ActionCable thread pool | `server_mode` | | `action_cable_pool_pending_tasks` | Gauge | 13.4 | Number of tasks waiting to be executed in ActionCable thread pool | `server_mode` | | `action_cable_pool_tasks_total` | Gauge | 13.4 | Total number of tasks executed in ActionCable thread pool | `server_mode` | +| `gitlab_ci_trace_operations_total` | Counter | 13.4 | Total amount of different operations on a build trace | `operation` | +| `gitlab_ci_trace_bytes_total` | Counter | 13.4 | Total amount of build trace bytes transferred | | | `action_cable_single_client_transmissions_total` | Counter | 13.10 | The number of ActionCable messages transmitted to any client in any channel | `server_mode` | | `action_cable_subscription_confirmations_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients confirmed | `server_mode` | | `action_cable_subscription_rejections_total` | Counter | 13.10 | The number of ActionCable subscriptions from clients rejected | `server_mode` | -| `action_cable_transmitted_bytes` | Histogram | 14.1 | Message size, in bytes, transmitted over action cable | `operation`, `channel` | +| `action_cable_transmitted_bytes` | Histogram | 14.1 | Message size, in bytes, transmitted over action cable | `operation`, `channel` | | `gitlab_issuable_fast_count_by_state_total` | Counter | 13.5 | Total number of row count operations on issue/merge request list pages | | | `gitlab_issuable_fast_count_by_state_failures_total` | Counter | 13.5 | Number of soft-failed row count operations on issue/merge request list pages | | +| `gitlab_ci_trace_finalize_duration_seconds` | Histogram | 13.6 | Duration of build trace chunks migration to object storage | | | `gitlab_external_http_total` | Counter | 13.8 | Total number of HTTP calls to external systems | `controller`, `action` | | `gitlab_external_http_duration_seconds` | Counter | 13.8 | Duration in seconds spent on each HTTP call to external systems | | | `gitlab_external_http_exception_total` | Counter | 13.8 | Total number of exceptions raised when making external HTTP calls | | @@ -132,15 +135,15 @@ The following metrics are available: | `pipeline_graph_link_calculation_duration_seconds` | Histogram | 13.9 | Total time spent calculating links, in seconds | | | `pipeline_graph_links_total` | Histogram | 13.9 | Number of links per graph | | | `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | -| `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | -| `gitlab_ci_difference_live_vs_actual_minutes` | Histogram | 13.12 | Difference between CI minute consumption counted while jobs were running (live) vs when jobs are complete (actual). Used to enforce CI minute consumption limits on long running jobs. | `plan` | -| `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | +| `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | +| `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | | `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | | `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | | `email_receiver_error` | Counter | 14.1 | Total number of errors when processing incoming emails | | | `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | | `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | | `gitlab_snowplow_successful_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission successes | | +| `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `type` | ## Metrics controlled by a feature flag diff --git a/doc/administration/monitoring/prometheus/puma_exporter.md b/doc/administration/monitoring/prometheus/puma_exporter.md new file mode 100644 index 00000000000..c348e74afaf --- /dev/null +++ b/doc/administration/monitoring/prometheus/puma_exporter.md @@ -0,0 +1,27 @@ +--- +stage: Monitor +group: Monitor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Puma exporter **(FREE SELF)** + +You can use the [Puma exporter](https://github.com/sapcc/puma-exporter) +to measure various Puma metrics. + +To enable the Puma exporter: + +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines. Make sure + `puma['exporter_enabled']` is set to `true`: + + ```ruby + puma['exporter_enabled'] = true + puma['exporter_address'] = "127.0.0.1" + puma['exporter_port'] = 8083 + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +Prometheus begins collecting performance data from the Puma exporter exposed at `localhost:8083`. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 0cab46a95c9..f18c39af24a 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -20,12 +20,46 @@ file system performance, see ## Gitaly and NFS deprecation +Starting with GitLab version 14.0, support for NFS to store Git repository data will be deprecated. Technical customer support and engineering support will be available for the 14.x releases. Engineering will fix bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). + +At the end of the 14.12 milestone (tenatively June 22nd, 2022) technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels. + +For those customers still running earlier versions of GitLab, [our support eligibility and maintenance policy applies](https://about.gitlab.com/support/statement-of-support.html#version-support). + +For the 14.x releases, we will continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: + +- Performance issues or timeouts accessing Git data +- Commits or branches vanish +- GitLab intermittently returns the wrong Git data (such as reporting that a repository has no branches) + +Assistance will be limited to activities like: + +- Verifying developers' workflow uses features like protected branches +- Reviewing GitLab event data from the database to advise if it looks like a force push over-wrote branches +- Verifying that NFS client mount options match our [documented recommendations](#mount-options) +- Analyzing the GitLab Workhorse and Rails logs, and determining that `500` errors being seen in the environment are caused by slow responses from Gitaly + +GitLab support will be unable to continue with the investigation if: + +- The date of the request is on or after the release of GitLab version 15.0, and +- Support Engineers and Management determine that all reasonable non-NFS root causes have been exhausted + +If the issue is reproducible, or if it happens intermittently but regularly, GitLab Support will investigate providing the issue reproduces without the use of NFS. In order to reproduce without NFS, the affected repositories should be migrated to a different Gitaly shard, such as Gitaly cluster or a standalone Gitaly VM, backed with block storage. + +### Why remove NFS for Git repository data + +{:.no-toc} + +NFS is not well-suited to a workload consisting of many small files, like Git repositories. NFS does provide a number of configuration options designed to improve performance. However, over time, a number of these mount options have proven to result in inconsistencies across multiple nodes mounting the NFS volume, up to and including data loss. Addressing these inconsistencies consume extraordinary development and support engineer time that hamper our ability to develop [Gitaly Cluster](gitaly/praefect.md), our purpose-built solution to addressing the deficiencies of NFS in this environment. + +Please note that Gitaly Cluster provides highly-available Git repository storage. If this is not a requirement, single-node Gitaly backed by block storage is a suitable substitute. + Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be unavailable from GitLab 15.0. No further enhancements are planned for this feature. Read: -- The [Gitaly and NFS deprecation notice](gitaly/index.md#nfs-deprecation-notice). +- [Moving beyond NFS](gitaly/index.md#moving-beyond-nfs). - About the [correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Known kernel version incompatibilities @@ -153,7 +187,7 @@ If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab NOTE: From GitLab 12.7, Rugged is not automatically enabled if Puma thread count is greater than `1`. -If you want to use Rugged with Puma, [set Puma thread count to `1`](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). +If you want to use Rugged with Puma, [set Puma thread count to `1`](../install/requirements.md#puma-settings). If you want to use Rugged with Puma thread count more than `1`, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code). @@ -217,7 +251,7 @@ use the `hard` option, because (from the man page): > use the soft option only when client responsiveness is more important than data integrity Other vendors make similar recommendations, including -[SAP](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's +[System Applications and Products in Data Processing (SAP)](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's [knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), they highlight that if the NFS client driver caches data, `soft` means there is no certainty if writes by GitLab are actually on disk. @@ -351,7 +385,7 @@ Any `Operation not permitted` errors means you should investigate your NFS serve If the traffic between your NFS server and NFS client(s) is subject to port filtering by a firewall, then you need to reconfigure that firewall to allow NFS communication. -[This guide from TDLP](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +[This guide from The Linux Documentation Project (TDLP)](https://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to search for and review the specific documentation for your operating system or distribution and your firewall software. @@ -370,8 +404,8 @@ sudo ufw allow from <client_ip_address> to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss WARNING: -Engineering support for NFS for Git repositories is deprecated. Read the -[Gitaly and NFS deprecation notice](gitaly/index.md#nfs-deprecation-notice). +Engineering support for NFS for Git repositories is deprecated. Read about +[moving beyond NFS](gitaly/index.md#moving-beyond-nfs). Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. For example, we have seen: diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index e30ad4d8248..8aa5af4c2bf 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -104,12 +104,12 @@ In the case of lookup failures (which are common), the `authorized_keys` file is still scanned. So Git SSH performance would still be slow for many users as long as a large file exists. -To disable any more writes to the `authorized_keys` file: +To disable writes to the `authorized_keys` file: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Network**. 1. Expand **Performance optimization**. -1. Clear the **Write to "authorized_keys" file** checkbox. +1. Clear the **Use authorized_keys file to authenticate SSH keys** checkbox. 1. Select **Save changes**. Again, confirm that SSH is working by removing your user's SSH key in the UI, @@ -123,10 +123,14 @@ or for asking users to re-add their keys. This is a brief overview. Please refer to the above instructions for more context. -1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file) -1. Enable writes to the `authorized_keys` file in Application Settings +1. [Rebuild the `authorized_keys` file](../raketasks/maintenance.md#rebuild-authorized_keys-file). +1. Enable writes to the `authorized_keys` file. + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Settings > Network**. + 1. Expand **Performance optimization**. + 1. Select the **Use authorized_keys file to authenticate SSH keys** checkbox. 1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. -1. Reload `sshd`: `sudo service sshd reload` +1. Reload `sshd`: `sudo service sshd reload`. ## Compiling a custom version of OpenSSH for CentOS 6 diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 61a9ec8e7d4..8aeaadc17e9 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -27,7 +27,7 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate to Gitaly Cluster](../gitaly/index.md#migrate-to-gitaly-cluster). +- [Migrating to Gitaly Cluster](../gitaly/index.md#migrating-to-gitaly-cluster). ### Move Repositories diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 814e742b026..77dc4eb180b 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -21,8 +21,7 @@ upload the new keys to GitLab. WARNING: OpenSSH version 6.9+ is required because that version introduced the `AuthorizedPrincipalsCommand` configuration option. If -using CentOS 6, you can [follow these -instructions](fast_ssh_key_lookup.html#compiling-a-custom-version-of-openssh-for-centos-6) +using CentOS 6, you can [follow these instructions](fast_ssh_key_lookup.md#compiling-a-custom-version-of-openssh-for-centos-6) to compile an up-to-date version. ## Why use OpenSSH certificates? @@ -132,8 +131,8 @@ requirement for it, we effectively only care about the "key ID" being correct. Once that's extracted GitLab enforces its own ACLs for that user (for example, what projects the user can access). -So it's OK to e.g. be overly generous in what you accept, since if the -user e.g. has no access to GitLab at all it just errors out with a +It's therefore fine to be overly generous in what you accept. For example, if the user has no access +to GitLab, an error is produced with a message about an invalid user. message about this being an invalid user. ## Interaction with the `authorized_keys` file diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 45bea065995..95d6135c28c 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package defaults +# Package defaults **(FREE SELF)** Unless configuration is specified in the `/etc/gitlab/gitlab.rb` file, the package will assume the defaults as noted below. @@ -14,42 +14,42 @@ the package will assume the defaults as noted below. See the table below for the list of ports that the Omnibus GitLab assigns by default: -| Component | On by default | Communicates via | Alternative | Connection port | -| :----------------------------------------------------: | :------------: | :--------------: | :---------: | :------------------------------------: | -| <a name="gitlab-rails"></a> GitLab Rails | Yes | Port | X | 80 or 443 | -| <a name="gitlab-shell"></a> GitLab Shell | Yes | Port | X | 22 | -| <a name="postgresql"></a> PostgreSQL | Yes | Socket | Port (5432) | X | -| <a name="redis"></a> Redis | Yes | Socket | Port (6379) | X | -| <a name="puma"></a> Puma | Yes | Socket | Port (8080) | X | -| <a name="gitlab-workhorse"></a> GitLab Workhorse | Yes | Socket | Port (8181) | X | -| <a name="nginx-status"></a> NGINX status | Yes | Port | X | 8060 | -| <a name="prometheus"></a> Prometheus | Yes | Port | X | 9090 | -| <a name="node-exporter"></a> Node exporter | Yes | Port | X | 9100 | -| <a name="redis-exporter"></a> Redis exporter | Yes | Port | X | 9121 | -| <a name="postgres-exporter"></a> PostgreSQL exporter | Yes | Port | X | 9187 | -| <a name="pgbouncer-exporter"></a> PgBouncer exporter | No | Port | X | 9188 | -| <a name="gitlab-exporter"></a> GitLab Exporter | Yes | Port | X | 9168 | -| <a name="sidekiq-exporter"></a> Sidekiq exporter | Yes | Port | X | 8082 | -| <a name="puma-exporter"></a> Puma exporter | No | Port | X | 8083 | -| <a name="geo-postgresql"></a> Geo PostgreSQL | No | Socket | Port (5431) | X | -| <a name="redis-sentinel"></a> Redis Sentinel | No | Port | X | 26379 | -| <a name="incoming-email"></a> Incoming email | No | Port | X | 143 | -| <a name="elasticsearch"></a> Elastic search | No | Port | X | 9200 | -| <a name="gitlab-pages"></a> GitLab Pages | No | Port | X | 80 or 443 | -| <a name="gitlab-registry-web"></a> GitLab Registry | No* | Port | X | 80, 443 or 5050 | -| <a name="gitlab-registry"></a> GitLab Registry | No | Port | X | 5000 | -| <a name="ldap"></a> LDAP | No | Port | X | Depends on the component configuration | -| <a name="kerberos"></a> Kerberos | No | Port | X | 8443 or 8088 | -| <a name="omniauth"></a> OmniAuth | Yes | Port | X | Depends on the component configuration | -| <a name="smtp"></a> SMTP | No | Port | X | 465 | -| <a name="remote-syslog"></a> Remote syslog | No | Port | X | 514 | -| <a name="mattermost"></a> Mattermost | No | Port | X | 8065 | -| <a name="mattermost-web"></a> Mattermost | No | Port | X | 80 or 443 | -| <a name="pgbouncer"></a> PgBouncer | No | Port | X | 6432 | -| <a name="consul"></a> Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | -| <a name="patroni"></a> Patroni | No | Port | X | 8008 | -| <a name="gitlab-kas"></a> GitLab KAS | No | Port | X | 8150 | -| <a name="gitaly"></a> Gitaly | No | Port | X | 8075 | +| Component | On by default | Communicates via | Alternative | Connection port | +|:-------------------:|:-------------:|:----------------:|:-----------:|:------------------------------------------:| +| GitLab Rails | Yes | Port | X | 80 or 443 | +| GitLab Shell | Yes | Port | X | 22 | +| PostgreSQL | Yes | Socket | Port (5432) | X | +| Redis | Yes | Socket | Port (6379) | X | +| Puma | Yes | Socket | Port (8080) | X | +| GitLab Workhorse | Yes | Socket | Port (8181) | X | +| NGINX status | Yes | Port | X | 8060 | +| Prometheus | Yes | Port | X | 9090 | +| Node exporter | Yes | Port | X | 9100 | +| Redis exporter | Yes | Port | X | 9121 | +| PostgreSQL exporter | Yes | Port | X | 9187 | +| PgBouncer exporter | No | Port | X | 9188 | +| GitLab Exporter | Yes | Port | X | 9168 | +| Sidekiq exporter | Yes | Port | X | 8082 | +| Puma exporter | No | Port | X | 8083 | +| Geo PostgreSQL | No | Socket | Port (5431) | X | +| Redis Sentinel | No | Port | X | 26379 | +| Incoming email | No | Port | X | 143 | +| Elastic search | No | Port | X | 9200 | +| GitLab Pages | No | Port | X | 80 or 443 | +| GitLab Registry | No* | Port | X | 80, 443 or 5050 | +| GitLab Registry | No | Port | X | 5000 | +| LDAP | No | Port | X | Depends on the component configuration | +| Kerberos | No | Port | X | 8443 or 8088 | +| OmniAuth | Yes | Port | X | Depends on the component configuration | +| SMTP | No | Port | X | 465 | +| Remote syslog | No | Port | X | 514 | +| Mattermost | No | Port | X | 8065 | +| Mattermost | No | Port | X | 80 or 443 | +| PgBouncer | No | Port | X | 6432 | +| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | +| Patroni | No | Port | X | 8008 | +| GitLab KAS | No | Port | X | 8150 | +| Gitaly | No | Port | X | 8075 | Legend: @@ -59,7 +59,7 @@ Legend: - `Alternative` - If it is possible to configure the component to use different type of communication. The type is listed with default port used in that case. - `Connection port` - Port on which the component communicates. -GitLab also expects a filesystem to be ready for the storage of Git repositories +GitLab also expects a file system to be ready for the storage of Git repositories and various other files. Note that if you are using NFS (Network File System), files will be carried diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md index 251dbe1e20e..35b333241b2 100644 --- a/doc/administration/package_information/deprecated_os.md +++ b/doc/administration/package_information/deprecated_os.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# OS Versions that are no longer supported +# OS Versions that are no longer supported **(FREE SELF)** GitLab provides omnibus packages for operating systems only until their EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index cc16661442a..80ce72d54f5 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Deprecation policy +# Deprecation policy **(FREE SELF)** The Omnibus GitLab packages come with number of different libraries and services which offers users plethora of configuration options. @@ -60,7 +60,7 @@ We should aim to not remove sensitive configuration in the *next major* release See the table below for some examples: -| Config. type | Deprecation announced | Final minor release | Remove | +| Configuration type | Deprecation announced | Final minor release | Remove | | -------- | -------- | -------- | -------- | | Sensitive | 10.1.0 | 10.9.0 | 11.0.0 | | Sensitive | 10.7.0 | 10.9.0 | 12.0.0 | @@ -90,6 +90,6 @@ the feature will continue working the same way as if you had `gitlab_rails['bett However, setting the old version of configuration will print out a deprecation notice at the end of installation/upgrade/reconfigure run. -With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid config. +With GitLab 11, `gitlab_rails['configuration'] = true` will no longer work and you will have to manually change the configuration in `/etc/gitlab/gitlab.rb` to the new valid configuration. **Note** If this configuration option is sensitive and can put integrity of the installation or data in danger, installation/upgrade will be aborted. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index e18fb621b89..12f3274ecab 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package information +# Package information **(FREE SELF)** The Omnibus GitLab package is bundled with all dependencies required for GitLab to function correctly. More details can be found @@ -15,10 +15,10 @@ at [bundling dependencies document](omnibus_packages.md). The released package versions are in the format `MAJOR.MINOR.PATCH-EDITION.OMNIBUS_RELEASE` | Component | Meaning | Example | -| --------- | ------- | ------- | -| MAJOR.MINOR.PATCH | The GitLab version this corresponds to | 13.3.0 | -| EDITION | The edition of GitLab this corresponds to | ee | -| OMNIBUS_RELEASE | The omnibus release. Usually, this will be 0. This will be incremented if we need to build a new package without changing the GitLab version. | 0 | +|-------------------|---------|---------| +| MAJOR.MINOR.PATCH | The GitLab version this corresponds to. | 13.3.0 | +| EDITION | The edition of GitLab this corresponds to. | ee | +| OMNIBUS_RELEASE | The Omnibus GitLab release. Usually, this will be 0. This is incremented if we need to build a new package without changing the GitLab version. | 0 | ## Licenses @@ -26,23 +26,22 @@ See [licensing](licensing.md) ## Defaults -The Omnibus GitLab package requires various configuration to get the -components in working order. -If the configuration is not provided, the package will use the default -values assumed in the package. +The Omnibus GitLab package requires various configuration to get the components +in working order. If the configuration is not provided, the package will use +the default values assumed in the package. These defaults are noted in the package [defaults document](defaults.md). ## Checking the versions of bundled software -Once the Omnibus GitLab package is installed, all versions of the bundled +After the Omnibus GitLab package is installed, all versions of the bundled libraries are located in `/opt/gitlab/version-manifest.txt`. If you don't have the package installed, you can always check the Omnibus GitLab [source repository](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master), specifically the -[config directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). +[configuration directory](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/config). -For example, if you take a look at the `8-6-stable` branch, you can conclude that +For example, if you examine the `8-6-stable` branch, you can conclude that 8.6 packages were running [Ruby 2.1.8](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-6-stable/config/projects/gitlab.rb#L48). Or, that 8.5 packages were bundled with [NGINX 1.9.0](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-5-stable/config/software/nginx.rb#L20). @@ -70,11 +69,10 @@ To view a diff between your configuration file and the latest version, run: sudo gitlab-ctl diff-config ``` -_**Note:** This command is available from GitLab 8.17_ - -**Important:** If you are copy-pasting the output of this command into your -`/etc/gitlab/gitlab.rb` configuration file, make sure to omit leading `+` and `-` -on each line. +WARNING: +If you are pasting the output of this command into your +`/etc/gitlab/gitlab.rb` configuration file, omit any leading `+` and `-` +characters on each line. ## Init system detection diff --git a/doc/administration/package_information/licensing.md b/doc/administration/package_information/licensing.md index 8557a94bf93..02358c66993 100644 --- a/doc/administration/package_information/licensing.md +++ b/doc/administration/package_information/licensing.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package Licensing +# Package Licensing **(FREE SELF)** ## License diff --git a/doc/administration/package_information/omnibus_packages.md b/doc/administration/package_information/omnibus_packages.md index aa73534fc55..115d6c394ad 100644 --- a/doc/administration/package_information/omnibus_packages.md +++ b/doc/administration/package_information/omnibus_packages.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Omnibus based packages and images +# Omnibus based packages and images **(FREE SELF)** Below you can find some basic information on why GitLab provides packages and a Docker image that come with bundled dependencies. diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 89da4864872..252e0cad76d 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# PostgreSQL versions shipped with Omnibus GitLab +# PostgreSQL versions shipped with Omnibus GitLab **(FREE SELF)** NOTE: This table lists only GitLab versions where a significant change happened in the diff --git a/doc/administration/package_information/signed_packages.md b/doc/administration/package_information/signed_packages.md index fb994809460..fb605f8d5be 100644 --- a/doc/administration/package_information/signed_packages.md +++ b/doc/administration/package_information/signed_packages.md @@ -4,9 +4,9 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Package Signatures +# Package Signatures **(FREE SELF)** -As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (e.g. `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) +As of the release of GitLab 9.5 on August 22, 2017, GitLab provides signed Omnibus GitLab packages for RPM and DEB based distributions. This means that all packages provided on <https://packages.gitlab.com> are signed, starting with `9.5.0`, and all future versions of supported branches (for example `9.3.x` and `9.4.x` after August 22, 2017). Any package version prior to August 22, 2017, will not be signed. Please pass the appropriate argument to your package manager. (Example: `yum --nogpgcheck`) Omnibus GitLab packages produced by GitLab are created via the [Omnibus](https://github.com/chef/omnibus) tool, for which GitLab has added DEB signing via `debsigs` in [our own fork](https://gitlab.com/gitlab-org/omnibus). This addition, combined with the existing functionality of RPM signing, allows GitLab to provide signed packages for all supported distributions using DEB or RPM. diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 1067474c8b4..7e711bb5740 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -119,6 +119,11 @@ GitLab from source respectively. Ensure you choose a port different than the one that Registry listens to (`5000` by default), otherwise conflicts occur. +NOTE: +Host and container firewall rules must be configured to allow traffic in through the port listed +under the `registry_external_url` line, rather than the port listed under +`gitlab_rails['registry_port']` (default `5000`). + **Omnibus GitLab installations** 1. Your `/etc/gitlab/gitlab.rb` should contain the Registry URL as well as the @@ -151,6 +156,19 @@ otherwise conflicts occur. If your certificate provider provides the CA Bundle certificates, append them to the TLS certificate file. +An administrator may want the container registry listening on an arbitrary port such as `5678`. +However, the registry and application server are behind an AWS application load balancer that only +listens on ports `80` and `443`. The admin may simply remove the port number for +`registry_external_url`, so HTTP or HTTPS is assumed. Then, the rules apply that map the load +balancer to the registry from ports `80` or `443` to the arbitrary port. This is important if users +rely on the `docker login` example in the container registry. Here's an example: + +```ruby +registry_external_url 'https://registry-gitlab.example.com' +registry_nginx['redirect_http_to_https'] = true +registry_nginx['listen_port'] = 5678 +``` + **Installations from source** 1. Open `/home/git/gitlab/config/gitlab.yml`, find the `registry` entry and diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 32e7e191011..a394a32fc18 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -29,8 +29,8 @@ To enable the dependency proxy feature: gitlab_rails['dependency_proxy_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. -1. Enable the [Puma web server](https://docs.gitlab.com/omnibus/settings/puma.html). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Enable the [Puma web server](../operations/puma.md). **Helm chart installations** @@ -88,7 +88,7 @@ To change the local storage path: gitlab_rails['dependency_proxy_storage_path'] = "/mnt/dependency_proxy" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** @@ -145,7 +145,7 @@ This section describes the earlier configuration format. } ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** diff --git a/doc/administration/packages/img/gitlab-registry-architecture.png b/doc/administration/packages/img/gitlab-registry-architecture.png Binary files differindex 742678d5e11..d6e8f935ad2 100644 --- a/doc/administration/packages/img/gitlab-registry-architecture.png +++ b/doc/administration/packages/img/gitlab-registry-architecture.png diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 37473d35573..90f2d9127fe 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -68,7 +68,7 @@ To enable the Packages feature: gitlab_rails['packages_enabled'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Installations from source** @@ -80,7 +80,7 @@ To enable the Packages feature: enabled: true ``` -1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **Helm Chart installations** @@ -92,7 +92,7 @@ To enable the Packages feature: enabled: true ``` -1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations "How to reconfigure Helm GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#helm-chart-installations) for the changes to take effect. ## Rate limits diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 8b7af5ee170..8a0d3f552bf 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -7,12 +7,6 @@ description: 'Learn how to administer GitLab Pages.' # GitLab Pages administration **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80) in GitLab EE 8.3. -> - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173) in GitLab EE 8.5. -> - GitLab Pages [was ported](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14605) to Community Edition in GitLab 8.17. -> - Support for subgroup project's websites was -> [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30548) in GitLab 11.8. - GitLab Pages allows for hosting of static sites. It must be configured by an administrator. Separate [user documentation](../../user/project/pages/index.md) is available. @@ -23,10 +17,10 @@ GitLab from source, see ## Overview -GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a simple HTTP server +GitLab Pages makes use of the [GitLab Pages daemon](https://gitlab.com/gitlab-org/gitlab-pages), a basic HTTP server written in Go that can listen on an external IP address and provide support for custom domains and custom certificates. It supports dynamic certificates through -SNI and exposes pages using HTTP2 by default. +Server Name Indication (SNI) and exposes pages using HTTP2 by default. You are encouraged to read its [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md) to fully understand how it works. @@ -89,7 +83,7 @@ added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/-/issue ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -you need to add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the +add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext @@ -99,9 +93,9 @@ host that GitLab runs. For example, an entry would look like this: Where `example.io` is the domain GitLab Pages is served from, `192.0.2.1` is the IPv4 address of your GitLab instance, and `2001:db8::1` is the -IPv6 address. If you don't have IPv6, you can omit the AAAA record. +IPv6 address. If you don't have IPv6, you can omit the `AAAA` record. -#### Custom domains +#### DNS configuration for custom domains If support for custom domains is needed, the Pages root domain and its subdomains should point to the secondary IP (which is dedicated for the Pages daemon). `<namespace>.<pages root domain>` should @@ -131,7 +125,7 @@ Depending on your needs, you can set up GitLab Pages in 4 different ways. The following examples are listed from the easiest setup to the most advanced one. The absolute minimum requirement is to set up the wildcard DNS -since that is needed in all configurations. +because that is needed in all configurations. ### Wildcard domains @@ -143,7 +137,7 @@ since that is needed in all configurations. URL scheme: `http://<namespace>.example.io/<project_slug>` -This is the minimum setup that you can use Pages with. It is the base for all +The following is the minimum setup that you can use Pages with. It is the base for all other setups as described below. NGINX proxies all requests to the daemon. The Pages daemon doesn't listen to the outside world. @@ -180,8 +174,8 @@ outside world. pages_nginx['redirect_http_to_https'] = true ``` -1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` -then you'll need to also add the full paths as shown below: +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key`, +you must also add the full paths as shown below: ```ruby pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt" @@ -201,7 +195,7 @@ Multiple wildcards for one instance is not supported. Only one wildcard per inst Below is a table of all configuration settings known to Pages in Omnibus GitLab, and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, and take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -Most of these settings don't need to be configured manually unless you need more granular +Most of these settings don't have to be configured manually unless you need more granular control over how the Pages daemon runs and serves content in your environment. | Setting | Description | @@ -382,8 +376,6 @@ To enable it: ### Access control -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. - GitLab Pages access control can be configured per-project, and allows access to a Pages site to be controlled based on a user's membership to that project. @@ -444,7 +436,7 @@ You can enforce [Access Control](#access-control) for all GitLab Pages websites on your GitLab instance. By doing so, only logged-in users have access to them. This setting overrides Access Control set by users in individual projects. -This can be useful to preserve information published with Pages websites to the users +This can be helpful to restrict information published with Pages websites to the users of your instance only. To do that: @@ -491,7 +483,7 @@ For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https:/ > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7. WARNING: -These are advanced settings. The recommended default values are set inside GitLab Pages. You should +These instructions deal with some advanced settings of your GitLab instance. The recommended default values are set inside GitLab Pages. You should change these settings only if absolutely necessary. Use extreme caution. GitLab Pages can serve content from ZIP archives through object storage (an @@ -524,9 +516,6 @@ After an archive reaches `zip_cache_expiration`, it's marked as expired and remo ## Activate verbose logging for daemon -Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in -Omnibus GitLab 11.1. - Follow the steps below to configure verbose logging of GitLab Pages daemon. 1. By default the daemon only logs with `INFO` level. @@ -603,8 +592,6 @@ the below steps to do a no downtime transfer to a new storage location. ## Configure listener for reverse proxy requests -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in Omnibus GitLab 11.1. - Follow the steps below to configure the proxy listener of GitLab Pages. 1. By default the listener is configured to listen for requests on `localhost:8090`. @@ -775,7 +762,7 @@ WARNING: From [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) to [GitLab 13.12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages can either use `disk` or `gitlab` domain configuration source. -We highly advise you to use `gitlab` configuration source as it will make transition to newer versions easier. +We highly advise you to use `gitlab` configuration source as it makes transitions to newer versions easier. To explicitly enable API source: @@ -1019,14 +1006,13 @@ Starting from GitLab 13.12, this setting also disables the [legacy storage](#mig In GitLab 14.0 a number of breaking changes were introduced which may require some user intervention. The steps below describe the best way to migrate without causing any downtime for your GitLab instance. -If you run GitLab on a single server, then most likely the upgrade process to 14.0 will go smoothly for you -and you will not notice any problem after upgrading. +A GitLab instance running on a single server typically upgrades to 14.0 smoothly, and there should be minimal issues after the upgrade is complete. Regardless, we recommend everyone follow the migration steps to ensure a successful upgrade. If at any point you run into issues, consult the [troubleshooting section](#troubleshooting). -If your current GitLab version is lower than 13.12, then you first need to update to 13.12. +If your current GitLab version is lower than 13.12, then you must first update to 13.12. Updating directly to 14.0 is [not supported](../../update/index.md#upgrade-paths) -and may cause downtime for some web-sites hosted on GitLab Pages. Once you update to 13.12, +and may cause downtime for some web-sites hosted on GitLab Pages. After you update to 13.12, migrate GitLab Pages to prepare them for GitLab 14.0: 1. Set [`domain_config_source` to `gitlab`](#domain-source-configuration-before-140), which @@ -1077,7 +1063,7 @@ This issue is fixed in GitLab 14.3 and above, try upgrading GitLab first. GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like `/tmp/gitlab-pages-*`. -Within the jail, a bundle of trusted certificates is +In the jail, a bundle of trusted certificates is provided at `/etc/ssl/ca-bundle.pem`. It's [copied there](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/51) from `/opt/gitlab/embedded/ssl/certs/cacert.pem` @@ -1206,26 +1192,10 @@ To stop `systemd` from cleaning the Pages related content: sudo gitlab-ctl restart gitlab-pages ``` -### 404 error after transferring the project to a different group or user, or changing project path - -If you encounter a `404 Not Found` error a Pages site after transferring a project to -another group or user, or changing project path, you must trigger a domain configuration -update for Pages. To do so, write something in the `.update` file. The Pages daemon -monitors for changes to this file, and reloads the configuration when changes occur. - -Use this example to fix a `404 Not Found` error after transferring a project or changing -a project path with Pages: - -```shell -date > /var/opt/gitlab/gitlab-rails/shared/pages/.update -``` - -If you've customized the Pages storage path, adjust the command above to use your custom path. - ### 404 error after promoting a Geo secondary to a primary node -These are due to the Pages files not being among the -[supported data types](../geo/replication/datatypes.md#limitations-on-replicationverification). +Pages files are not among the +[supported data types](../geo/replication/datatypes.md#limitations-on-replicationverification) for replication in Geo. After a secondary node is promoted to a primary node, attempts to access a Pages site result in a `404 Not Found` error. It is possible to copy the subfolders and files in the [Pages path](#change-storage-path) to the new primary node to resolve this. @@ -1233,11 +1203,11 @@ For example, you can adapt the `rsync` strategy from the [moving repositories documentation](../operations/moving_repositories.md). Alternatively, run the CI pipelines of those projects that contain a `pages` job again. -## 404 or 500 error when accessing GitLab Pages in a Geo setup +### 404 or 500 error when accessing GitLab Pages in a Geo setup Pages sites are only available on the primary Geo site, while the codebase of the project is available on all sites. -If you try to access a Pages page on a secondary site, you will get a 404 or 500 HTTP code depending on the access control. +If you try to access a Pages page on a secondary site, a 404 or 500 HTTP code is returned depending on the access control. Read more which [features don't support Geo replication/verification](../geo/replication/datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 2e0820b69c9..dc569a81abf 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -23,10 +23,7 @@ replication and failover requires: - A minimum of three database nodes. - A minimum of three `Consul` server nodes. -- A minimum of one `pgbouncer` service node, but it's recommended to have one - per database node. - - An internal load balancer (TCP) is required when there is more than one - `pgbouncer` service node. +- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node. An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node.  @@ -35,40 +32,31 @@ sure you have redundant connectivity between all Database and GitLab instances to avoid the network becoming a single point of failure. NOTE: -As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with +As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is supported only with Patroni. See the [Patroni](#patroni) section for further details. Starting with GitLab 14.0, only PostgreSQL 12 is -shipped with Omnibus GitLab and thus Patroni becomes mandatory for replication and failover. +shipped with Omnibus GitLab, and thus Patroni becomes mandatory for replication and failover. ### Database node Each database node runs three services: -`PostgreSQL` - The database itself. - -`Patroni` - Communicates with other Patroni services in the cluster and handles -failover when issues with the leader server occurs. The failover procedure -consists of: - -- Selecting a new leader for the cluster. -- Promoting the new node to leader. -- Instructing remaining servers to follow the new leader node. - -`Consul` agent - To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. +- `PostgreSQL`: The database itself. +- `Patroni`: Communicates with other Patroni services in the cluster and handles failover when issues with the leader server occurs. The failover procedure consists of: + - Selecting a new leader for the cluster. + - Promoting the new node to leader. + - Instructing remaining servers to follow the new leader node. +- `Consul` agent: To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. ### Consul server node -The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes wait until such Consul leader is elected. +The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap; otherwise, database nodes wait until such Consul leader is elected. ### PgBouncer node Each PgBouncer node runs two services: -`PgBouncer` - The database connection pooler itself. - -`Consul` agent - Watches the status of the PostgreSQL service definition on the -Consul cluster. If that status changes, Consul runs a script which updates the -PgBouncer configuration to point to the new PostgreSQL leader node and reloads -the PgBouncer service. +- `PgBouncer`: The database connection pooler itself. +- `Consul` agent: Watches the status of the PostgreSQL service definition on the Consul cluster. If that status changes, Consul runs a script which updates the PgBouncer configuration to point to the new PostgreSQL leader node and reloads the PgBouncer service. ### Connection flow @@ -106,8 +94,7 @@ When using default setup, minimum configuration requires: - `CONSUL_USERNAME`. The default user for Omnibus GitLab is `gitlab-consul` - `CONSUL_DATABASE_PASSWORD`. Password for the database user. -- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. - Can be generated with: +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME @@ -118,8 +105,7 @@ When using default setup, minimum configuration requires: Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - - If you are using a different username, you have to specify it through the - `CONSUL_USERNAME` variable. +- If you are using a different username, you have to specify it through the `CONSUL_USERNAME` variable. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -129,10 +115,8 @@ Few notes on the service itself: When configuring PostgreSQL, we do the following: -- Set `max_replication_slots` to double the number of database nodes. - Patroni uses one extra slot per node when initiating the replication. -- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. - This prevents replication from using up all of the available database connections. +- Set `max_replication_slots` to double the number of database nodes. Patroni uses one extra slot per node when initiating the replication. +- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. This prevents replication from using up all of the available database connections. In this document we are assuming 3 database nodes, which makes this configuration: @@ -151,7 +135,7 @@ You need the following password information for the application's database user: - `POSTGRESQL_USERNAME`. The default user for Omnibus GitLab is `gitlab` - `POSTGRESQL_USER_PASSWORD`. The password for the database user - `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. - Can be generated with: + It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME @@ -170,8 +154,7 @@ When using a default setup, the minimum configuration requires: - `PGBOUNCER_USERNAME`. The default user for Omnibus GitLab is `pgbouncer` - `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. -- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. - Can be generated with: +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. It can be generated with: ```shell sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME @@ -181,8 +164,7 @@ When using a default setup, the minimum configuration requires: Few things to remember about the service itself: -- The service runs as the same system account as the database - - In the package, this is by default `gitlab-psql` +- The service runs as the same system account as the database. In the package, this is by default `gitlab-psql` - If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you need to specify this username. - Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text @@ -206,7 +188,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). -Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly +Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, or others are strictly controlled by Patroni. These configurations override the original settings that you make with the `postgresql[...]` configuration key. Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication. Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example, @@ -215,7 +197,7 @@ configuration key. NOTE: The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore -any replication setting of PostgreSQL (it is overwritten anyway). Then you can remove any `repmgr[...]` or +any replication setting of PostgreSQL (which is overwritten). Then, you can remove any `repmgr[...]` or repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. Here is an example: @@ -282,7 +264,7 @@ on each node for the changes to take effect. Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure) becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. -If you choose an arbitrary order you do not have any predetermined leader. +If you choose an arbitrary order, you do not have any predetermined leader. #### Enable Monitoring @@ -296,7 +278,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. # Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true - # Set the network addresses that the exporters will listen on + # Set the network addresses that the exporters must listen on node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' ``` @@ -340,9 +322,9 @@ patroni['tls_ca_file'] = '/path/to/ca.pem' When TLS is enabled, mutual authentication of the API server and client is possible for all endpoints, the extent of which depends on the `patroni['tls_client_mode']` attribute: -- `none` (default): the API will not check for any client certificates. -- `optional`: client certificates are required for all [unsafe](https://patroni.readthedocs.io/en/latest/security.html#protecting-the-rest-api) API calls. -- `required`: client certificates are required for all API calls. +- `none` (default): The API does not check for any client certificates. +- `optional`: Client certificates are required for all [unsafe](https://patroni.readthedocs.io/en/latest/security.html#protecting-the-rest-api) API calls. +- `required`: Client certificates are required for all API calls. The client certificates are verified against the CA certificate that is specified with the `patroni['tls_ca_file']` attribute. Therefore, this attribute is required for mutual TLS authentication. You also need to specify PEM-formatted client certificate and private key files. @@ -450,9 +432,9 @@ authentication mode (`patroni['tls_client_mode']`), must each have the same valu #### Configure the internal load balancer -If you're running more than one PgBouncer node as recommended, then you need to set up a TCP internal load balancer to serve each correctly. This can be accomplished with any reputable TCP load balancer. +If you're running more than one PgBouncer node as recommended, you must set up a TCP internal load balancer to serve each correctly. This can be accomplished with any reputable TCP load balancer. -As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): +As an example, here's how you could do it with [HAProxy](https://www.haproxy.org/): ```plaintext global @@ -554,7 +536,7 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.33`: PostgreSQL 3 - `10.6.0.41`: GitLab application -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. +All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. @@ -675,7 +657,7 @@ This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer s It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. -In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. +In this example, we start with all servers on the same 10.6.0.0/16 private network range; they can connect to each freely other on those addresses. Here is a list and description of each machine and the assigned IP: @@ -684,7 +666,7 @@ Here is a list and description of each machine and the assigned IP: - `10.6.0.23`: PostgreSQL 3 - `10.6.0.31`: GitLab application -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. +All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes. The `external_url` for GitLab is `http://gitlab.example.com` @@ -787,7 +769,7 @@ Patroni is an opinionated solution for PostgreSQL high-availability. It takes th The fundamental [architecture](#example-recommended-setup-manual-steps) (mentioned above) does not change for Patroni. You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election propagates to the Patroni cluster as well. -Patroni monitors the cluster and handles any failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. +Patroni monitors the cluster and handles any failover. When the primary node fails, it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. With Patroni, the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures and starts PostgreSQL which it communicates with directly over a Unix socket. This means that if the Consul cluster is not functional or does not have a leader, Patroni and by extension PostgreSQL does not start. Patroni also exposes a REST API which can be accessed via its [default port](../package_information/defaults.md) on each node. @@ -847,7 +829,7 @@ Investigate further if: - `reply_time` is not current. The `lsn` fields relate to which write-ahead-log segments have been replicated. -Run the following on the leader to find out the current LSN: +Run the following on the leader to find out the current Log Sequence Number (LSN): ```shell echo 'SELECT pg_current_wal_lsn();' | gitlab-psql @@ -918,7 +900,7 @@ patroni['remove_data_directory_on_diverged_timelines'] = false |-|-| |`use_pg_rewind`|Try running `pg_rewind` on the former cluster leader before it rejoins the database cluster.| |`remove_data_directory_on_rewind_failure`|If `pg_rewind` fails, remove the local PostgreSQL data directory and re-replicate from the current cluster leader.| -|`remove_data_directory_on_diverged_timelines`|If `pg_rewind` cannot be used and the former leader's timeline has diverged from the current one, then delete the local data directory and re-replicate from the current cluster leader.| +|`remove_data_directory_on_diverged_timelines`|If `pg_rewind` cannot be used and the former leader's timeline has diverged from the current one, delete the local data directory and re-replicate from the current cluster leader.| ### Database authorization for Patroni @@ -936,7 +918,7 @@ You can use `gitlab-ctl patroni members` to check the status of the cluster memb is the primary or a replica. When Patroni is enabled, it exclusively controls PostgreSQL's startup, -shutdown, and restart. This means, to shut down PostgreSQL on a certain node you must shutdown Patroni on the same node with: +shutdown, and restart. This means, to shut down PostgreSQL on a certain node, you must shutdown Patroni on the same node with: ```shell sudo gitlab-ctl stop patroni @@ -974,7 +956,7 @@ When a Geo secondary site is replicating from a primary site that uses `Patroni` sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> ``` -Otherwise, the replication will not happen, even if the original node gets re-added as a follower node. This re-syncs your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. +Otherwise, the replication does not happen, even if the original node gets re-added as a follower node. This re-syncs your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. ### Recovering the Patroni cluster @@ -1097,7 +1079,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: sudo gitlab-ctl pg-upgrade -V 12 ``` -1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader: +1. Check the status of the leader and cluster. You can proceed only if you have a healthy leader: ```shell gitlab-ctl patroni check-leader @@ -1192,7 +1174,7 @@ If replication is not occurring, it may be necessary to reinitialize a replica. WARNING: This is a destructive process and may lead the cluster into a bad state. Make sure that you have a healthy backup before running this process. -As a last resort, if your Patroni cluster is in an unknown/bad state and no node can start, you can +As a last resort, if your Patroni cluster is in an unknown or bad state and no node can start, you can reset the Patroni state in Consul completely, resulting in a reinitialized Patroni cluster when the first Patroni node starts. @@ -1248,7 +1230,7 @@ To fix the problem, ensure the loopback interface is included in the CIDR addres 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Check that [all the replicas are synchronized](#check-replication-status) -### Errors in Patroni logs: the requested start point is ahead of the WAL flush position +### Errors in Patroni logs: the requested start point is ahead of the Write Ahead Log (WAL) flush position This error indicates that the database is not replicating: diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 7cd7ecc26f7..1d60b8c6f50 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -14,7 +14,8 @@ Even though Git is very resilient and tries to prevent data integrity issues, there are times when things go wrong. The following Rake tasks intend to help GitLab administrators diagnose problem repositories so they can be fixed. -There are 3 things that are checked to determine integrity. +These Rake tasks use three different methods to determine the integrity of Git +repositories. 1. Git repository file system check ([`git fsck`](https://git-scm.com/docs/git-fsck)). This step verifies the connectivity and validity of objects in the repository. @@ -37,7 +38,7 @@ exactly which repositories are causing the trouble. ### Check project code repositories This task loops through the project code repositories and runs the integrity check -described previously. If a project uses a pool repository, that will also be checked. +described previously. If a project uses a pool repository, that is also checked. Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). **Omnibus Installation** @@ -67,7 +68,7 @@ source repository. This task loops through all repositories on the GitLab server and outputs checksums in the format `<PROJECT ID>,<CHECKSUM>`. -- If a repository doesn't exist, the project ID will have a blank checksum. +- If a repository doesn't exist, the project ID is a blank checksum. - If a repository exists but is empty, the output checksum is `0000000000000000000000000000000000000000`. - Projects which don't exist are skipped. @@ -85,9 +86,9 @@ sudo -u git -H bundle exec rake gitlab:git:checksum_projects RAILS_ENV=productio For example, if: -- Project with ID#2 doesn't exist, it will be skipped. -- Project with ID#4 doesn't have a repository, its checksum will be blank. -- Project with ID#5 has an empty repository, its checksum will be `0000000000000000000000000000000000000000`. +- Project with ID#2 doesn't exist, it is skipped. +- Project with ID#4 doesn't have a repository, its checksum is blank. +- Project with ID#5 has an empty repository, its checksum is `0000000000000000000000000000000000000000`. The output would then look something like: @@ -105,7 +106,7 @@ Optionally, specific project IDs can be checksummed by setting an environment variable `CHECKSUM_PROJECT_IDS` with a list of comma-separated integers, for example: ```shell -CHECKSUM_PROJECT_IDS="1,3" sudo gitlab-rake gitlab:git:checksum_projects +sudo CHECKSUM_PROJECT_IDS="1,3" gitlab-rake gitlab:git:checksum_projects ``` ## Uploaded files integrity @@ -115,7 +116,7 @@ These integrity checks can detect missing files. Additionally, for locally stored files, checksums are generated and stored in the database upon upload, and these checks verify them against current files. -Currently, integrity checks are supported for the following types of file: +Integrity checks are supported for the following types of file: - CI artifacts (Available from version 10.7.0) - LFS objects (Available from version 10.6.0) @@ -206,7 +207,7 @@ above. ### Dangling objects -The `gitlab:git:fsck` task can find dangling objects such as: +The `gitlab-rake gitlab:git:fsck` task can find dangling objects such as: ```plaintext dangling blob a12... diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 5ddab999efe..d770361864e 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -115,7 +115,7 @@ component servers like [Gitaly](../gitaly/configure_gitaly.md#run-gitaly-on-its- You may also have a look at our troubleshooting guides for: - [GitLab](../index.md#troubleshooting) -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html#troubleshooting) +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) To run `gitlab:check`, run: @@ -247,7 +247,7 @@ have been corrupted, you should reinstall the omnibus package. Sometimes you need to know if your GitLab installation can connect to a TCP service on another machine (for example a PostgreSQL or web server) -in order to troubleshoot proxy issues. +to troubleshoot proxy issues. A Rake task is included to help you with this. **Omnibus Installation** @@ -334,13 +334,13 @@ This is an experimental feature that isn't enabled by default. It requires Postg Database indexes can be rebuilt regularly to reclaim space and maintain healthy levels of index bloat over time. -In order to rebuild the two indexes with the highest estimated bloat, use the following Rake task: +To rebuild the two indexes with the highest estimated bloat, use the following Rake task: ```shell sudo gitlab-rake gitlab:db:reindex ``` -In order to target a specific index, use the following Rake task: +To target a specific index, use the following Rake task: ```shell sudo gitlab-rake gitlab:db:reindex['public.a_specific_index'] @@ -352,7 +352,7 @@ The following index types are not supported: 1. Partitioned indexes 1. Expression indexes -Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables in order to enable annotations: +Optionally, this Rake task sends annotations to a Grafana (4.6 or later) endpoint. Use the following custom environment variables to enable annotations: 1. `GRAFANA_API_URL` - Grafana's base URL, for example `http://some-host:3000`. 1. `GRAFANA_API_KEY` - Grafana API key with at least `Editor role`. diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index cf45d3e15cf..2fbcb2a62e7 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -66,7 +66,7 @@ sudo gitlab-ctl start puma If you want to allow users to use the GitLab UI, then you'll need to ensure that the database is read-only: -1. Take a [GitLab backup](../raketasks/backup_restore.md#back-up-gitlab) +1. Take a [GitLab backup](../raketasks/backup_restore.md) in case things don't go as expected. 1. Enter PostgreSQL on the console as an administrator user: diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 48db734b1af..db652a80780 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -647,6 +647,7 @@ persistence classes. | `shared_state` | Store session-related and other persistent data. | | `actioncable` | Pub/Sub queue backend for ActionCable. | | `trace_chunks` | Store [CI trace chunks](../job_logs.md#enable-or-disable-incremental-logging) data. | +| `rate_limiting` | Store [rate limiting](../../user/admin_area/settings/user_and_ip_rate_limits.md) state. | To make this work with Sentinel: @@ -659,6 +660,7 @@ To make this work with Sentinel: gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL gitlab_rails['redis_actioncable_instance'] = REDIS_ACTIONCABLE_URL gitlab_rails['redis_trace_chunks_instance'] = REDIS_TRACE_CHUNKS_URL + gitlab_rails['redis_rate_limiting_instance'] = REDIS_RATE_LIMITING_URL # Configure the Sentinels gitlab_rails['redis_cache_sentinels'] = [ @@ -681,6 +683,10 @@ To make this work with Sentinel: { host: TRACE_CHUNKS_SENTINEL_HOST, port: 26379 }, { host: TRACE_CHUNKS_SENTINEL_HOST2, port: 26379 } ] + gitlab_rails['redis_rate_limiting_sentinels'] = [ + { host: RATE_LIMITING_SENTINEL_HOST, port: 26379 }, + { host: RATE_LIMITING_SENTINEL_HOST2, port: 26379 } + ] ``` - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 42482475d26..8a3b88780a1 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -44,8 +44,7 @@ Omnibus GitLab: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab - application servers](#set-up-the-gitlab-rails-application-instance). + Redis password. These will be necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). [Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 0fd597e6a2d..e731085b0ce 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest 10k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 10,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -792,15 +778,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -812,7 +792,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -830,8 +810,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -843,8 +827,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -878,10 +873,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -889,11 +880,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -905,15 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -949,15 +948,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -967,133 +957,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1103,8 +975,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,8 +992,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1141,13 +1028,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1155,8 +1038,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1168,16 +1055,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1202,15 +1093,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1220,129 +1102,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1509,7 +1277,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1836,7 +1604,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1851,36 +1619,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2032,30 +1804,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2127,7 +1899,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2271,7 +2043,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2295,6 +2067,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2341,7 +2116,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2355,7 +2130,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> @@ -2415,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2430,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2474,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index ea40e150e58..ae832c2226f 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -18,6 +18,8 @@ many organizations. > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you +> can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **[Latest 1k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** @@ -58,3 +60,12 @@ Elasticsearch cluster design and requirements are dependent on your specific data. For recommended best practices about how to set up your Elasticsearch cluster alongside your instance, read how to [choose the optimal cluster configuration](../../integration/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +## Cloud Native Hybrid reference architecture with Helm Charts + +Cloud Native Hybrid Reference Architecture is an alternative approach where select _stateless_ +components are deployed in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/), +and _stateful_ components are deployed in compute VMs with Omnibus. + +The [2k GitLab Cloud Native Hybrid](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (non HA) and [3k GitLab Cloud Native Hybrid](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (HA) reference architectures are the smallest we recommend in Kubernetes. +For environments that need to serve less users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f500434d75b..267f81efd91 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest 25k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,9 @@ To set up GitLab and its components to accommodate up to 25,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non + Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +186,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -794,15 +781,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -814,7 +795,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -832,10 +813,14 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role'] + # Specify server role as 'redis_sentinel_role' 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. @@ -845,8 +830,18 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 + # Port of primary Redis server for Sentinel, uncomment to change to non default. + # Defaults to `6379`. + # redis['master_port'] = 6379 + # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node. + redis['master_name'] = 'gitlab-redis-cache' + + # The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -880,10 +875,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -891,14 +882,18 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = `0.0.0.0` + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. + # Set bind to '0.0.0.0' to listen on all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.6.0.52' @@ -907,16 +902,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.51' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - # Set the Redis Cache instance as an LRU # 90% of available RAM in MB redis['maxmemory'] = '13500mb' @@ -952,17 +950,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -970,134 +958,15 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: +### Configure the Redis Persistent cluster -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster - -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1107,8 +976,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1120,8 +993,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1145,13 +1029,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1160,7 +1040,11 @@ You can specify multiple roles, like sentinel and Redis, as: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1172,16 +1056,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1211,15 +1098,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1229,138 +1107,16 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - ## Configure Gitaly Cluster -[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended +fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1527,7 +1283,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1854,7 +1610,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1869,36 +1625,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2052,9 +1812,9 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] ## Second cluster that will host the queues, shared state, and actionable @@ -2063,19 +1823,19 @@ On each node perform the following: gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2147,7 +1907,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2289,9 +2049,9 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better -in larger setups as object storage is typically much more performant, reliable, -and scalable. +Object storage is also recommended over [NFS](#configure-nfs) and in general +it's better in larger setups as object storage is typically much more performant, +reliable, and scalable. GitLab has been tested on a number of object storage providers: @@ -2313,6 +2073,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2359,7 +2122,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2373,7 +2136,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2427,12 +2190,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2442,6 +2203,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2486,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 99dd29c3a83..23b15b563f7 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,6 +13,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest 2k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** @@ -302,8 +303,8 @@ further configuration steps. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the [GitLab - application server](#configure-gitlab-rails) later. + plain text password. These will be necessary when configuring the + [GitLab application server](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) are supported and can be added if needed. @@ -385,8 +386,8 @@ Omnibus: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab - application servers](#configure-gitlab-rails) later. + Redis password. These will be necessary when + [configuring the GitLab application servers](#configure-gitlab-rails) later. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -903,6 +904,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -964,7 +968,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index da36968f053..575dd22b729 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,6 +22,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest 3k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** @@ -33,8 +34,8 @@ For a full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | @@ -48,6 +49,7 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1065,6 +1067,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1230,7 +1236,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1559,7 +1565,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1574,6 +1580,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis redis['master_name'] = 'gitlab-redis' @@ -2011,6 +2021,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2071,7 +2084,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Supported modifications for lower user counts (HA) @@ -2150,8 +2163,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2161,6 +2174,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b071b48cbd9..be44f464e7e 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,6 +12,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest 50k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** @@ -22,18 +23,16 @@ full list of reference architectures, see | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State<sup>2</sup>| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -41,6 +40,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -82,11 +82,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +132,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +163,8 @@ To set up GitLab and its components to accommodate up to 50,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +185,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -802,15 +788,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -822,7 +802,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -840,8 +820,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server role as 'redis_master_role' with Sentinel and enable Consul agent + roles(['redis_sentinel_role', 'redis_master_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -853,8 +837,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -888,10 +883,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -899,11 +890,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent + roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -915,15 +910,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -952,25 +951,18 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus + node you configured and add or replace the file of the same name on this + server. If this is the first Omnibus node you are configuring then you + can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + to take effect. -1. Go through the steps again for all the other replica nodes, and + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -978,126 +970,7 @@ are supported and can be added if needed. </a> </div> -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster This is the section where we install and set up the new Redis Queues instances. @@ -1105,7 +978,7 @@ Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1115,8 +988,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1128,8 +1005,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1153,13 +1041,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). +#### Configure the replica Redis Persistent nodes -#### Configure the replica Redis Queues nodes - -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1167,8 +1051,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1180,16 +1068,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1215,144 +1107,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - -<div align="right"> - <a type="button" class="btn btn-default" href="#setup-components"> - Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> - </a> -</div> - -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -1365,6 +1120,10 @@ To configure the Sentinel Queues server: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1531,7 +1290,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1858,7 +1617,7 @@ To configure the Sidekiq nodes, on each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1873,36 +1632,40 @@ To configure the Sidekiq nodes, on each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly @@ -2063,30 +1826,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER>@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2158,7 +1921,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2300,7 +2063,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2324,6 +2087,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2370,7 +2136,7 @@ cluster alongside your instance, read how to </a> </div> -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2384,7 +2150,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2438,12 +2204,10 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State<sup>2</sup> | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2453,6 +2217,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -2497,11 +2262,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 4dfe628039a..a5526986be1 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,6 +19,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) +> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest 5k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** @@ -30,8 +31,8 @@ costly-to-operate environment by using the | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| @@ -45,6 +46,7 @@ costly-to-operate environment by using the 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: @@ -1056,6 +1058,10 @@ The following IPs will be used as an example: [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +NOTE: +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). +For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. + The recommended cluster setup includes the following components: - 3 Gitaly nodes: Replicated storage of Git repositories. @@ -1222,7 +1228,7 @@ the details of each Gitaly node that makes up the cluster. Each storage is also and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. -Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. The following IPs will be used as an example: @@ -1549,7 +1555,7 @@ To configure the Sidekiq nodes, one each one: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Open `/etc/gitlab/gitlab.rb` with your editor: +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: ```ruby # Avoid running unnecessary services on the Sidekiq server @@ -1564,6 +1570,10 @@ To configure the Sidekiq nodes, one each one: gitlab_exporter['enable'] = false nginx['enable'] = false + # External URL + ## This should match the URL of the external load balancer + external_url 'https://gitlab.example.com' + # Redis ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -2005,6 +2015,9 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based on what features you intend to use: @@ -2065,7 +2078,7 @@ unavailable from GitLab 15.0. No further enhancements are planned for this featu Read: -- The [Gitaly and NFS deprecation notice](../gitaly/index.md#nfs-deprecation-notice). +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) @@ -2120,8 +2133,8 @@ services where applicable): | PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Object storage<sup>4</sup> | n/a | n/a | n/a | @@ -2131,6 +2144,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reply_by_email.md b/doc/administration/reply_by_email.md index c249f48b768..055f40ead64 100644 --- a/doc/administration/reply_by_email.md +++ b/doc/administration/reply_by_email.md @@ -45,6 +45,8 @@ following headers, in this order: 1. `To` header 1. `References` header +1. `Delivered-To` header +1. `Envelope-To` header If it finds a reply key, it leaves your reply as a comment on the entity the notification was about (issue, merge request, commit...). diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index e7edfb9f338..ed50d0e7263 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -11,9 +11,10 @@ GitLab stores [repositories](../user/project/repository/index.md) on repository storage is either: - A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly to the directory where the repositories are stored. This method is - deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in - GitLab 14.0. +- A `path`, which points directly to the directory where the repositories are stored. GitLab + directly accessing a directory containing repositories + [is deprecated](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). + GitLab should be configured to access GitLab repositories though a `gitaly_address`. GitLab allows you to define multiple repository storages to distribute the storage load between several mount points. For example: @@ -173,4 +174,4 @@ information. ## Move repositories To move a repository to a different repository storage (for example, from `default` to `storage2`), use the -same process as [migrating to Gitaly Cluster](gitaly/index.md#migrate-to-gitaly-cluster). +same process as [migrating to Gitaly Cluster](gitaly/index.md#migrating-to-gitaly-cluster). diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index e753832f2c3..4aee88ed9cb 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -104,6 +104,16 @@ you want using steps 1 and 2 from the GitLab downloads page. You must also copy the `registry.key` file to each Sidekiq node. +1. Define the `external_url`. To maintain uniformity of links across nodes, the + `external_url` on the Sidekiq server should point to the external URL that users + will use to access GitLab. This will either be the `external_url` set on your + application server or the URL of a external load balancer which will route traffic + to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + ``` + 1. Run `gitlab-ctl reconfigure`. You will need to restart the Sidekiq nodes after an update has occurred and database @@ -194,6 +204,9 @@ gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] # Container Registry URL for cleanup jobs registry_external_url 'https://registry.example.com' gitlab_rails['registry_api_url'] = "https://registry.example.com" + +# External URL (this should match the URL used to access your GitLab instance) +external_url 'https://gitlab.example.com' ``` ## Further reading diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index ebc1723076b..8fdd87a5b2d 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -69,16 +69,16 @@ The key needs to be readable by the GitLab system user (`git` by default). The key needs to be readable by the GitLab system user (`git` by default). -### How to convert S/MIME PKCS#12 / PFX format to PEM encoding +### How to convert S/MIME PKCS #12 format to PEM encoding -Typically S/MIME certificates are handled in binary PKCS#12 format (`.pfx` or `.p12` -extensions), which contain the following in a single encrypted file: +Typically S/MIME certificates are handled in binary Public Key Cryptography Standards (PKCS) #12 format +(`.pfx` or `.p12` extensions), which contain the following in a single encrypted file: - Public certificate - Intermediate certificates (if any) - Private key -To export the required files in PEM encoding from the PKCS#12 file, the +To export the required files in PEM encoding from the PKCS #12 file, the `openssl` command can be used: ```shell diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index b6076c8ed26..f4339263d34 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -14,22 +14,22 @@ The global time zone configuration parameter can be changed in `config/gitlab.ym Uncomment and customize if you want to change the default time zone of the GitLab application. -## Viewing available timezones +## Viewing available time zones To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. NOTE: -This Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). +This Rake task does not list time zones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). ## Changing time zone in Omnibus installations -GitLab defaults its time zone to UTC. It has a global timezone configuration parameter in `/etc/gitlab/gitlab.rb`. +GitLab defaults its time zone to UTC. It has a global time zone configuration parameter in `/etc/gitlab/gitlab.rb`. -To obtain a list of timezones, log in to your GitLab application server and run a command that generates a list of timezones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. +To obtain a list of time zones, log in to your GitLab application server and run a command that generates a list of time zones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. -To update, add the timezone that best applies to your location. For example: +To update, add the time zone that best applies to your location. For example: ```ruby gitlab_rails['time_zone'] = 'America/New_York' @@ -48,8 +48,12 @@ gitlab-ctl restart > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/29669) in GitLab 14.1. -A user can set their time zone in their profile. If a user has not set their time zone, it defaults -to the time zone [configured at the instance level](#changing-your-time-zone). On GitLab.com, the -default time zone is UTC. +Users can set their time zone in their profile. On GitLab.com, the default time zone is UTC. + +New users do not have a default time zone in [GitLab 14.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/340795). New users must +explicitly set their time zone before it displays on their profile. + +In GitLab 14.3 and earlier, users with no configured time zone default to the time zone +[configured at the instance level](#changing-your-time-zone). For more information, see [Set your time zone](../user/profile/index.md#set-your-time-zone). diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index aea891b8a77..e00243aca0a 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -225,7 +225,7 @@ gitlab_rails['env'] = { ``` For source installations, set the environment variable. -Refer to [Puma Worker timeout](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). +Refer to [Puma Worker timeout](../operations/puma.md#worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 491db37d9da..109f451be5a 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -26,7 +26,7 @@ As GitLab changes, changes to the code are inevitable, and so some scripts may not work as they once used to. These are not kept up-to-date as these scripts/commands were added as they were found/needed. As mentioned above, we recommend running these scripts under the supervision of a -Support Engineer, who can also verify that they will continue to work as they +Support Engineer, who can also verify that they continue to work as they should and, if needed, update the script for the latest version of GitLab. ## Find specific methods for an object @@ -38,8 +38,6 @@ Array.methods.grep(/sing/) ## Find method source -Works for [non-instrumented methods](../../development/instrumentation.md#checking-instrumented-methods): - ```ruby instance_of_object.method(:foo).source_location @@ -187,7 +185,7 @@ Project.update_all(visibility_level: 0) ```ruby # -# This section will list all the projects which are pending deletion +# This section lists all the projects which are pending deletion # projects = Project.where(pending_delete: true) projects.each do |p| @@ -197,7 +195,7 @@ projects.each do |p| end # -# Assign a user (the root user will do) +# Assign a user (the root user does) # user = User.find_by_username('root') @@ -257,7 +255,7 @@ namespace = Namespace.find_by_full_path("<new_namespace>") ### For Removing webhooks that is getting timeout due to large webhook logs ```ruby -# ID will be the webhook_id +# ID is the webhook_id hook=WebHook.find(ID) WebHooks::DestroyService.new(current_user).execute(hook) @@ -399,7 +397,7 @@ projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") ### Recreate WARNING: -This is a destructive operation, the Wiki will be empty. +This is a destructive operation, the Wiki becomes empty. A Projects Wiki can be recreated by this command: @@ -476,13 +474,13 @@ Projects::ImportExport::ExportService.new(project, user).execute If the project you wish to export is available at `https://gitlab.example.com/baltig/pipeline-templates`, the value to use for `PROJECT_PATH` would be `baltig/pipeline-templates`. -If this all runs successfully, you will see output like the following before being returned to the Rails console prompt: +If this all runs successfully, you see an output like the following before being returned to the Rails console prompt: ```ruby => nil ``` -The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +The exported project is located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. ## Repository @@ -490,27 +488,29 @@ The exported project will be located within a `.tar.gz` file in `/var/opt/gitlab If it seems that a commit has gone "missing", search the sequence of pushes to a repository. [This StackOverflow article](https://stackoverflow.com/questions/13468027/the-mystery-of-the-missing-commit-across-merges) -describes how you can end up in this state without a force push. +describes how you can end up in this state without a force push. Another cause can be a misconfigured [server hook](../server_hooks.md) that changes a HEAD ref via a `git reset` operation. + +If you look at the output from the sample code below for the target branch, you +see a discontinuity in the from/to commits as you step through the output. The `commit_from` of each new push should equal the `commit_to` of the previous push. A break in that sequence indicates one or more commits have been "lost" from the repository history. -If you look at the output from the sample code below for the target branch, you will -see a discontinuity in the from/to commits as you step through the output. Each new -push should be "from" the "to" SHA of the previous push. When this discontinuity happens, -you will see two pushes with the same "from" SHA: +The following example checks the last 100 pushes and prints the `commit_from` and `commit_to` entries: ```ruby -p = Project.find_with_namespace('u/p') +p = Project.find_by_full_path('u/p') p.events.pushed_action.last(100).each do |e| - printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username) + printf "%-20.20s %8s...%8s (%s) +", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) end ``` -GitLab 9.5 and above: +Example output showing break in sequence at line 4: -```ruby -p = Project.find_by_full_path('u/p') -p.events.pushed_action.last(100).each do |e| - printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) -end +```plaintext +master f21b07713251e04575908149bdc8ac1f105aabc3...6bc56c1f46244792222f6c85b11606933af171de (root) +master 6bc56c1f46244792222f6c85b11606933af171de...132da6064f5d3453d445fd7cb452b148705bdc1b (root) +master 132da6064f5d3453d445fd7cb452b148705bdc1b...a62e1e693150a2e46ace0ce696cd4a52856dfa65 (root) +master 58b07b719a4b0039fec810efa52f479ba1b84756...f05321a5b5728bd8a89b7bf530aa44043c951dce (root) +master f05321a5b5728bd8a89b7bf530aa44043c951dce...7d02e575fd790e76a3284ee435368279a5eb3773 (root) ``` ## Mirrors @@ -558,7 +558,7 @@ end ```ruby u = User.new(username: 'test_user', email: 'test@example.com', name: 'Test User', password: 'password', password_confirmation: 'password') -u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user will recieve confirmation e-mail +u.skip_confirmation! # Use it only if you wish user to be automatically confirmed. If skipped, user receives confirmation e-mail u.save! ``` @@ -612,7 +612,7 @@ identifier = Analytics::UsageTrends::Measurement.identifiers[:billable_users] ```ruby users = User.where('id NOT IN (select distinct(user_id) from project_authorizations)') -# How many users will be removed? +# How many users are removed? users.count # If that count looks sane: @@ -726,6 +726,18 @@ group.require_two_factor_authentication=false group.save ``` +## Authentication + +### Re-enable standard web sign-in form + +Re-enable the standard username and password-based sign-in form if it was disabled as a [Sign-in restriction](../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). + +You can use this method when a configured external authentication provider (through SSO or an LDAP configuration) is facing an outage and direct sign-in access to GitLab is required. + +```ruby +Gitlab::CurrentSettings.update!(password_authentication_enabled_for_web: true) +``` + ## SCIM ### Fixing bad SCIM identities @@ -1061,7 +1073,7 @@ encrypted credentials to allow manual reentry: If `User OTP Secret Bad count:` is detected. For each user listed disable/enable two-factor authentication. -The following script will search in some of the tables for encrypted tokens that are +The following script searches in some of the tables for encrypted tokens that are causing decryption errors, and update or reset as needed: ```shell @@ -1123,7 +1135,7 @@ Geo::ProjectRegistry.sync_failed('repository') ### Resync repositories -#### Queue up all repositories for resync. Sidekiq will handle each sync +#### Queue up all repositories for resync. Sidekiq handles each sync ```ruby Geo::ProjectRegistry.update_all(resync_repository: true, resync_wiki: true) @@ -1170,10 +1182,10 @@ registry.replicator.send(:download) #### Verify package files on the secondary manually -This will iterate over all package files on the secondary, looking at the +This iterates over all package files on the secondary, looking at the `verification_checksum` stored in the database (which came from the primary) and then calculate this value on the secondary to check if they match. This -won't change anything in the UI: +does not change anything in the UI: ```ruby # Run on secondary @@ -1235,7 +1247,7 @@ Gitlab::UsageData.to_json ### Generate a fresh new Service Ping -This will also refresh the cached Service Ping displayed in the admin area +This also refreshes the cached Service Ping displayed in the Admin Area ```ruby Gitlab::UsageData.to_json(force_refresh: true) @@ -1269,7 +1281,7 @@ cluster = Clusters::Cluster.find_by(name: 'cluster_name') Delete cluster without associated resources: ```ruby -# Find an admin user +# Find users with the Administrator role user = User.find_by(username: 'admin_user') # Find the cluster with the ID @@ -1289,7 +1301,7 @@ Open the rails console (`gitlab rails c`) and run the following command to see a ApplicationSetting.last.attributes ``` -Among other attributes, in the output you will notice that all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), like: `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, `elasticsearch_pause_indexing`, and so on. +Among other attributes, the output contains all the settings available in the [Elasticsearch Integration page](../../integration/elasticsearch.md), such as `elasticsearch_indexing`, `elasticsearch_url`, `elasticsearch_replicas`, and `elasticsearch_pause_indexing`. #### Setting attributes diff --git a/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png b/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png Binary files differnew file mode 100644 index 00000000000..197cfa17da2 --- /dev/null +++ b/doc/administration/troubleshooting/img/database-query-dialog_v14_3.png diff --git a/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png b/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png Binary files differnew file mode 100644 index 00000000000..a63ebf9ecf2 --- /dev/null +++ b/doc/administration/troubleshooting/img/obtaining-a-session-cookie-for-request_v14_3.png diff --git a/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png b/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png Binary files differnew file mode 100644 index 00000000000..a19585d7a89 --- /dev/null +++ b/doc/administration/troubleshooting/img/paste-request-id-into-progress-bar_v14_3.png diff --git a/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png b/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png Binary files differnew file mode 100644 index 00000000000..b8314056c9b --- /dev/null +++ b/doc/administration/troubleshooting/img/select-request-id-from-request-selector-drop-down-menu_v14_3.png diff --git a/doc/administration/troubleshooting/img/view-pg-details_v14_3.png b/doc/administration/troubleshooting/img/view-pg-details_v14_3.png Binary files differnew file mode 100644 index 00000000000..1fe12462e4e --- /dev/null +++ b/doc/administration/troubleshooting/img/view-pg-details_v14_3.png diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 9eadbad171e..b66e88a8d60 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -14,7 +14,7 @@ having an issue with GitLab, you may want to check your [support options](https: first, before attempting to use this information. WARNING: -If you are administering GitLab you are expected to know these commands for your distribution +If you administer GitLab you are expected to know these commands for your distribution of choice. If you are a GitLab Support Engineer, consider this a cross-reference to translate `yum` -> `apt-get` and the like. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 80be30a6509..01a4c4af65b 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -45,7 +45,7 @@ following issues: ``` - The error `SSL certificate problem: unable to get local issuer certificate` - is displayed when setting up a [mirror](../../user/project/repository/repository_mirroring.md#repository-mirroring) + is displayed when setting up a [mirror](../../user/project/repository/mirror/index.md) from this GitLab instance. - `openssl` works when specifying the path to the certificate: @@ -115,7 +115,7 @@ and the restart the runner by running `gitlab-runner restart`. ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate -When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/repository_mirroring.md) +When configuring a local GitLab instance to [mirror a repository](../../user/project/repository/mirror/index.md) from a remote GitLab instance that uses a self-signed certificate, you may see the `SSL certificate problem: self signed certificate` error message in the user interface. diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index 3518f52e6f6..3bafbed4b3f 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -135,3 +135,69 @@ You can use the [performance bar](../monitoring/performance/performance_bar.md) To view the data, the correlation ID of the request must match the same session as the user viewing the performance bar. For API requests, this means that you must perform the request using the session cookie of the signed-in user. + +For example, if you want to view the database queries executed for the following API endpoint: + +```shell +https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1 +``` + +First, enable the **Developer Tools** panel. See [Getting the correlation ID in your browser](#getting-the-correlation-id-in-your-browser) for details on how to do this. + +After developer tools have been enabled, obtain a session cookie as follows: + +1. Visit <https://gitlab.com> while logged in. +1. (Optional) Select **Fetch/XHR** request filter in the **Developer Tools** panel. This step is described for Google Chrome developer tools and is not strictly necessary, it just makes it easier to find the correct request. +1. Select the `results?request_id=<some-request-id>` request on the left hand side. +1. The session cookie is displayed under the `Request Headers` section of the `Headers` panel. Right-click on the cookie value and select `Copy value`. + + + +You have the value of the session cookie copied to your clipboard, for example: + +```shell +experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow +``` + +Use the value of the session cookie to craft an API request by pasting it into a custom header of a `curl` request: + +```shell +$ curl --include "https://gitlab.com/api/v4/groups/2564205/projects?with_security_reports=true&page=1&per_page=1" \ +--header 'cookie: experimentation_subject_id=<subject-id>; _gitlab_session=<session-id>; event_filter=all; visitor_id=<visitor-id>; perf_bar_enabled=true; sidebar_collapsed=true; diff_view=inline; sast_entry_point_dismissed=true; auto_devops_settings_dismissed=true; cf_clearance=<cf-clearance>; collapsed_gutter=false; frequently_used_emojis=clap,thumbsup,rofl,tada,eyes,bow' + + date: Tue, 28 Sep 2021 03:55:33 GMT + content-type: application/json + ... + x-request-id: 01FGN8P881GF2E5J91JYA338Y3 + ... + [ + { + "id":27497069, + "description":"Analyzer for images used on live K8S containers based on Starboard" + }, + "container_registry_image_prefix":"registry.gitlab.com/gitlab-org/security-products/analyzers/cluster-image-scanning", + "..." + ] +``` + +The response contains the data from the API endpoint, and a `correlation_id` value, returned in the `x-request-id` header, as described in the [Identify the correlation ID for a request](#identify-the-correlation-id-for-a-request) section. + +You can then view the database details for this request: + +1. Paste the `x-request-id` value into the `request details` field of the [performance bar](../monitoring/performance/performance_bar.md) and press <kbd>Enter/Return</kbd>. This example uses the `x-request-id` value `01FGN8P881GF2E5J91JYA338Y3`, returned by the above response: + +  + +1. A new request is inserted into the `Request Selector` dropdown on the right-hand side of the Performance Bar. Select the new request to view the metrics of the API request: + +  + + <!-- vale gitlab.Substitutions = NO --> +1. Select the `pg` link in the Progress Bar to view the database queries executed by the API request: + +  + <!-- vale gitlab.Substitutions = YES --> + + The database query dialog is displayed: + +  diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 681ce87edb6..891c11afaf4 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -4,40 +4,56 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Modifying global user settings +# Modify global user settings **(FREE SELF)** GitLab administrators can modify user settings for the entire GitLab instance. -## Disallow users creating top-level groups +## Prevent users from creating top-level groups -By default, new users can create top-level groups. To disable this, modify the appropriate configuration file, -and then [reconfigure and restart GitLab](restart_gitlab.md). +By default, new users can create top-level groups. To disable your users' +ability to create top-level groups: -For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: +**Omnibus GitLab installations** -```ruby -gitlab_rails['gitlab_default_can_create_group'] = false -``` +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: -For source installations, uncomment the following line in `config/gitlab.yml`: + ```ruby + gitlab_rails['gitlab_default_can_create_group'] = false + ``` -```yaml -# default_can_create_group: false # default: true -``` +1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). -## Disallow users changing usernames +**Source installations** -By default, new users can change their usernames. To disable this, modify the appropriate configuration file, -and then [reconfigure and restart GitLab](restart_gitlab.md). +1. Edit `config/gitlab.yml` and uncomment the following line: -For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: + ```yaml + # default_can_create_group: false # default: true + ``` -```ruby -gitlab_rails['gitlab_username_changing_enabled'] = false -``` +1. [Restart GitLab](restart_gitlab.md#installations-from-source). -For source installations, uncomment the following line in `config/gitlab.yml`: +## Prevent users from changing their usernames -```yaml -# username_changing_enabled: false # default: true - User can change their username/namespace -``` +By default, new users can change their usernames. To disable your users' +ability to change their usernames: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['gitlab_username_changing_enabled'] = false + ``` + +1. [Reconfigure and restart GitLab](restart_gitlab.md#omnibus-installations). + +**Source installations** + +1. Edit `config/gitlab.yml` and uncomment the following line: + + ```yaml + # username_changing_enabled: false # default: true - User can change their username/namespace + ``` + +1. [Restart GitLab](restart_gitlab.md#installations-from-source). diff --git a/doc/api/README.md b/doc/api/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/api/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/api/access_requests.md b/doc/api/access_requests.md index 1634184a374..df830a1607d 100644 --- a/doc/api/access_requests.md +++ b/doc/api/access_requests.md @@ -2,13 +2,10 @@ stage: Manage group: Access info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api --- # Group and project access requests API **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18583) in GitLab 8.11. - ## Valid access levels The access levels are defined in the `Gitlab::Access` module, and the diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 8706b1e7e76..7dc3fd1db21 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -42,6 +42,7 @@ The following API resources are available in the project context: | [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | | [Feature Flags](feature_flags.md) | `/projects/:id/feature_flags` | | [Feature Flag User Lists](feature_flag_user_lists.md) | `/projects/:id/feature_flags_user_lists` | +| [Integrations](integrations.md) | `/projects/:id/integrations` | | [Invitations](invitations.md) | `/projects/:id/invitations` (also available for groups) | | [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | | [Issues Statistics](issues_statistics.md) | `/projects/:id/issues_statistics` (also available for groups and standalone) | @@ -82,7 +83,7 @@ The following API resources are available in the project context: | [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | | [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | | [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Services](services.md) | `/projects/:id/services` | +| [Services](services.md) (renamed to [Integrations](integrations.md)) | `/projects/:id/services` | | [Tags](tags.md) | `/projects/:id/repository/tags` | | [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | | [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | diff --git a/doc/api/applications.md b/doc/api/applications.md index 2b5eff68010..22d858bd68a 100644 --- a/doc/api/applications.md +++ b/doc/api/applications.md @@ -34,14 +34,14 @@ Parameters: |:---------------|:--------|:---------|:---------------------------------| | `name` | string | yes | Name of the application. | | `redirect_uri` | string | yes | Redirect URI of the application. | -| `scopes` | string | yes | Scopes of the application. | +| `scopes` | string | yes | Scopes of the application. You can specify multiple scopes by separating each scope using a space. | | `confidential` | boolean | no | The application is used where the client secret can be kept confidential. Native mobile apps and Single Page Apps are considered non-confidential. Defaults to `true` if not supplied | Example request: ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - --data "name=MyApplication&redirect_uri=http://redirect.uri&scopes=" \ + --data "name=MyApplication&redirect_uri=http://redirect.uri&scopes=api read_user email" \ "https://gitlab.example.com/api/v4/applications" ``` diff --git a/doc/api/commits.md b/doc/api/commits.md index e91da23596f..94d1ced181c 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api --- # Commits API **(FREE)** @@ -75,8 +74,6 @@ Example response: ## Create a commit with multiple files and actions -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6096) in GitLab 8.13. - Create a commit by posting a JSON payload ```plaintext @@ -256,8 +253,6 @@ Example response: ## Get references a commit is pushed to -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15026) in GitLab 10.6 - Get all references (from branches or tags) a commit is pushed to. The pagination parameters `page` and `per_page` can be used to restrict the list of references. @@ -291,8 +286,6 @@ Example response: ## Cherry-pick a commit -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8047) in GitLab 8.15. - Cherry-picks a commit to a given branch. ```plaintext @@ -366,8 +359,6 @@ dry run. ## Revert a commit -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22919) in GitLab 11.5. - Reverts a commit in a given branch. ```plaintext @@ -622,7 +613,7 @@ Example response: ## Commit status -In GitLab 8.1 and later, this is the new commit status API. +This is the commit status API for use with GitLab. ### List the statuses of a commit @@ -752,8 +743,6 @@ Example response: ## List Merge Requests associated with a commit -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18004) in GitLab 10.7. - Get a list of Merge Requests related to the specified commit. ```plaintext diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index 2885cc7d803..1b9778a01b3 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -350,7 +350,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ ``` This action doesn't delete blobs. To delete them and recycle disk space, -[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). +[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/index.html#removing-unused-layers-not-referenced-by-manifests). ## Delete registry repository tags in bulk @@ -388,7 +388,7 @@ This API call performs the following operations: These operations are executed asynchronously and can take time to get executed. You can run this at most once an hour for a given container repository. This action doesn't delete blobs. To delete them and recycle disk space, -[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/README.html#removing-unused-layers-not-referenced-by-manifests). +[run the garbage collection](https://docs.gitlab.com/omnibus/maintenance/index.html#removing-unused-layers-not-referenced-by-manifests). WARNING: The number of tags deleted by this API is limited on GitLab.com diff --git a/doc/api/dependencies.md b/doc/api/dependencies.md index 6e9c37980ac..b421a32b88a 100644 --- a/doc/api/dependencies.md +++ b/doc/api/dependencies.md @@ -34,7 +34,7 @@ GET /projects/:id/dependencies?package_manager=yarn,bundler | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | -| `package_manager` | string array | no | Returns dependencies belonging to specified package manager. Valid values: `bundler`, `composer`, `conan`, `maven`, `npm`, `pip` or `yarn`. | +| `package_manager` | string array | no | Returns dependencies belonging to specified package manager. Valid values: `bundler`, `composer`, `conan`, `go`, `maven`, `npm`, `nuget`, `pip`, `yarn`, or `sbt`. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/dependencies" diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index 3de7ff4ac44..c7189586230 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21811) in GitLab 12.9. -Get a list of all deploy tokens across the GitLab instance. This endpoint requires administrator access. +Get a list of all deploy tokens across the GitLab instance. This endpoint requires the Administrator role. ```plaintext GET /deploy_tokens diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index 8c82446db2e..4fbd2b0fa80 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Manage +group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, api --- diff --git a/doc/api/dora4_project_analytics.md b/doc/api/dora4_project_analytics.md index 5a6e1576a3d..f69c918c6e2 100644 --- a/doc/api/dora4_project_analytics.md +++ b/doc/api/dora4_project_analytics.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Manage +group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, api --- diff --git a/doc/api/environments.md b/doc/api/environments.md index 5187ac7c1b2..8188e0e7b85 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -64,6 +64,8 @@ Example of response "slug": "review-fix-foo-dfjre3", "external_url": "https://review-fix-foo-dfjre3.gitlab.example.com", "state": "available", + "created_at": "2019-05-25T18:55:13.252Z", + "updated_at": "2019-05-27T18:55:13.252Z", "last_deployment": { "id": 100, "iid": 34, @@ -176,7 +178,9 @@ Example response: "name": "deploy", "slug": "deploy", "external_url": "https://deploy.gitlab.example.com", - "state": "available" + "state": "available", + "created_at": "2019-05-25T18:55:13.252Z", + "updated_at": "2019-05-27T18:55:13.252Z" } ``` @@ -210,7 +214,9 @@ Example response: "name": "staging", "slug": "staging", "external_url": "https://staging.gitlab.example.com", - "state": "available" + "state": "available", + "created_at": "2019-05-25T18:55:13.252Z", + "updated_at": "2019-05-27T18:55:13.252Z" } ``` @@ -302,6 +308,8 @@ Example response: "name": "deploy", "slug": "deploy", "external_url": "https://deploy.gitlab.example.com", - "state": "stopped" + "state": "stopped", + "created_at": "2019-05-25T18:55:13.252Z", + "updated_at": "2019-05-27T18:55:13.252Z" } ``` diff --git a/doc/api/error_tracking.md b/doc/api/error_tracking.md index 1abe5522840..203c1a23996 100644 --- a/doc/api/error_tracking.md +++ b/doc/api/error_tracking.md @@ -73,7 +73,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68384) in GitLab 14.3. -For [integrated error tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) feature that is behind a disabled feature flag. Only for project maintainers. +For [integrated error tracking](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) feature. Only for project maintainers. ### List project client keys diff --git a/doc/api/events.md b/doc/api/events.md index 8800e7f7f9b..2d173f0053f 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -87,10 +87,6 @@ GitLab removes events older than 3 years from the events table for performance r ## List currently authenticated user's events ->**Notes:** -> This endpoint was introduced in GitLab 9.3. -> `read_user` access was introduced in GitLab 11.3. - Get a list of events for the authenticated user. Scope `read_user` or `api` is required. ```plaintext @@ -163,10 +159,6 @@ Example response: ### Get user contribution events ->**Notes:** -> Documentation was formerly located in the [Users API pages](users.md). -> `read_user` access was introduced in GitLab 11.3. - Get the contribution events for the specified user, sorted from newest to oldest. Scope `read_user` or `api` is required. ```plaintext diff --git a/doc/api/experiments.md b/doc/api/experiments.md index c5e217a3d66..669d00454bd 100644 --- a/doc/api/experiments.md +++ b/doc/api/experiments.md @@ -4,7 +4,7 @@ group: Expansion info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Experiments API +# Experiments API (GitLab team only) **(FREE SAAS)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/262725) in GitLab 13.5. diff --git a/doc/api/freeze_periods.md b/doc/api/freeze_periods.md index 6ca69d047da..6dc4e8745d6 100644 --- a/doc/api/freeze_periods.md +++ b/doc/api/freeze_periods.md @@ -94,7 +94,7 @@ POST /projects/:id/freeze_periods | `id` | integer or string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `freeze_start` | string | yes | Start of the Freeze Period in [cron](https://crontab.guru/) format. | | `freeze_end` | string | yes | End of the Freeze Period in [cron](https://crontab.guru/) format. | -| `cron_timezone` | string | no | The timezone for the cron fields, defaults to UTC if not provided. | +| `cron_timezone` | string | no | The time zone for the cron fields, defaults to UTC if not provided. | Example request: @@ -131,7 +131,7 @@ PUT /projects/:id/freeze_periods/:freeze_period_id | `freeze_period_id` | integer or string | yes | The database ID of the Freeze Period. | | `freeze_start` | string | no | Start of the Freeze Period in [cron](https://crontab.guru/) format. | | `freeze_end` | string | no | End of the Freeze Period in [cron](https://crontab.guru/) format. | -| `cron_timezone` | string | no | The timezone for the cron fields. | +| `cron_timezone` | string | no | The time zone for the cron fields. | Example request: diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index c4e73f9c058..e0f18f931f5 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -25,13 +25,30 @@ Items (fields, enums, etc) that have been removed according to our [deprecation in [Removed Items](../removed_items.md). <!-- vale off --> -<!-- Docs linting disabled after this line. --> +<!-- Vale linting disabled after this line. --> <!-- See https://docs.gitlab.com/ee/development/documentation/testing.html#disable-vale-tests --> +<!-- markdownlint-disable MD044 --> +<!-- MD044/proper-names test disabled after this line to make page compatible with markdownlint-cli 0.29.0. --> +<!-- See https://docs.gitlab.com/ee/development/documentation/testing.html#disable-markdownlint-tests --> + ## `Query` type The `Query` type contains the API's top-level entry points for all executable queries. +### `Query.boardList` + +Find an issue board list. + +Returns [`BoardList`](#boardlist). + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="queryboardlistid"></a>`id` | [`ListID!`](#listid) | Global ID of the list. | +| <a id="queryboardlistissuefilters"></a>`issueFilters` | [`BoardIssueInput`](#boardissueinput) | Filters applied when getting issue metadata in the board list. | + ### `Query.ciApplicationSettings` CI related settings that apply to the entire instance. @@ -469,6 +486,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="queryvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="queryvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | +| <a id="queryvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="queryvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="queryvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="queryvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -1266,6 +1284,9 @@ Input type: `CreateIssueInput` | <a id="mutationcreateissueepicid"></a>`epicId` | [`EpicID`](#epicid) | ID of an epic to associate the issue with. | | <a id="mutationcreateissuehealthstatus"></a>`healthStatus` | [`HealthStatus`](#healthstatus) | Desired health status. | | <a id="mutationcreateissueiid"></a>`iid` | [`Int`](#int) | IID (internal ID) of a project issue. Only admins and project owners can modify. | +| <a id="mutationcreateissueiterationcadenceid"></a>`iterationCadenceId` | [`IterationsCadenceID`](#iterationscadenceid) | Global iteration cadence ID. Required when `iterationWildcardId` is provided. | +| <a id="mutationcreateissueiterationid"></a>`iterationId` | [`IterationID`](#iterationid) | Global iteration ID. Mutually exlusive argument with `iterationWildcardId`. | +| <a id="mutationcreateissueiterationwildcardid"></a>`iterationWildcardId` | [`IssueCreationIterationWildcardId`](#issuecreationiterationwildcardid) | Iteration wildcard ID. Supported values are: `CURRENT`. Mutually exclusive argument with `iterationId`. iterationCadenceId also required when this argument is provided. | | <a id="mutationcreateissuelabelids"></a>`labelIds` | [`[LabelID!]`](#labelid) | IDs of labels to be added to the issue. | | <a id="mutationcreateissuelabels"></a>`labels` | [`[String!]`](#string) | Labels of the issue. | | <a id="mutationcreateissuelocked"></a>`locked` | [`Boolean`](#boolean) | Indicates discussion is locked on the issue. | @@ -1408,6 +1429,56 @@ Input type: `CreateTestCaseInput` | <a id="mutationcreatetestcaseerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationcreatetestcasetestcase"></a>`testCase` | [`Issue`](#issue) | Test case created. | +### `Mutation.customerRelationsContactCreate` + +Input type: `CustomerRelationsContactCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcustomerrelationscontactcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcustomerrelationscontactcreatedescription"></a>`description` | [`String`](#string) | Description of or notes for the contact. | +| <a id="mutationcustomerrelationscontactcreateemail"></a>`email` | [`String`](#string) | Email address of the contact. | +| <a id="mutationcustomerrelationscontactcreatefirstname"></a>`firstName` | [`String!`](#string) | First name of the contact. | +| <a id="mutationcustomerrelationscontactcreategroupid"></a>`groupId` | [`GroupID!`](#groupid) | Group for the contact. | +| <a id="mutationcustomerrelationscontactcreatelastname"></a>`lastName` | [`String!`](#string) | Last name of the contact. | +| <a id="mutationcustomerrelationscontactcreateorganizationid"></a>`organizationId` | [`CustomerRelationsOrganizationID`](#customerrelationsorganizationid) | Organization for the contact. | +| <a id="mutationcustomerrelationscontactcreatephone"></a>`phone` | [`String`](#string) | Phone number of the contact. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcustomerrelationscontactcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcustomerrelationscontactcreatecontact"></a>`contact` | [`CustomerRelationsContact`](#customerrelationscontact) | Contact after the mutation. | +| <a id="mutationcustomerrelationscontactcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + +### `Mutation.customerRelationsContactUpdate` + +Input type: `CustomerRelationsContactUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcustomerrelationscontactupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcustomerrelationscontactupdatedescription"></a>`description` | [`String`](#string) | Description of or notes for the contact. | +| <a id="mutationcustomerrelationscontactupdateemail"></a>`email` | [`String`](#string) | Email address of the contact. | +| <a id="mutationcustomerrelationscontactupdatefirstname"></a>`firstName` | [`String`](#string) | First name of the contact. | +| <a id="mutationcustomerrelationscontactupdateid"></a>`id` | [`CustomerRelationsContactID!`](#customerrelationscontactid) | Global ID of the contact. | +| <a id="mutationcustomerrelationscontactupdatelastname"></a>`lastName` | [`String`](#string) | Last name of the contact. | +| <a id="mutationcustomerrelationscontactupdateorganizationid"></a>`organizationId` | [`CustomerRelationsOrganizationID`](#customerrelationsorganizationid) | Organization of the contact. | +| <a id="mutationcustomerrelationscontactupdatephone"></a>`phone` | [`String`](#string) | Phone number of the contact. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcustomerrelationscontactupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcustomerrelationscontactupdatecontact"></a>`contact` | [`CustomerRelationsContact`](#customerrelationscontact) | Contact after the mutation. | +| <a id="mutationcustomerrelationscontactupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.customerRelationsOrganizationCreate` Input type: `CustomerRelationsOrganizationCreateInput` @@ -1418,7 +1489,7 @@ Input type: `CustomerRelationsOrganizationCreateInput` | ---- | ---- | ----------- | | <a id="mutationcustomerrelationsorganizationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcustomerrelationsorganizationcreatedefaultrate"></a>`defaultRate` | [`Float`](#float) | Standard billing rate for the organization. | -| <a id="mutationcustomerrelationsorganizationcreatedescription"></a>`description` | [`String`](#string) | Description or notes for the organization. | +| <a id="mutationcustomerrelationsorganizationcreatedescription"></a>`description` | [`String`](#string) | Description of or notes for the organization. | | <a id="mutationcustomerrelationsorganizationcreategroupid"></a>`groupId` | [`GroupID!`](#groupid) | Group for the organization. | | <a id="mutationcustomerrelationsorganizationcreatename"></a>`name` | [`String!`](#string) | Name of the organization. | @@ -1440,7 +1511,7 @@ Input type: `CustomerRelationsOrganizationUpdateInput` | ---- | ---- | ----------- | | <a id="mutationcustomerrelationsorganizationupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcustomerrelationsorganizationupdatedefaultrate"></a>`defaultRate` | [`Float`](#float) | Standard billing rate for the organization. | -| <a id="mutationcustomerrelationsorganizationupdatedescription"></a>`description` | [`String`](#string) | Description or notes for the organization. | +| <a id="mutationcustomerrelationsorganizationupdatedescription"></a>`description` | [`String`](#string) | Description of or notes for the organization. | | <a id="mutationcustomerrelationsorganizationupdateid"></a>`id` | [`CustomerRelationsOrganizationID!`](#customerrelationsorganizationid) | Global ID of the organization. | | <a id="mutationcustomerrelationsorganizationupdatename"></a>`name` | [`String`](#string) | Name of the organization. | @@ -2432,6 +2503,64 @@ Input type: `ExportRequirementsInput` | <a id="mutationexportrequirementsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationexportrequirementserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.externalAuditEventDestinationCreate` + +Input type: `ExternalAuditEventDestinationCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationcreatedestinationurl"></a>`destinationUrl` | [`String!`](#string) | Destination URL. | +| <a id="mutationexternalauditeventdestinationcreategrouppath"></a>`groupPath` | [`ID!`](#id) | Group path. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationexternalauditeventdestinationcreateexternalauditeventdestination"></a>`externalAuditEventDestination` | [`ExternalAuditEventDestination`](#externalauditeventdestination) | Destination created. | + +### `Mutation.externalAuditEventDestinationDestroy` + +Input type: `ExternalAuditEventDestinationDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationdestroyid"></a>`id` | [`AuditEventsExternalAuditEventDestinationID!`](#auditeventsexternalauditeventdestinationid) | ID of external audit event destination to destroy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationdestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + +### `Mutation.externalAuditEventDestinationUpdate` + +Input type: `ExternalAuditEventDestinationUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationupdatedestinationurl"></a>`destinationUrl` | [`String`](#string) | Destination URL to change. | +| <a id="mutationexternalauditeventdestinationupdateid"></a>`id` | [`AuditEventsExternalAuditEventDestinationID!`](#auditeventsexternalauditeventdestinationid) | ID of external audit event destination to destroy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationexternalauditeventdestinationupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationexternalauditeventdestinationupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationexternalauditeventdestinationupdateexternalauditeventdestination"></a>`externalAuditEventDestination` | [`ExternalAuditEventDestination`](#externalauditeventdestination) | Updated destination. | + ### `Mutation.gitlabSubscriptionActivate` Input type: `GitlabSubscriptionActivateInput` @@ -3904,6 +4033,8 @@ Input type: `RunnersRegistrationTokenResetInput` ### `Mutation.scanExecutionPolicyCommit` +Commits the `policy_yaml` content to the assigned security policy project for the given project(`project_path`). + Input type: `ScanExecutionPolicyCommitInput` #### Arguments @@ -3925,6 +4056,8 @@ Input type: `ScanExecutionPolicyCommitInput` ### `Mutation.securityPolicyProjectAssign` +Assigns the specified project(`security_policy_project_id`) as security policy project for the given project(`project_path`). If the project already has a security policy project, this reassigns the project's security policy project with the given `security_policy_project_id`. + Input type: `SecurityPolicyProjectAssignInput` #### Arguments @@ -3944,6 +4077,8 @@ Input type: `SecurityPolicyProjectAssignInput` ### `Mutation.securityPolicyProjectCreate` +Creates and assigns a security policy project for the given project(`project_path`). + Input type: `SecurityPolicyProjectCreateInput` #### Arguments @@ -4269,6 +4404,26 @@ Input type: `UpdateDependencyProxyImageTtlGroupPolicyInput` | <a id="mutationupdatedependencyproxyimagettlgrouppolicydependencyproxyimagettlpolicy"></a>`dependencyProxyImageTtlPolicy` | [`DependencyProxyImageTtlGroupPolicy`](#dependencyproxyimagettlgrouppolicy) | Group image TTL policy after mutation. | | <a id="mutationupdatedependencyproxyimagettlgrouppolicyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.updateDependencyProxySettings` + +Input type: `UpdateDependencyProxySettingsInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationupdatedependencyproxysettingsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationupdatedependencyproxysettingsenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates whether the policy is enabled or disabled. | +| <a id="mutationupdatedependencyproxysettingsgrouppath"></a>`groupPath` | [`ID!`](#id) | Group path for the group dependency proxy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationupdatedependencyproxysettingsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationupdatedependencyproxysettingsdependencyproxysetting"></a>`dependencyProxySetting` | [`DependencyProxySetting`](#dependencyproxysetting) | Group dependency proxy settings after mutation. | +| <a id="mutationupdatedependencyproxysettingserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.updateEpic` Input type: `UpdateEpicInput` @@ -4560,13 +4715,13 @@ Input type: `VulnerabilityCreateInput` | <a id="mutationvulnerabilitycreatedismissedat"></a>`dismissedAt` | [`Time`](#time) | Timestamp of when the vulnerability state changed to dismissed (defaults to creation time if status is `dismissed`). | | <a id="mutationvulnerabilitycreateidentifiers"></a>`identifiers` | [`[VulnerabilityIdentifierInput!]!`](#vulnerabilityidentifierinput) | Array of CVE or CWE identifiers for the vulnerability. | | <a id="mutationvulnerabilitycreatemessage"></a>`message` | [`String`](#string) | Additional information about the vulnerability. | +| <a id="mutationvulnerabilitycreatename"></a>`name` | [`String!`](#string) | Name of the vulnerability. | | <a id="mutationvulnerabilitycreateproject"></a>`project` | [`ProjectID!`](#projectid) | ID of the project to attach the vulnerability to. | | <a id="mutationvulnerabilitycreateresolvedat"></a>`resolvedAt` | [`Time`](#time) | Timestamp of when the vulnerability state changed to resolved (defaults to creation time if status is `resolved`). | -| <a id="mutationvulnerabilitycreatescannername"></a>`scannerName` | [`String!`](#string) | Name of the security scanner used to discover the vulnerability. | +| <a id="mutationvulnerabilitycreatescanner"></a>`scanner` | [`VulnerabilityScannerInput!`](#vulnerabilityscannerinput) | Information about the scanner used to discover the vulnerability. | | <a id="mutationvulnerabilitycreateseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (defaults to `unknown`). | | <a id="mutationvulnerabilitycreatesolution"></a>`solution` | [`String`](#string) | How to fix this vulnerability. | | <a id="mutationvulnerabilitycreatestate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (defaults to `detected`). | -| <a id="mutationvulnerabilitycreatetitle"></a>`title` | [`String!`](#string) | Title of the vulnerability. | #### Fields @@ -5173,6 +5328,7 @@ The edge type for [`CiRunner`](#cirunner). | ---- | ---- | ----------- | | <a id="cirunneredgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="cirunneredgenode"></a>`node` | [`CiRunner`](#cirunner) | The item at the end of the edge. | +| <a id="cirunneredgeweburl"></a>`webUrl` | [`String`](#string) | Web URL of the runner. The value depends on where you put this field in the query. You can use it for projects or groups. | #### `CiStageConnection` @@ -5915,6 +6071,29 @@ The edge type for [`Event`](#event). | <a id="eventedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="eventedgenode"></a>`node` | [`Event`](#event) | The item at the end of the edge. | +#### `ExternalAuditEventDestinationConnection` + +The connection type for [`ExternalAuditEventDestination`](#externalauditeventdestination). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="externalauditeventdestinationconnectionedges"></a>`edges` | [`[ExternalAuditEventDestinationEdge]`](#externalauditeventdestinationedge) | A list of edges. | +| <a id="externalauditeventdestinationconnectionnodes"></a>`nodes` | [`[ExternalAuditEventDestination]`](#externalauditeventdestination) | A list of nodes. | +| <a id="externalauditeventdestinationconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `ExternalAuditEventDestinationEdge` + +The edge type for [`ExternalAuditEventDestination`](#externalauditeventdestination). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="externalauditeventdestinationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="externalauditeventdestinationedgenode"></a>`node` | [`ExternalAuditEventDestination`](#externalauditeventdestination) | The item at the end of the edge. | + #### `GroupConnection` The connection type for [`Group`](#group). @@ -6503,6 +6682,7 @@ The connection type for [`Package`](#package). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="packageconnectioncount"></a>`count` | [`Int!`](#int) | Total count of collection. | | <a id="packageconnectionedges"></a>`edges` | [`[PackageEdge]`](#packageedge) | A list of edges. | | <a id="packageconnectionnodes"></a>`nodes` | [`[Package]`](#package) | A list of nodes. | | <a id="packageconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | @@ -8072,7 +8252,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="boardepicancestorsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="boardepicancestorsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="boardepicancestorsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="boardepicancestorssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="boardepicancestorssearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="boardepicancestorssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="boardepicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="boardepicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -8105,7 +8285,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="boardepicchildrenmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="boardepicchildrenmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="boardepicchildrennot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="boardepicchildrensearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="boardepicchildrensearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="boardepicchildrensort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="boardepicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="boardepicchildrenstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -8391,6 +8571,7 @@ Represents the total number of issues and their weights for a particular day. | ---- | ---- | ----------- | | <a id="cirunneraccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel!`](#cirunneraccesslevel) | Access level of the runner. | | <a id="cirunneractive"></a>`active` | [`Boolean!`](#boolean) | Indicates the runner is allowed to receive jobs. | +| <a id="cirunneradminurl"></a>`adminUrl` | [`String`](#string) | Admin URL of the runner. Only available for adminstrators. | | <a id="cirunnercontactedat"></a>`contactedAt` | [`Time`](#time) | Last contact from the runner. | | <a id="cirunnerdescription"></a>`description` | [`String`](#string) | Description of the runner. | | <a id="cirunnerid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner. | @@ -8407,6 +8588,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cirunnershortsha"></a>`shortSha` | [`String`](#string) | First eight characters of the runner's token used to authenticate new job requests. Used as the runner's unique ID. | | <a id="cirunnerstatus"></a>`status` | [`CiRunnerStatus!`](#cirunnerstatus) | Status of the runner. | | <a id="cirunnertaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | +| <a id="cirunneruserpermissions"></a>`userPermissions` | [`RunnerPermissions!`](#runnerpermissions) | Permissions for the current user on the resource. | | <a id="cirunnerversion"></a>`version` | [`String`](#string) | Version of the runner. | ### `CiStage` @@ -8737,7 +8919,7 @@ A custom emoji uploaded by user. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="customerrelationscontactcreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp the contact was created. | -| <a id="customerrelationscontactdescription"></a>`description` | [`String`](#string) | Description or notes for the contact. | +| <a id="customerrelationscontactdescription"></a>`description` | [`String`](#string) | Description of or notes for the contact. | | <a id="customerrelationscontactemail"></a>`email` | [`String`](#string) | Email address of the contact. | | <a id="customerrelationscontactfirstname"></a>`firstName` | [`String!`](#string) | First name of the contact. | | <a id="customerrelationscontactid"></a>`id` | [`ID!`](#id) | Internal ID of the contact. | @@ -8754,7 +8936,7 @@ A custom emoji uploaded by user. | ---- | ---- | ----------- | | <a id="customerrelationsorganizationcreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp the organization was created. | | <a id="customerrelationsorganizationdefaultrate"></a>`defaultRate` | [`Float`](#float) | Standard billing rate for the organization. | -| <a id="customerrelationsorganizationdescription"></a>`description` | [`String`](#string) | Description or notes for the organization. | +| <a id="customerrelationsorganizationdescription"></a>`description` | [`String`](#string) | Description of or notes for the organization. | | <a id="customerrelationsorganizationid"></a>`id` | [`ID!`](#id) | Internal ID of the organization. | | <a id="customerrelationsorganizationname"></a>`name` | [`String!`](#string) | Name of the organization. | | <a id="customerrelationsorganizationupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp the organization was last updated. | @@ -9477,7 +9659,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="epicancestorsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="epicancestorsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="epicancestorsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="epicancestorssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="epicancestorssearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="epicancestorssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="epicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="epicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -9510,7 +9692,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="epicchildrenmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="epicchildrenmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="epicchildrennot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="epicchildrensearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="epicchildrensearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="epicchildrensort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="epicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="epicchildrenstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -9634,6 +9816,7 @@ Relationship between an epic and an issue. | <a id="epicissueconfidential"></a>`confidential` | [`Boolean!`](#boolean) | Indicates the issue is confidential. | | <a id="epicissuecreatenoteemail"></a>`createNoteEmail` | [`String`](#string) | User specific email address for the issue. | | <a id="epicissuecreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp of when the issue was created. | +| <a id="epicissuecustomerrelationscontacts"></a>`customerRelationsContacts` | [`CustomerRelationsContactConnection`](#customerrelationscontactconnection) | Customer relations contacts of the issue. (see [Connections](#connections)) | | <a id="epicissuedescription"></a>`description` | [`String`](#string) | Description of the issue. | | <a id="epicissuedescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="epicissuedesigncollection"></a>`designCollection` | [`DesignCollection`](#designcollection) | Collection of design images associated with this issue. | @@ -9806,6 +9989,18 @@ Representing an event. | <a id="eventid"></a>`id` | [`ID!`](#id) | ID of the event. | | <a id="eventupdatedat"></a>`updatedAt` | [`Time!`](#time) | When this event was updated. | +### `ExternalAuditEventDestination` + +Represents an external resource to send audit events to. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="externalauditeventdestinationdestinationurl"></a>`destinationUrl` | [`String!`](#string) | External destination to send audit events to. | +| <a id="externalauditeventdestinationgroup"></a>`group` | [`Group!`](#group) | Group the destination belongs to. | +| <a id="externalauditeventdestinationid"></a>`id` | [`ID!`](#id) | ID of the destination. | + ### `ExternalIssue` Represents an external issue. @@ -9976,7 +10171,7 @@ four standard [pagination arguments](#connection-pagination-arguments): ##### `GeoNode.uploadRegistries` -Find Upload registries on this Geo node Available only when feature flag `geo_upload_replication` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. +Find Upload registries on this Geo node. Returns [`UploadRegistryConnection`](#uploadregistryconnection). @@ -10031,6 +10226,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupemailsdisabled"></a>`emailsDisabled` | [`Boolean`](#boolean) | Indicates if a group has email notifications disabled. | | <a id="groupepicboards"></a>`epicBoards` | [`EpicBoardConnection`](#epicboardconnection) | Find epic boards. (see [Connections](#connections)) | | <a id="groupepicsenabled"></a>`epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace. | +| <a id="groupexternalauditeventdestinations"></a>`externalAuditEventDestinations` | [`ExternalAuditEventDestinationConnection`](#externalauditeventdestinationconnection) | External locations that receive audit events belonging to the group. Available only when feature flag `ff_external_audit_events_namespace` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. (see [Connections](#connections)) | | <a id="groupfullname"></a>`fullName` | [`String!`](#string) | Full name of the namespace. | | <a id="groupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. | | <a id="groupid"></a>`id` | [`ID!`](#id) | ID of the namespace. | @@ -10181,7 +10377,7 @@ Returns [`Epic`](#epic). | <a id="groupepicmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="groupepicmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="groupepicnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="groupepicsearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="groupepicsearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="groupepicsort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="groupepicstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="groupepicstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -10226,7 +10422,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupepicsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="groupepicsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | | <a id="groupepicsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | -| <a id="groupepicssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="groupepicssearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="groupepicssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="groupepicsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | | <a id="groupepicsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | @@ -10269,11 +10465,13 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupissuesauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | | <a id="groupissuesclosedafter"></a>`closedAfter` | [`Time`](#time) | Issues closed after this date. | | <a id="groupissuesclosedbefore"></a>`closedBefore` | [`Time`](#time) | Issues closed before this date. | +| <a id="groupissuesconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter for confidential issues. If "false", excludes confidential issues. If "true", returns only confidential issues. | | <a id="groupissuescreatedafter"></a>`createdAfter` | [`Time`](#time) | Issues created after this date. | | <a id="groupissuescreatedbefore"></a>`createdBefore` | [`Time`](#time) | Issues created before this date. | | <a id="groupissuesepicid"></a>`epicId` | [`String`](#string) | ID of an epic associated with the issues, "none" and "any" values are supported. | | <a id="groupissuesiid"></a>`iid` | [`String`](#string) | IID of the issue. For example, "1". | | <a id="groupissuesiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of issues. For example, `["1", "2"]`. | +| <a id="groupissuesincludesubepics"></a>`includeSubepics` | [`Boolean`](#boolean) | Whether to include subepics when filtering issues by epicId. | | <a id="groupissuesincludesubgroups"></a>`includeSubgroups` | [`Boolean`](#boolean) | Include issues belonging to subgroups. | | <a id="groupissuesiterationid"></a>`iterationId` | [`[ID]`](#id) | List of iteration Global IDs applied to the issue. | | <a id="groupissuesiterationwildcardid"></a>`iterationWildcardId` | [`IterationWildcardId`](#iterationwildcardid) | Filter by iteration ID wildcard. | @@ -10282,7 +10480,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupissuesmilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="groupissuesmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="groupissuesnot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | -| <a id="groupissuessearch"></a>`search` | [`String`](#string) | Search query for issue title or description. | +| <a id="groupissuessearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="groupissuessort"></a>`sort` | [`IssueSort`](#issuesort) | Sort issues by this criteria. | | <a id="groupissuesstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this issue. | | <a id="groupissuestypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter issues by the given issue types. | @@ -10520,6 +10718,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="groupvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="groupvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | +| <a id="groupvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="groupvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="groupvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="groupvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -10781,6 +10980,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | <a id="issueconfidential"></a>`confidential` | [`Boolean!`](#boolean) | Indicates the issue is confidential. | | <a id="issuecreatenoteemail"></a>`createNoteEmail` | [`String`](#string) | User specific email address for the issue. | | <a id="issuecreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp of when the issue was created. | +| <a id="issuecustomerrelationscontacts"></a>`customerRelationsContacts` | [`CustomerRelationsContactConnection`](#customerrelationscontactconnection) | Customer relations contacts of the issue. (see [Connections](#connections)) | | <a id="issuedescription"></a>`description` | [`String`](#string) | Description of the issue. | | <a id="issuedescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="issuedesigncollection"></a>`designCollection` | [`DesignCollection`](#designcollection) | Collection of design images associated with this issue. | @@ -11948,10 +12148,10 @@ Nuget metadata. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="nugetmetadataiconurl"></a>`iconUrl` | [`String!`](#string) | Icon URL of the Nuget package. | +| <a id="nugetmetadataiconurl"></a>`iconUrl` | [`String`](#string) | Icon URL of the Nuget package. | | <a id="nugetmetadataid"></a>`id` | [`PackagesNugetMetadatumID!`](#packagesnugetmetadatumid) | ID of the metadatum. | -| <a id="nugetmetadatalicenseurl"></a>`licenseUrl` | [`String!`](#string) | License URL of the Nuget package. | -| <a id="nugetmetadataprojecturl"></a>`projectUrl` | [`String!`](#string) | Project URL of the Nuget package. | +| <a id="nugetmetadatalicenseurl"></a>`licenseUrl` | [`String`](#string) | License URL of the Nuget package. | +| <a id="nugetmetadataprojecturl"></a>`projectUrl` | [`String`](#string) | Project URL of the Nuget package. | ### `OncallParticipantType` @@ -11985,6 +12185,7 @@ Represents a package in the Package Registry. Note that this type is in beta and | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="packagecandestroy"></a>`canDestroy` | [`Boolean!`](#boolean) | Whether the user can destroy the package. | | <a id="packagecreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | | <a id="packageid"></a>`id` | [`PackagesPackageID!`](#packagespackageid) | ID of the package. | | <a id="packagemetadata"></a>`metadata` | [`PackageMetadata`](#packagemetadata) | Package metadata. | @@ -12044,6 +12245,7 @@ Represents a package details in the Package Registry. Note that this type is in | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="packagedetailstypecandestroy"></a>`canDestroy` | [`Boolean!`](#boolean) | Whether the user can destroy the package. | | <a id="packagedetailstypecreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | | <a id="packagedetailstypedependencylinks"></a>`dependencyLinks` | [`PackageDependencyLinkConnection`](#packagedependencylinkconnection) | Dependency link. (see [Connections](#connections)) | | <a id="packagedetailstypeid"></a>`id` | [`PackagesPackageID!`](#packagespackageid) | ID of the package. | @@ -12179,6 +12381,7 @@ Represents a file or directory in the project repository that has been locked. | <a id="pipelineconfigsource"></a>`configSource` | [`PipelineConfigSourceEnum`](#pipelineconfigsourceenum) | Configuration source of the pipeline (UNKNOWN_SOURCE, REPOSITORY_SOURCE, AUTO_DEVOPS_SOURCE, WEBIDE_SOURCE, REMOTE_SOURCE, EXTERNAL_PROJECT_SOURCE, BRIDGE_SOURCE, PARAMETER_SOURCE, COMPLIANCE_SOURCE). | | <a id="pipelinecoverage"></a>`coverage` | [`Float`](#float) | Coverage percentage. | | <a id="pipelinecreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp of the pipeline's creation. | +| <a id="pipelinedastprofile"></a>`dastProfile` | [`DastProfile`](#dastprofile) | DAST profile associated with the pipeline. Returns `null`if `dast_view_scans` feature flag is disabled. | | <a id="pipelinedetailedstatus"></a>`detailedStatus` | [`DetailedStatus!`](#detailedstatus) | Detailed status of the pipeline. | | <a id="pipelinedownstream"></a>`downstream` | [`PipelineConnection`](#pipelineconnection) | Pipelines this pipeline will trigger. (see [Connections](#connections)) | | <a id="pipelineduration"></a>`duration` | [`Int`](#int) | Duration of the pipeline in seconds. | @@ -12624,7 +12827,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectdastsitevalidationsnormalizedtargeturls"></a>`normalizedTargetUrls` | [`[String!]`](#string) | Normalized URL of the target to be scanned. | -| <a id="projectdastsitevalidationsstatus"></a>`status` | [`DastSiteValidationStatusEnum`](#dastsitevalidationstatusenum) | Status of the site validation. Ignored if `dast_failed_site_validations` feature flag is disabled. | +| <a id="projectdastsitevalidationsstatus"></a>`status` | [`DastSiteValidationStatusEnum`](#dastsitevalidationstatusenum) | Status of the site validation. | ##### `Project.environment` @@ -12702,11 +12905,13 @@ Returns [`Issue`](#issue). | <a id="projectissueauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | | <a id="projectissueclosedafter"></a>`closedAfter` | [`Time`](#time) | Issues closed after this date. | | <a id="projectissueclosedbefore"></a>`closedBefore` | [`Time`](#time) | Issues closed before this date. | +| <a id="projectissueconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter for confidential issues. If "false", excludes confidential issues. If "true", returns only confidential issues. | | <a id="projectissuecreatedafter"></a>`createdAfter` | [`Time`](#time) | Issues created after this date. | | <a id="projectissuecreatedbefore"></a>`createdBefore` | [`Time`](#time) | Issues created before this date. | | <a id="projectissueepicid"></a>`epicId` | [`String`](#string) | ID of an epic associated with the issues, "none" and "any" values are supported. | | <a id="projectissueiid"></a>`iid` | [`String`](#string) | IID of the issue. For example, "1". | | <a id="projectissueiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of issues. For example, `["1", "2"]`. | +| <a id="projectissueincludesubepics"></a>`includeSubepics` | [`Boolean`](#boolean) | Whether to include subepics when filtering issues by epicId. | | <a id="projectissueiterationid"></a>`iterationId` | [`[ID]`](#id) | List of iteration Global IDs applied to the issue. | | <a id="projectissueiterationwildcardid"></a>`iterationWildcardId` | [`IterationWildcardId`](#iterationwildcardid) | Filter by iteration ID wildcard. | | <a id="projectissuelabelname"></a>`labelName` | [`[String]`](#string) | Labels applied to this issue. | @@ -12714,7 +12919,7 @@ Returns [`Issue`](#issue). | <a id="projectissuemilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuemyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuenot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | -| <a id="projectissuesearch"></a>`search` | [`String`](#string) | Search query for issue title or description. | +| <a id="projectissuesearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuesort"></a>`sort` | [`IssueSort`](#issuesort) | Sort issues by this criteria. | | <a id="projectissuestate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this issue. | | <a id="projectissuetypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter issues by the given issue types. | @@ -12738,6 +12943,7 @@ Returns [`IssueStatusCountsType`](#issuestatuscountstype). | <a id="projectissuestatuscountsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | | <a id="projectissuestatuscountsclosedafter"></a>`closedAfter` | [`Time`](#time) | Issues closed after this date. | | <a id="projectissuestatuscountsclosedbefore"></a>`closedBefore` | [`Time`](#time) | Issues closed before this date. | +| <a id="projectissuestatuscountsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter for confidential issues. If "false", excludes confidential issues. If "true", returns only confidential issues. | | <a id="projectissuestatuscountscreatedafter"></a>`createdAfter` | [`Time`](#time) | Issues created after this date. | | <a id="projectissuestatuscountscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Issues created before this date. | | <a id="projectissuestatuscountsiid"></a>`iid` | [`String`](#string) | IID of the issue. For example, "1". | @@ -12747,7 +12953,7 @@ Returns [`IssueStatusCountsType`](#issuestatuscountstype). | <a id="projectissuestatuscountsmilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuestatuscountsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuestatuscountsnot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | -| <a id="projectissuestatuscountssearch"></a>`search` | [`String`](#string) | Search query for issue title or description. | +| <a id="projectissuestatuscountssearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuestatuscountstypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter issues by the given issue types. | | <a id="projectissuestatuscountsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Issues updated after this date. | | <a id="projectissuestatuscountsupdatedbefore"></a>`updatedBefore` | [`Time`](#time) | Issues updated before this date. | @@ -12772,11 +12978,13 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectissuesauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author of the issue. | | <a id="projectissuesclosedafter"></a>`closedAfter` | [`Time`](#time) | Issues closed after this date. | | <a id="projectissuesclosedbefore"></a>`closedBefore` | [`Time`](#time) | Issues closed before this date. | +| <a id="projectissuesconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter for confidential issues. If "false", excludes confidential issues. If "true", returns only confidential issues. | | <a id="projectissuescreatedafter"></a>`createdAfter` | [`Time`](#time) | Issues created after this date. | | <a id="projectissuescreatedbefore"></a>`createdBefore` | [`Time`](#time) | Issues created before this date. | | <a id="projectissuesepicid"></a>`epicId` | [`String`](#string) | ID of an epic associated with the issues, "none" and "any" values are supported. | | <a id="projectissuesiid"></a>`iid` | [`String`](#string) | IID of the issue. For example, "1". | | <a id="projectissuesiids"></a>`iids` | [`[String!]`](#string) | List of IIDs of issues. For example, `["1", "2"]`. | +| <a id="projectissuesincludesubepics"></a>`includeSubepics` | [`Boolean`](#boolean) | Whether to include subepics when filtering issues by epicId. | | <a id="projectissuesiterationid"></a>`iterationId` | [`[ID]`](#id) | List of iteration Global IDs applied to the issue. | | <a id="projectissuesiterationwildcardid"></a>`iterationWildcardId` | [`IterationWildcardId`](#iterationwildcardid) | Filter by iteration ID wildcard. | | <a id="projectissueslabelname"></a>`labelName` | [`[String]`](#string) | Labels applied to this issue. | @@ -12784,7 +12992,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectissuesmilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuesmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuesnot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | -| <a id="projectissuessearch"></a>`search` | [`String`](#string) | Search query for issue title or description. | +| <a id="projectissuessearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuessort"></a>`sort` | [`IssueSort`](#issuesort) | Sort issues by this criteria. | | <a id="projectissuesstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this issue. | | <a id="projectissuestypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter issues by the given issue types. | @@ -13196,6 +13404,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="projectvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="projectvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | +| <a id="projectvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | | <a id="projectvulnerabilitiesprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="projectvulnerabilitiesreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="projectvulnerabilitiesscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by VulnerabilityScanner.externalId. | @@ -13524,7 +13733,7 @@ Returns [`[String!]`](#string). ##### `Repository.paginatedTree` -Paginated tree of the repository. Available only when feature flag `paginated_tree_graphql_query` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. +Paginated tree of the repository. Available only when feature flag `paginated_tree_graphql_query` is enabled. This flag is enabled by default. Returns [`TreeConnection`](#treeconnection). @@ -13676,6 +13885,16 @@ Counts of requirements by their state. | <a id="runnerarchitecturedownloadlocation"></a>`downloadLocation` | [`String!`](#string) | Download location for the runner for the platform architecture. | | <a id="runnerarchitecturename"></a>`name` | [`String!`](#string) | Name of the runner platform architecture. | +### `RunnerPermissions` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="runnerpermissionsdeleterunner"></a>`deleteRunner` | [`Boolean!`](#boolean) | Indicates the user can perform `delete_runner` on this resource. | +| <a id="runnerpermissionsreadrunner"></a>`readRunner` | [`Boolean!`](#boolean) | Indicates the user can perform `read_runner` on this resource. | +| <a id="runnerpermissionsupdaterunner"></a>`updateRunner` | [`Boolean!`](#boolean) | Indicates the user can perform `update_runner` on this resource. | + ### `RunnerPlatform` #### Fields @@ -13850,6 +14069,7 @@ A Sentry error. | <a id="sentrydetailederrorgitlabcommitpath"></a>`gitlabCommitPath` | [`String`](#string) | Path to the GitLab page for the GitLab commit attributed to the error. | | <a id="sentrydetailederrorgitlabissuepath"></a>`gitlabIssuePath` | [`String`](#string) | URL of GitLab Issue. | | <a id="sentrydetailederrorid"></a>`id` | [`ID!`](#id) | ID (global ID) of the error. | +| <a id="sentrydetailederrorintegrated"></a>`integrated` | [`Boolean`](#boolean) | Error tracking backend. | | <a id="sentrydetailederrorlastreleaselastcommit"></a>`lastReleaseLastCommit` | [`String`](#string) | Commit the error was last seen. | | <a id="sentrydetailederrorlastreleaseshortversion"></a>`lastReleaseShortVersion` | [`String`](#string) | Release short version the error was last seen. | | <a id="sentrydetailederrorlastreleaseversion"></a>`lastReleaseVersion` | [`String`](#string) | Release version the error was last seen. | @@ -14709,8 +14929,10 @@ Represents a vulnerability. | <a id="vulnerabilityhassolutions"></a>`hasSolutions` | [`Boolean`](#boolean) | Indicates whether there is a solution available for this vulnerability. | | <a id="vulnerabilityid"></a>`id` | [`ID!`](#id) | GraphQL ID of the vulnerability. | | <a id="vulnerabilityidentifiers"></a>`identifiers` | [`[VulnerabilityIdentifier!]!`](#vulnerabilityidentifier) | Identifiers of the vulnerability. | +| <a id="vulnerabilitylinks"></a>`links` | [`[VulnerabilityLink!]!`](#vulnerabilitylink) | List of links associated with the vulnerability. | | <a id="vulnerabilitylocation"></a>`location` | [`VulnerabilityLocation`](#vulnerabilitylocation) | Location metadata for the vulnerability. Its fields depend on the type of security scan that found the vulnerability. | | <a id="vulnerabilitymergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request that fixes the vulnerability. | +| <a id="vulnerabilitymessage"></a>`message` | [`String`](#string) | Short text description of the vulnerability. This may include the finding's specific information. | | <a id="vulnerabilitynotes"></a>`notes` | [`NoteConnection!`](#noteconnection) | All notes on this noteable. (see [Connections](#connections)) | | <a id="vulnerabilityprimaryidentifier"></a>`primaryIdentifier` | [`VulnerabilityIdentifier`](#vulnerabilityidentifier) | Primary identifier of the vulnerability. | | <a id="vulnerabilityproject"></a>`project` | [`Project`](#project) | Project on which the vulnerability was found. | @@ -14956,6 +15178,17 @@ Represents an issue link of a vulnerability. | <a id="vulnerabilityissuelinkissue"></a>`issue` | [`Issue!`](#issue) | Issue attached to issue link. | | <a id="vulnerabilityissuelinklinktype"></a>`linkType` | [`VulnerabilityIssueLinkType!`](#vulnerabilityissuelinktype) | Type of the issue link. | +### `VulnerabilityLink` + +Represents a link related to a vulnerability. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilitylinkname"></a>`name` | [`String`](#string) | Name of the link. | +| <a id="vulnerabilitylinkurl"></a>`url` | [`String!`](#string) | URL of the link. | + ### `VulnerabilityLocationContainerScanning` Represents the location of a vulnerability found by a container security scan. @@ -15345,10 +15578,10 @@ Values for sorting runners. | Value | Description | | ----- | ----------- | -| <a id="cirunnerstatusactive"></a>`ACTIVE` | A runner that is active. | -| <a id="cirunnerstatusnot_connected"></a>`NOT_CONNECTED` | A runner that is not connected. | -| <a id="cirunnerstatusoffline"></a>`OFFLINE` | A runner that is offline. | -| <a id="cirunnerstatusonline"></a>`ONLINE` | A runner that is online. | +| <a id="cirunnerstatusactive"></a>`ACTIVE` | A runner that is not paused. | +| <a id="cirunnerstatusnot_connected"></a>`NOT_CONNECTED` | A runner that has never contacted this instance. | +| <a id="cirunnerstatusoffline"></a>`OFFLINE` | A runner that has not contacted this instance within the last 2 hours. | +| <a id="cirunnerstatusonline"></a>`ONLINE` | A runner that contacted this instance within the last 2 hours. | | <a id="cirunnerstatuspaused"></a>`PAUSED` | A runner that is paused. | ### `CiRunnerType` @@ -15425,6 +15658,7 @@ Conan file types. | <a id="containerexpirationpolicyolderthanenumfourteen_days"></a>`FOURTEEN_DAYS` | 14 days until tags are automatically removed. | | <a id="containerexpirationpolicyolderthanenumninety_days"></a>`NINETY_DAYS` | 90 days until tags are automatically removed. | | <a id="containerexpirationpolicyolderthanenumseven_days"></a>`SEVEN_DAYS` | 7 days until tags are automatically removed. | +| <a id="containerexpirationpolicyolderthanenumsixty_days"></a>`SIXTY_DAYS` | 60 days until tags are automatically removed. | | <a id="containerexpirationpolicyolderthanenumthirty_days"></a>`THIRTY_DAYS` | 30 days until tags are automatically removed. | ### `ContainerRepositoryCleanupStatus` @@ -15750,6 +15984,14 @@ State of a GitLab issue or merge request. | <a id="issuablestatelocked"></a>`locked` | Discussion has been locked. | | <a id="issuablestateopened"></a>`opened` | In open state. | +### `IssueCreationIterationWildcardId` + +Iteration ID wildcard values for issue creation. + +| Value | Description | +| ----- | ----------- | +| <a id="issuecreationiterationwildcardidcurrent"></a>`CURRENT` | Current iteration. | + ### `IssueSort` Values for sorting issues. @@ -16013,7 +16255,7 @@ Milestone ID wildcard values. | <a id="milestonewildcardidany"></a>`ANY` | Milestone is assigned. | | <a id="milestonewildcardidnone"></a>`NONE` | No milestone is assigned. | | <a id="milestonewildcardidstarted"></a>`STARTED` | Milestone assigned is open and started (start date <= today). | -| <a id="milestonewildcardidupcoming"></a>`UPCOMING` | Milestone assigned is due closest in the future (due date > today). | +| <a id="milestonewildcardidupcoming"></a>`UPCOMING` | Milestone assigned is due in the future (due date > today). | ### `MoveType` @@ -16455,6 +16697,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumregistration_enabled_callout"></a>`REGISTRATION_ENABLED_CALLOUT` | Callout feature name for registration_enabled_callout. | | <a id="usercalloutfeaturenameenumsecurity_configuration_devops_alert"></a>`SECURITY_CONFIGURATION_DEVOPS_ALERT` | Callout feature name for security_configuration_devops_alert. | | <a id="usercalloutfeaturenameenumsecurity_configuration_upgrade_banner"></a>`SECURITY_CONFIGURATION_UPGRADE_BANNER` | Callout feature name for security_configuration_upgrade_banner. | +| <a id="usercalloutfeaturenameenumsecurity_newsletter_callout"></a>`SECURITY_NEWSLETTER_CALLOUT` | Callout feature name for security_newsletter_callout. | | <a id="usercalloutfeaturenameenumsuggest_pipeline"></a>`SUGGEST_PIPELINE` | Callout feature name for suggest_pipeline. | | <a id="usercalloutfeaturenameenumsuggest_popover_dismissed"></a>`SUGGEST_POPOVER_DISMISSED` | Callout feature name for suggest_popover_dismissed. | | <a id="usercalloutfeaturenameenumtabs_position_highlight"></a>`TABS_POSITION_HIGHLIGHT` | Callout feature name for tabs_position_highlight. | @@ -16646,6 +16889,12 @@ A `AnalyticsDevopsAdoptionEnabledNamespaceID` is a global ID. It is encoded as a An example `AnalyticsDevopsAdoptionEnabledNamespaceID` is: `"gid://gitlab/Analytics::DevopsAdoption::EnabledNamespace/1"`. +### `AuditEventsExternalAuditEventDestinationID` + +A `AuditEventsExternalAuditEventDestinationID` is a global ID. It is encoded as a string. + +An example `AuditEventsExternalAuditEventDestinationID` is: `"gid://gitlab/AuditEvents::ExternalAuditEventDestination/1"`. + ### `AwardableID` A `AwardableID` is a global ID. It is encoded as a string. @@ -16732,6 +16981,12 @@ A `CustomEmojiID` is a global ID. It is encoded as a string. An example `CustomEmojiID` is: `"gid://gitlab/CustomEmoji/1"`. +### `CustomerRelationsContactID` + +A `CustomerRelationsContactID` is a global ID. It is encoded as a string. + +An example `CustomerRelationsContactID` is: `"gid://gitlab/CustomerRelations::Contact/1"`. + ### `CustomerRelationsOrganizationID` A `CustomerRelationsOrganizationID` is a global ID. It is encoded as a string. @@ -17906,6 +18161,7 @@ Represents an escalation rule. | <a id="negatedissuefilterinputmilestonetitle"></a>`milestoneTitle` | [`[String!]`](#string) | Milestone not applied to this issue. | | <a id="negatedissuefilterinputmilestonewildcardid"></a>`milestoneWildcardId` | [`NegatedMilestoneWildcardId`](#negatedmilestonewildcardid) | Filter by negated milestone wildcard values. | | <a id="negatedissuefilterinputmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="negatedissuefilterinputtypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filters out issues by the given issue types. | | <a id="negatedissuefilterinputweight"></a>`weight` | [`String`](#string) | Weight not applied to the issue. | ### `OncallRotationActivePeriodInputType` @@ -18057,3 +18313,23 @@ A time-frame defined as a closed inclusive range of two dates. | <a id="vulnerabilityidentifierinputexternaltype"></a>`externalType` | [`String`](#string) | External type of the vulnerability identifier. | | <a id="vulnerabilityidentifierinputname"></a>`name` | [`String!`](#string) | Name of the vulnerability identifier. | | <a id="vulnerabilityidentifierinputurl"></a>`url` | [`String!`](#string) | URL of the vulnerability identifier. | + +### `VulnerabilityScannerInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityscannerinputid"></a>`id` | [`String!`](#string) | Unique ID that identifies the scanner. | +| <a id="vulnerabilityscannerinputname"></a>`name` | [`String!`](#string) | Human readable value that identifies the analyzer, not required to be unique. | +| <a id="vulnerabilityscannerinputurl"></a>`url` | [`String!`](#string) | Link to more information about the analyzer. | +| <a id="vulnerabilityscannerinputvendor"></a>`vendor` | [`VulnerabilityScannerVendorInput`](#vulnerabilityscannervendorinput) | Information about vendor/maintainer of the scanner. | +| <a id="vulnerabilityscannerinputversion"></a>`version` | [`String!`](#string) | Version of the scanner. | + +### `VulnerabilityScannerVendorInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="vulnerabilityscannervendorinputname"></a>`name` | [`String!`](#string) | Name of the vendor/maintainer. | diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index 83373368fc6..81ca45c4531 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -250,7 +250,7 @@ Parameters: NOTE: `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added -through the ["Add existing Kubernetes cluster"](../user/project/clusters/add_remove_clusters.md#add-existing-cluster) option or +through the ["Add existing Kubernetes cluster"](../user/project/clusters/add_existing_cluster.md) option or through the ["Add existing cluster to group"](#add-existing-cluster-to-group) endpoint. Example request: diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index 0a431cfdfbf..212a62516f1 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -57,8 +57,12 @@ GET /groups/:id/export/download | `id` | integer/string | yes | ID of the group owned by the authenticated user | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name \ - --remote-name "https://gitlab.example.com/api/v4/groups/1/export/download" +group=1 +token=secret +curl --request GET\ + --header "PRIVATE-TOKEN: ${token}" \ + --output download_group_${group}.tar.gz \ + "https://gitlab.example.com/api/v4/groups/${group}/export/download" ``` ```shell diff --git a/doc/api/group_repository_storage_moves.md b/doc/api/group_repository_storage_moves.md index a893bffb1f5..9d4120ec355 100644 --- a/doc/api/group_repository_storage_moves.md +++ b/doc/api/group_repository_storage_moves.md @@ -10,7 +10,7 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53016) in GitLab 13.9. Group repositories can be moved between storages. This API can help you when -[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrate-to-gitaly-cluster), for +[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrating-to-gitaly-cluster), for example, or to migrate a [group wiki](../user/project/wiki/index.md#group-wikis). As group repository storage moves are processed, they transition through different states. Values diff --git a/doc/api/groups.md b/doc/api/groups.md index bd4c7de567c..7efecfc2c9c 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -830,7 +830,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ## Transfer project to group -Transfer a project to the Group namespace. Available only to instance administrators, although an [alternative API endpoint](projects.md#transfer-a-project-to-a-new-namespace) is available which does not require instance administrator access. Transferring projects may fail when tagged packages exist in the project's repository. +Transfer a project to the Group namespace. Available only to instance administrators, although an [alternative API endpoint](projects.md#transfer-a-project-to-a-new-namespace) is available which does not require instance administrator role. Transferring projects may fail when tagged packages exist in the project's repository. ```plaintext POST /groups/:id/projects/:project_id diff --git a/doc/api/instance_clusters.md b/doc/api/instance_clusters.md index 4e0ec3bd433..58f88b26bc4 100644 --- a/doc/api/instance_clusters.md +++ b/doc/api/instance_clusters.md @@ -8,10 +8,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36001) in GitLab 13.2. -Instance-level Kubernetes clusters allow you to connect a Kubernetes cluster to the GitLab instance, which enables you to use the same cluster across multiple projects. [More information](../user/instance/clusters/index.md) +With [instance-level Kubernetes clusters](../user/instance/clusters/index.md), +you can connect a Kubernetes cluster to the GitLab instance and use the same cluster across all of +the projects within your instance. NOTE: -Users need administrator access to use these endpoints. +Users need the Administrator role to use these endpoints. ## List instance clusters @@ -240,7 +242,7 @@ Parameters: NOTE: `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added -through the [Add existing Kubernetes cluster](../user/project/clusters/add_remove_clusters.md#add-existing-cluster) option or +through the [Add existing Kubernetes cluster](../user/project/clusters/add_existing_cluster.md) option or through the [Add existing instance cluster](#add-existing-instance-cluster) endpoint. Example request: diff --git a/doc/api/integrations.md b/doc/api/integrations.md new file mode 100644 index 00000000000..3c649e8d044 --- /dev/null +++ b/doc/api/integrations.md @@ -0,0 +1,1566 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Integrations API **(FREE)** + +This API enables you to work with external services that integrate with GitLab. + +NOTE: +In GitLab 14.4, the `services` endpoint was [renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/334500) to `integrations`. +Calls to the Integrations API can be made to both `/projects/:id/services` and `/projects/:id/integrations`. +The examples in this document refer to the endpoint at `/projects/:id/integrations`. + +This API requires an access token with the [Maintainer or Owner role](../user/permissions.md). + +## List all active integrations + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21330) in GitLab 12.7. + +Get a list of all active project integrations. + +```plaintext +GET /projects/:id/integrations +``` + +Example response: + +```json +[ + { + "id": 75, + "title": "Jenkins CI", + "slug": "jenkins", + "created_at": "2019-11-20T11:20:25.297Z", + "updated_at": "2019-11-20T12:24:37.498Z", + "active": true, + "commit_events": true, + "push_events": true, + "issues_events": true, + "confidential_issues_events": true, + "merge_requests_events": true, + "tag_push_events": false, + "note_events": true, + "confidential_note_events": true, + "pipeline_events": true, + "wiki_page_events": true, + "job_events": true, + "comment_on_event_enabled": true + }, + { + "id": 76, + "title": "Alerts endpoint", + "slug": "alerts", + "created_at": "2019-11-20T11:20:25.297Z", + "updated_at": "2019-11-20T12:24:37.498Z", + "active": true, + "commit_events": true, + "push_events": true, + "issues_events": true, + "confidential_issues_events": true, + "merge_requests_events": true, + "tag_push_events": true, + "note_events": true, + "confidential_note_events": true, + "pipeline_events": true, + "wiki_page_events": true, + "job_events": true, + "comment_on_event_enabled": true + } +] +``` + +## Asana + +Add commit messages as comments to Asana tasks. + +See also the [Asana integration documentation](../user/project/integrations/asana.md). + +### Create/Edit Asana integration + +Set Asana integration for a project. + +```plaintext +PUT /projects/:id/integrations/asana +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `api_key` | string | true | User API token. User must have access to task. All comments are attributed to this user. | +| `restrict_to_branch` | string | false | Comma-separated list of branches to be are automatically inspected. Leave blank to include all branches. | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Asana integration + +Delete Asana integration for a project. + +```plaintext +DELETE /projects/:id/integrations/asana +``` + +### Get Asana integration settings + +Get Asana integration settings for a project. + +```plaintext +GET /projects/:id/integrations/asana +``` + +## Assembla + +Project Management Software (Source Commits Endpoint) + +### Create/Edit Assembla integration + +Set Assembla integration for a project. + +```plaintext +PUT /projects/:id/integrations/assembla +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | The authentication token +| `subdomain` | string | false | The subdomain setting | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Assembla integration + +Delete Assembla integration for a project. + +```plaintext +DELETE /projects/:id/integrations/assembla +``` + +### Get Assembla integration settings + +Get Assembla integration settings for a project. + +```plaintext +GET /projects/:id/integrations/assembla +``` + +## Atlassian Bamboo CI + +A continuous integration and build server + +### Create/Edit Atlassian Bamboo CI integration + +Set Atlassian Bamboo CI integration for a project. + +> You must set up automatic revision labeling and a repository trigger in Bamboo. + +```plaintext +PUT /projects/:id/integrations/bamboo +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `bamboo_url` | string | true | Bamboo root URL. For example, `https://bamboo.example.com`. | +| `build_key` | string | true | Bamboo build plan key like KEY | +| `username` | string | true | A user with API access, if applicable | +| `password` | string | true | Password of the user | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Atlassian Bamboo CI integration + +Delete Atlassian Bamboo CI integration for a project. + +```plaintext +DELETE /projects/:id/integrations/bamboo +``` + +### Get Atlassian Bamboo CI integration settings + +Get Atlassian Bamboo CI integration settings for a project. + +```plaintext +GET /projects/:id/integrations/bamboo +``` + +## Bugzilla + +Bugzilla Issue Tracker + +### Create/Edit Bugzilla integration + +Set Bugzilla integration for a project. + +```plaintext +PUT /projects/:id/integrations/bugzilla +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `new_issue_url` | string | true | New Issue URL | +| `issues_url` | string | true | Issue URL | +| `project_url` | string | true | Project URL | +| `description` | string | false | Description | +| `title` | string | false | Title | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Bugzilla integration + +Delete Bugzilla integration for a project. + +```plaintext +DELETE /projects/:id/integrations/bugzilla +``` + +### Get Bugzilla integration settings + +Get Bugzilla integration settings for a project. + +```plaintext +GET /projects/:id/integrations/bugzilla +``` + +## Buildkite + +Continuous integration and deployments + +### Create/Edit Buildkite integration + +Set Buildkite integration for a project. + +```plaintext +PUT /projects/:id/integrations/buildkite +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | Buildkite project GitLab token | +| `project_url` | string | true | Pipeline URL. For example, `https://buildkite.com/example/pipeline` | +| `enable_ssl_verification` | boolean | false | DEPRECATED: This parameter has no effect since SSL verification is always enabled | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Buildkite integration + +Delete Buildkite integration for a project. + +```plaintext +DELETE /projects/:id/integrations/buildkite +``` + +### Get Buildkite integration settings + +Get Buildkite integration settings for a project. + +```plaintext +GET /projects/:id/integrations/buildkite +``` + +## Campfire + +Send notifications about push events to Campfire chat rooms. +[New users can no longer sign up for Campfire](https://basecamp.com/retired/campfire). + +### Create/Edit Campfire integration + +Set Campfire integration for a project. + +```plaintext +PUT /projects/:id/integrations/campfire +``` + +Parameters: + +| Parameter | Type | Required | Description | +|---------------|---------|----------|---------------------------------------------------------------------------------------------| +| `token` | string | true | Campfire API token. To find it, log into Campfire and select **My info**. | +| `subdomain` | string | false | Campfire subdomain. Text between `https://` and `.campfirenow.com` when you're logged in. | +| `room` | string | false | Campfire room. The last part of the URL when you're in a room. | +| `push_events` | boolean | false | Enable notifications for push events. | + +### Delete Campfire integration + +Delete Campfire integration for a project. + +```plaintext +DELETE /projects/:id/integrations/campfire +``` + +### Get Campfire integration settings + +Get Campfire integration settings for a project. + +```plaintext +GET /projects/:id/integrations/campfire +``` + +## Datadog + +Datadog system monitoring. + +### Create/Edit Datadog integration + +Set Datadog integration for a project. + +```plaintext +PUT /projects/:id/integrations/datadog +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `api_key` | string | true | API key used for authentication with Datadog | +| `api_url` | string | false | (Advanced) Define the full URL for your Datadog site directly | +| `datadog_site` | string | false | Choose the Datadog site to send data to. Set to `datadoghq.eu` to send data to the EU site | +| `datadog_service` | string | false | Name of this GitLab instance that all data will be tagged with | +| `datadog_env` | string | false | The environment tag that traces will be tagged with | + +### Delete Datadog integration + +Delete Datadog integration for a project. + +```plaintext +DELETE /projects/:id/integrations/datadog +``` + +### Get Datadog integration settings + +Get Datadog integration settings for a project. + +```plaintext +GET /projects/:id/integrations/datadog +``` + +## Unify Circuit + +Unify Circuit RTC and collaboration tool. + +### Create/Edit Unify Circuit integration + +Set Unify Circuit integration for a project. + +```plaintext +PUT /projects/:id/integrations/unify-circuit +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | The Unify Circuit webhook. For example, `https://circuit.com/rest/v2/webhooks/incoming/...`. | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | + +### Delete Unify Circuit integration + +Delete Unify Circuit integration for a project. + +```plaintext +DELETE /projects/:id/integrations/unify-circuit +``` + +### Get Unify Circuit integration settings + +Get Unify Circuit integration settings for a project. + +```plaintext +GET /projects/:id/integrations/unify-circuit +``` + +## Webex Teams + +Webex Teams collaboration tool. + +### Create/Edit Webex Teams integration + +Set Webex Teams integration for a project. + +```plaintext +PUT /projects/:id/integrations/webex-teams +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | The Webex Teams webhook. For example, `https://api.ciscospark.com/v1/webhooks/incoming/...`. | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | + +### Delete Webex Teams integration + +Delete Webex Teams integration for a project. + +```plaintext +DELETE /projects/:id/integrations/webex-teams +``` + +### Get Webex Teams integration settings + +Get Webex Teams integration settings for a project. + +```plaintext +GET /projects/:id/integrations/webex-teams +``` + +## Custom Issue Tracker + +Custom issue tracker + +### Create/Edit Custom Issue Tracker integration + +Set Custom Issue Tracker integration for a project. + +```plaintext +PUT /projects/:id/integrations/custom-issue-tracker +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `new_issue_url` | string | true | New Issue URL | +| `issues_url` | string | true | Issue URL | +| `project_url` | string | true | Project URL | +| `description` | string | false | Description | +| `title` | string | false | Title | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Custom Issue Tracker integration + +Delete Custom Issue Tracker integration for a project. + +```plaintext +DELETE /projects/:id/integrations/custom-issue-tracker +``` + +### Get Custom Issue Tracker integration settings + +Get Custom Issue Tracker integration settings for a project. + +```plaintext +GET /projects/:id/integrations/custom-issue-tracker +``` + +## Discord + +Send notifications about project events to a Discord channel. + +### Create/Edit Discord integration + +Set Discord integration for a project. + +```plaintext +PUT /projects/:id/integrations/discord +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | Discord webhook. For example, `https://discord.com/api/webhooks/…` | + +### Delete Discord integration + +Delete Discord integration for a project. + +```plaintext +DELETE /projects/:id/integrations/discord +``` + +### Get Discord integration settings + +Get Discord integration settings for a project. + +```plaintext +GET /projects/:id/integrations/discord +``` + +## Drone CI + +Drone is a Continuous Integration platform built on Docker, written in Go + +### Create/Edit Drone CI integration + +Set Drone CI integration for a project. + +```plaintext +PUT /projects/:id/integrations/drone-ci +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | Drone CI project specific token | +| `drone_url` | string | true | `http://drone.example.com` | +| `enable_ssl_verification` | boolean | false | Enable SSL verification | +| `push_events` | boolean | false | Enable notifications for push events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | + +### Delete Drone CI integration + +Delete Drone CI integration for a project. + +```plaintext +DELETE /projects/:id/integrations/drone-ci +``` + +### Get Drone CI integration settings + +Get Drone CI integration settings for a project. + +```plaintext +GET /projects/:id/integrations/drone-ci +``` + +## Emails on Push + +Email the commits and diff of each push to a list of recipients. + +### Create/Edit Emails on Push integration + +Set Emails on Push integration for a project. + +```plaintext +PUT /projects/:id/integrations/emails-on-push +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `recipients` | string | true | Emails separated by whitespace | +| `disable_diffs` | boolean | false | Disable code diffs | +| `send_from_committer_email` | boolean | false | Send from committer | +| `push_events` | boolean | false | Enable notifications for push events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". Notifications are always fired for tag pushes. The default value is "all" | + +### Delete Emails on Push integration + +Delete Emails on Push integration for a project. + +```plaintext +DELETE /projects/:id/integrations/emails-on-push +``` + +### Get Emails on Push integration settings + +Get Emails on Push integration settings for a project. + +```plaintext +GET /projects/:id/integrations/emails-on-push +``` + +## Engineering Workflow Management (EWM) + +Use IBM Engineering Workflow Management (EWM) as a project's issue tracker. + +### Create/Edit EWM integration + +Set EWM integration for a project. + +```plaintext +PUT /projects/:id/integrations/ewm +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `new_issue_url` | string | true | The URL to create an issue in EWM | +| `project_url` | string | true | The URL to the project in EWM | +| `issues_url` | string | true | The URL to view an issue in EWM. Must contain `:id` | + +### Delete EWM integration + +Delete EWM integration for a project. + +```plaintext +DELETE /projects/:id/integrations/ewm +``` + +### Get EWM integration settings + +Get EWM integration settings for a project. + +```plaintext +GET /projects/:id/integrations/ewm +``` + +## Confluence integration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220934) in GitLab 13.2. + +Replaces the link to the internal wiki with a link to a Confluence Cloud Workspace. + +### Create/Edit Confluence integration + +Set Confluence integration for a project. + +```plaintext +PUT /projects/:id/integrations/confluence +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `confluence_url` | string | true | The URL of the Confluence Cloud Workspace hosted on atlassian.net. | + +### Delete Confluence integration + +Delete Confluence integration for a project. + +```plaintext +DELETE /projects/:id/integrations/confluence +``` + +### Get Confluence integration settings + +Get Confluence integration settings for a project. + +```plaintext +GET /projects/:id/integrations/confluence +``` + +## External wiki + +Replaces the link to the internal wiki with a link to an external wiki. + +### Create/Edit External wiki integration + +Set External wiki integration for a project. + +```plaintext +PUT /projects/:id/integrations/external-wiki +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `external_wiki_url` | string | true | The URL of the external wiki | + +### Delete External wiki integration + +Delete External wiki integration for a project. + +```plaintext +DELETE /projects/:id/integrations/external-wiki +``` + +### Get External wiki integration settings + +Get External wiki integration settings for a project. + +```plaintext +GET /projects/:id/integrations/external-wiki +``` + +## Flowdock + +Flowdock is a ChatOps application for collaboration in software engineering +companies. You can send notifications from GitLab events to Flowdock flows. +For integration instructions, see the [Flowdock documentation](https://www.flowdock.com/help/gitlab). + +### Create/Edit Flowdock integration + +Set Flowdock integration for a project. + +```plaintext +PUT /projects/:id/integrations/flowdock +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | Flowdock Git source token | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Flowdock integration + +Delete Flowdock integration for a project. + +```plaintext +DELETE /projects/:id/integrations/flowdock +``` + +### Get Flowdock integration settings + +Get Flowdock integration settings for a project. + +```plaintext +GET /projects/:id/integrations/flowdock +``` + +## GitHub **(PREMIUM)** + +Code collaboration software. + +### Create/Edit GitHub integration + +Set GitHub integration for a project. + +```plaintext +PUT /projects/:id/integrations/github +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | GitHub API token with `repo:status` OAuth scope | +| `repository_url` | string | true | GitHub repository URL | +| `static_context` | boolean | false | Append instance name instead of branch to [status check name](../user/project/integrations/github.md#static--dynamic-status-check-names) | + +### Delete GitHub integration + +Delete GitHub integration for a project. + +```plaintext +DELETE /projects/:id/integrations/github +``` + +### Get GitHub integration settings + +Get GitHub integration settings for a project. + +```plaintext +GET /projects/:id/integrations/github +``` + +## Hangouts Chat + +Google Workspace team collaboration tool. + +### Create/Edit Hangouts Chat integration + +Set Hangouts Chat integration for a project. + +```plaintext +PUT /projects/:id/integrations/hangouts-chat +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | The Hangouts Chat webhook. For example, `https://chat.googleapis.com/v1/spaces...`. | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | + +### Delete Hangouts Chat integration + +Delete Hangouts Chat integration for a project. + +```plaintext +DELETE /projects/:id/integrations/hangouts-chat +``` + +### Get Hangouts Chat integration settings + +Get Hangouts Chat integration settings for a project. + +```plaintext +GET /projects/:id/integrations/hangouts-chat +``` + +## Irker (IRC gateway) + +Send IRC messages, on update, to a list of recipients through an irker gateway. + +For more information, see the [irker integration documentation](../user/project/integrations/irker.md). + +### Create/Edit Irker (IRC gateway) integration + +Set Irker (IRC gateway) integration for a project. + +```plaintext +PUT /projects/:id/integrations/irker +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `recipients` | string | true | Recipients/channels separated by whitespaces | +| `default_irc_uri` | string | false | `irc://irc.network.net:6697/` | +| `server_host` | string | false | localhost | +| `server_port` | integer | false | 6659 | +| `colorize_messages` | boolean | false | Colorize messages | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Irker (IRC gateway) integration + +Delete Irker (IRC gateway) integration for a project. + +```plaintext +DELETE /projects/:id/integrations/irker +``` + +### Get Irker (IRC gateway) integration settings + +Get Irker (IRC gateway) integration settings for a project. + +```plaintext +GET /projects/:id/integrations/irker +``` + +## Jira + +Jira issue tracker. + +### Get Jira integration settings + +Get Jira integration settings for a project. + +```plaintext +GET /projects/:id/integrations/jira +``` + +### Create/Edit Jira integration + +Set Jira integration for a project. + +```plaintext +PUT /projects/:id/integrations/jira +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `url` | string | yes | The URL to the Jira project which is being linked to this GitLab project. For example, `https://jira.example.com`. | +| `api_url` | string | no | The base URL to the Jira instance API. Web URL value is used if not set. For example, `https://jira-api.example.com`. | +| `username` | string | yes | The username of the user created to be used with GitLab/Jira. | +| `password` | string | yes | The password of the user created to be used with GitLab/Jira. | +| `active` | boolean | no | Activates or deactivates the integration. Defaults to false (deactivated). | +| `jira_issue_transition_automatic` | boolean | no | Enable [automatic issue transitions](../integration/jira/issues.md#automatic-issue-transitions). Takes precedence over `jira_issue_transition_id` if enabled. Defaults to `false` | +| `jira_issue_transition_id` | string | no | The ID of one or more transitions for [custom issue transitions](../integration/jira/issues.md#custom-issue-transitions). Ignored if `jira_issue_transition_automatic` is enabled. Defaults to a blank string, which disables custom transitions. | +| `commit_events` | boolean | false | Enable notifications for commit events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `comment_on_event_enabled` | boolean | false | Enable comments inside Jira issues on each GitLab event (commit / merge request) | + +### Delete Jira integration + +Remove all previously Jira integrations from a project. + +```plaintext +DELETE /projects/:id/integrations/jira +``` + +## Slack Slash Commands + +Ability to receive slash commands from a Slack chat instance. + +### Get Slack Slash Command integration settings + +Get Slack Slash Command integration settings for a project. + +```plaintext +GET /projects/:id/integrations/slack-slash-commands +``` + +Example response: + +```json +{ + "id": 4, + "title": "Slack slash commands", + "slug": "slack-slash-commands", + "created_at": "2017-06-27T05:51:39-07:00", + "updated_at": "2017-06-27T05:51:39-07:00", + "active": true, + "push_events": true, + "issues_events": true, + "confidential_issues_events": true, + "merge_requests_events": true, + "tag_push_events": true, + "note_events": true, + "job_events": true, + "pipeline_events": true, + "comment_on_event_enabled": false, + "properties": { + "token": "<your_access_token>" + } +} +``` + +### Create/Edit Slack Slash Commands integration + +Set Slack Slash Command for a project. + +```plaintext +PUT /projects/:id/integrations/slack-slash-commands +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | yes | The Slack token | + +### Delete Slack Slash Command integration + +Delete Slack Slash Command integration for a project. + +```plaintext +DELETE /projects/:id/integrations/slack-slash-commands +``` + +## Mattermost Slash Commands + +Ability to receive slash commands from a Mattermost chat instance. + +### Get Mattermost Slash Command integration settings + +Get Mattermost Slash Command integration settings for a project. + +```plaintext +GET /projects/:id/integrations/mattermost-slash-commands +``` + +### Create/Edit Mattermost Slash Command integration + +Set Mattermost Slash Command for a project. + +```plaintext +PUT /projects/:id/integrations/mattermost-slash-commands +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | yes | The Mattermost token | +| `username` | string | no | The username to use to post the message | + +### Delete Mattermost Slash Command integration + +Delete Mattermost Slash Command integration for a project. + +```plaintext +DELETE /projects/:id/integrations/mattermost-slash-commands +``` + +## Packagist + +Update your project on Packagist (the main Composer repository) when commits or tags are pushed to GitLab. + +### Create/Edit Packagist integration + +Set Packagist integration for a project. + +```plaintext +PUT /projects/:id/integrations/packagist +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `username` | string | yes | The username of a Packagist account | +| `token` | string | yes | API token to the Packagist server | +| `server` | boolean | no | URL of the Packagist server. Leave blank for default: <https://packagist.org> | +| `push_events` | boolean | false | Enable notifications for push events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | + +### Delete Packagist integration + +Delete Packagist integration for a project. + +```plaintext +DELETE /projects/:id/integrations/packagist +``` + +### Get Packagist integration settings + +Get Packagist integration settings for a project. + +```plaintext +GET /projects/:id/integrations/packagist +``` + +## Pipeline-Emails + +Get emails for GitLab CI/CD pipelines. + +### Create/Edit Pipeline-Emails integration + +Set Pipeline-Emails integration for a project. + +```plaintext +PUT /projects/:id/integrations/pipelines-email +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `recipients` | string | yes | Comma-separated list of recipient email addresses | +| `add_pusher` | boolean | no | Add pusher to recipients list | +| `notify_only_broken_pipelines` | boolean | no | Notify only broken pipelines | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected. The default value is "default" | +| `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28271)) | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | + +### Delete Pipeline-Emails integration + +Delete Pipeline-Emails integration for a project. + +```plaintext +DELETE /projects/:id/integrations/pipelines-email +``` + +### Get Pipeline-Emails integration settings + +Get Pipeline-Emails integration settings for a project. + +```plaintext +GET /projects/:id/integrations/pipelines-email +``` + +## Pivotal Tracker + +Add commit messages as comments to Pivotal Tracker stories. + +See also the [Pivotal Tracker integration documentation](../user/project/integrations/pivotal_tracker.md). + +### Create/Edit Pivotal Tracker integration + +Set Pivotal Tracker integration for a project. + +```plaintext +PUT /projects/:id/integrations/pivotaltracker +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | The Pivotal Tracker token | +| `restrict_to_branch` | boolean | false | Comma-separated list of branches to automatically inspect. Leave blank to include all branches. | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Pivotal Tracker integration + +Delete Pivotal Tracker integration for a project. + +```plaintext +DELETE /projects/:id/integrations/pivotaltracker +``` + +### Get Pivotal Tracker integration settings + +Get Pivotal Tracker integration settings for a project. + +```plaintext +GET /projects/:id/integrations/pivotaltracker +``` + +## Prometheus + +Prometheus is a powerful time-series monitoring service. + +### Create/Edit Prometheus integration + +Set Prometheus integration for a project. + +```plaintext +PUT /projects/:id/integrations/prometheus +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `api_url` | string | true | Prometheus API Base URL. For example, `http://prometheus.example.com/`. | +| `google_iap_audience_client_id` | string | false | Client ID of the IAP secured resource (looks like IAP_CLIENT_ID.apps.googleusercontent.com) | +| `google_iap_service_account_json` | string | false | `credentials.json` file for your service account, like { "type": "service_account", "project_id": ... } | + +### Delete Prometheus integration + +Delete Prometheus integration for a project. + +```plaintext +DELETE /projects/:id/integrations/prometheus +``` + +### Get Prometheus integration settings + +Get Prometheus integration settings for a project. + +```plaintext +GET /projects/:id/integrations/prometheus +``` + +## Pushover + +Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop. + +### Create/Edit Pushover integration + +Set Pushover integration for a project. + +```plaintext +PUT /projects/:id/integrations/pushover +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `api_key` | string | true | Your application key | +| `user_key` | string | true | Your user key | +| `priority` | string | true | The priority | +| `device` | string | false | Leave blank for all active devices | +| `sound` | string | false | The sound of the notification | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Pushover integration + +Delete Pushover integration for a project. + +```plaintext +DELETE /projects/:id/integrations/pushover +``` + +### Get Pushover integration settings + +Get Pushover integration settings for a project. + +```plaintext +GET /projects/:id/integrations/pushover +``` + +## Redmine + +Redmine issue tracker + +### Create/Edit Redmine integration + +Set Redmine integration for a project. + +```plaintext +PUT /projects/:id/integrations/redmine +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `new_issue_url` | string | true | New Issue URL | +| `project_url` | string | true | Project URL | +| `issues_url` | string | true | Issue URL | +| `description` | string | false | Description | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete Redmine integration + +Delete Redmine integration for a project. + +```plaintext +DELETE /projects/:id/integrations/redmine +``` + +### Get Redmine integration settings + +Get Redmine integration settings for a project. + +```plaintext +GET /projects/:id/integrations/redmine +``` + +## Slack notifications + +Receive event notifications in Slack + +### Create/Edit Slack integration + +Set Slack integration for a project. + +```plaintext +PUT /projects/:id/integrations/slack +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | `https://hooks.slack.com/services/...` | +| `username` | string | false | username | +| `channel` | string | false | Default channel to use if others are not configured | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `commit_events` | boolean | false | Enable notifications for commit events | +| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `deployment_channel` | string | false | The name of the channel to receive deployment events notifications | +| `deployment_events` | boolean | false | Enable notifications for deployment events | +| `issue_channel` | string | false | The name of the channel to receive issues events notifications | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `job_events` | boolean | false | Enable notifications for job events | +| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `note_channel` | string | false | The name of the channel to receive note events notifications | +| `note_events` | boolean | false | Enable notifications for note events | +| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `push_channel` | string | false | The name of the channel to receive push events notifications | +| `push_events` | boolean | false | Enable notifications for push events | +| `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | + +### Delete Slack integration + +Delete Slack integration for a project. + +```plaintext +DELETE /projects/:id/integrations/slack +``` + +### Get Slack integration settings + +Get Slack integration settings for a project. + +```plaintext +GET /projects/:id/integrations/slack +``` + +## Microsoft Teams + +Group Chat Software + +### Create/Edit Microsoft Teams integration + +Set Microsoft Teams integration for a project. + +```plaintext +PUT /projects/:id/integrations/microsoft-teams +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | The Microsoft Teams webhook. For example, `https://outlook.office.com/webhook/...` | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | + +### Delete Microsoft Teams integration + +Delete Microsoft Teams integration for a project. + +```plaintext +DELETE /projects/:id/integrations/microsoft-teams +``` + +### Get Microsoft Teams integration settings + +Get Microsoft Teams integration settings for a project. + +```plaintext +GET /projects/:id/integrations/microsoft-teams +``` + +## Mattermost notifications + +Receive event notifications in Mattermost + +### Create/Edit Mattermost notifications integration + +Set Mattermost notifications integration for a project. + +```plaintext +PUT /projects/:id/integrations/mattermost +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `webhook` | string | true | The Mattermost webhook. For example, `http://mattermost_host/hooks/...` | +| `username` | string | false | username | +| `channel` | string | false | Default channel to use if others are not configured | +| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | +| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | +| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | +| `push_events` | boolean | false | Enable notifications for push events | +| `issues_events` | boolean | false | Enable notifications for issue events | +| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events | +| `tag_push_events` | boolean | false | Enable notifications for tag push events | +| `note_events` | boolean | false | Enable notifications for note events | +| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | +| `pipeline_events` | boolean | false | Enable notifications for pipeline events | +| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | +| `push_channel` | string | false | The name of the channel to receive push events notifications | +| `issue_channel` | string | false | The name of the channel to receive issues events notifications | +| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | +| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | +| `note_channel` | string | false | The name of the channel to receive note events notifications | +| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | +| `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | +| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | +| `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | + +### Delete Mattermost notifications integration + +Delete Mattermost notifications integration for a project. + +```plaintext +DELETE /projects/:id/integrations/mattermost +``` + +### Get Mattermost notifications integration settings + +Get Mattermost notifications integration settings for a project. + +```plaintext +GET /projects/:id/integrations/mattermost +``` + +## JetBrains TeamCity CI + +A continuous integration and build server + +### Create/Edit JetBrains TeamCity CI integration + +Set JetBrains TeamCity CI integration for a project. + +> The build configuration in TeamCity must use the build format number `%build.vcs.number%`. Configure monitoring of all branches so merge requests build. That setting is in the VSC root advanced settings. + +```plaintext +PUT /projects/:id/integrations/teamcity +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `teamcity_url` | string | true | TeamCity root URL. For example, `https://teamcity.example.com` | +| `build_type` | string | true | Build configuration ID | +| `username` | string | true | A user with permissions to trigger a manual build | +| `password` | string | true | The password of the user | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete JetBrains TeamCity CI integration + +Delete JetBrains TeamCity CI integration for a project. + +```plaintext +DELETE /projects/:id/integrations/teamcity +``` + +### Get JetBrains TeamCity CI integration settings + +Get JetBrains TeamCity CI integration settings for a project. + +```plaintext +GET /projects/:id/integrations/teamcity +``` + +## Jenkins CI + +A continuous integration and build server + +### Create/Edit Jenkins CI integration + +Set Jenkins CI integration for a project. + +```plaintext +PUT /projects/:id/integrations/jenkins +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `jenkins_url` | string | true | Jenkins URL like `http://jenkins.example.com`. | +| `project_name` | string | true | The URL-friendly project name. Example: `my_project_name`. | +| `username` | string | false | Username for authentication with the Jenkins server, if authentication is required by the server. | +| `password` | string | false | Password for authentication with the Jenkins server, if authentication is required by the server. | +| `push_events` | boolean | false | Enable notifications for push events. | +| `merge_requests_events` | boolean | false | Enable notifications for merge request events. | +| `tag_push_events` | boolean | false | Enable notifications for tag push events. | + +### Delete Jenkins CI integration + +Delete Jenkins CI integration for a project. + +```plaintext +DELETE /projects/:id/integrations/jenkins +``` + +### Get Jenkins CI integration settings + +Get Jenkins CI integration settings for a project. + +```plaintext +GET /projects/:id/integrations/jenkins +``` + +## Jenkins CI (Deprecated) integration + +A continuous integration and build server + +NOTE: +This integration was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1600) in GitLab 13.0. + +### Create/Edit Jenkins CI (Deprecated) integration + +Set Jenkins CI (Deprecated) integration for a project. + +```plaintext +PUT /projects/:id/integrations/jenkins-deprecated +``` + +Parameters: + +- `project_url` (**required**) - Jenkins project URL like `http://jenkins.example.com/job/my-project/` +- `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin +- `pass_unstable` (optional) - Unstable builds are treated as passing + +### Delete Jenkins CI (Deprecated) integration + +Delete Jenkins CI (Deprecated) integration for a project. + +```plaintext +DELETE /projects/:id/integrations/jenkins-deprecated +``` + +### Get Jenkins CI (Deprecated) integration settings + +Get Jenkins CI (Deprecated) integration settings for a project. + +```plaintext +GET /projects/:id/integrations/jenkins-deprecated +``` + +## MockCI + +Mock an external CI. See [`gitlab-org/gitlab-mock-ci-service`](https://gitlab.com/gitlab-org/gitlab-mock-ci-service) for an example of a companion mock integration. + +This integration is only available when your environment is set to development. + +### Create/Edit MockCI integration + +Set MockCI integration for a project. + +```plaintext +PUT /projects/:id/integrations/mock-ci +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `mock_service_url` | string | true | `http://localhost:4004` | + +### Delete MockCI integration + +Delete MockCI integration for a project. + +```plaintext +DELETE /projects/:id/integrations/mock-ci +``` + +### Get MockCI integration settings + +Get MockCI integration settings for a project. + +```plaintext +GET /projects/:id/integrations/mock-ci +``` + +## YouTrack + +YouTrack issue tracker + +### Create/Edit YouTrack integration + +Set YouTrack integration for a project. + +```plaintext +PUT /projects/:id/integrations/youtrack +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `issues_url` | string | true | Issue URL | +| `project_url` | string | true | Project URL | +| `description` | string | false | Description | +| `push_events` | boolean | false | Enable notifications for push events | + +### Delete YouTrack integration + +Delete YouTrack integration for a project. + +```plaintext +DELETE /projects/:id/integrations/youtrack +``` + +### Get YouTrack integration settings + +Get YouTrack integration settings for a project. + +```plaintext +GET /projects/:id/integrations/youtrack +``` diff --git a/doc/api/issues.md b/doc/api/issues.md index 97d0fd3ce8f..204d75e9ee4 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -1487,6 +1487,113 @@ NOTE: The `closed_by` attribute was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17042) in GitLab 10.6. This value is only present for issues closed after GitLab 10.6 and if the user account that closed the issue still exists. +## Clone an issue + +Clone the issue to given project. If the user has insufficient permissions, +an error message with status code `400` is returned. + +Copies as much data as possible as long as the target project contains equivalent labels, milestones, +and so on. + +```plaintext +POST /projects/:id/issues/:issue_iid/clone +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | ---------------------- | --------------------------------- | +| `id` | integer/string | **{check-circle}** Yes | ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | +| `issue_iid` | integer | **{check-circle}** Yes | Internal ID of a project's issue. | +| `to_project_id` | integer | **{check-circle}** Yes | ID of the new project. | +| `with_notes` | boolean | **{dotted-circle}** No | Clone the issue with [notes](notes.md). Default is `false`. | + +```shell +curl --request POST \ +--header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/1/clone?with_notes=true&to_project_id=6" +``` + +Example response: + +```json +{ + "id":290, + "iid":1, + "project_id":143, + "title":"foo", + "description":"closed", + "state":"opened", + "created_at":"2021-09-14T22:24:11.696Z", + "updated_at":"2021-09-14T22:24:11.696Z", + "closed_at":null, + "closed_by":null, + "labels":[ + + ], + "milestone":null, + "assignees":[ + { + "id":179, + "name":"John Doe2", + "username":"john", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/10fc7f102be8de7657fb4d80898bbfe3?s=80\u0026d=identicon", + "web_url":"https://gitlab.example.com/john" + } + ], + "author":{ + "id":179, + "name":"John Doe2", + "username":"john", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/10fc7f102be8de7657fb4d80898bbfe3?s=80\u0026d=identicon", + "web_url":"https://gitlab.example.com/john" + }, + "type":"ISSUE", + "assignee":{ + "id":179, + "name":"John Doe2", + "username":"john", + "state":"active", + "avatar_url":"https://www.gravatar.com/avatar/10fc7f102be8de7657fb4d80898bbfe3?s=80\u0026d=identicon", + "web_url":"https://gitlab.example.com/john" + }, + "user_notes_count":1, + "merge_requests_count":0, + "upvotes":0, + "downvotes":0, + "due_date":null, + "confidential":false, + "discussion_locked":null, + "issue_type":"issue", + "web_url":"https://gitlab.example.com/namespace1/project2/-/issues/1", + "time_stats":{ + "time_estimate":0, + "total_time_spent":0, + "human_time_estimate":null, + "human_total_time_spent":null + }, + "task_completion_status":{ + "count":0, + "completed_count":0 + }, + "blocking_issues_count":0, + "has_tasks":false, + "_links":{ + "self":"https://gitlab.example.com/api/v4/projects/143/issues/1", + "notes":"https://gitlab.example.com/api/v4/projects/143/issues/1/notes", + "award_emoji":"https://gitlab.example.com/api/v4/projects/143/issues/1/award_emoji", + "project":"https://gitlab.example.com/api/v4/projects/143" + }, + "references":{ + "short":"#1", + "relative":"#1", + "full":"namespace1/project2#1" + }, + "subscribed":true, + "moved_to_id":null, + "service_desk_reply_to":null +} +``` + ## Subscribe to an issue Subscribes the authenticated user to an issue to receive notifications. diff --git a/doc/api/members.md b/doc/api/members.md index 0b8cf686b8c..44e58f49d3b 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -336,7 +336,7 @@ The response represents only direct memberships. Inherited memberships are not i This API endpoint works on top-level groups only. It does not work on subgroups. -This API endpoint requires permission to admin memberships for the group. +This API endpoint requires permission to administer memberships for the group. This API endpoint takes [pagination](index.md#pagination) parameters `page` and `per_page` to restrict the list of memberships. @@ -561,7 +561,12 @@ Example response: ## Remove a member from a group or project -Removes a user from a group or project. +Removes a user from a group or project where the user has been explicitly assigned a role. + +The user needs to be a group member to qualify for removal. +For example, if the user was added directly to a project within the group but not this +group explicitly, you cannot use this API to remove them. See +[Remove a billable member from a group](#remove-a-billable-member-from-a-group) for an alternative approach. ```plaintext DELETE /groups/:id/members/:user_id diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index b4403e1d9b9..4ede95ea189 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api --- # Merge request approvals API **(PREMIUM)** @@ -15,8 +14,7 @@ in the project. Must be authenticated for all endpoints. ### Get Configuration -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/183) in GitLab 10.6. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. You can request information about a project's approval configuration using the following endpoint: @@ -44,8 +42,7 @@ GET /projects/:id/approvals ### Change configuration -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/183) in GitLab 10.6. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. If you are allowed to, you can change approval configuration using the following endpoint: @@ -180,7 +177,7 @@ GET /projects/:id/approval_rules ### Get a single project-level rule -> - Introduced 13.7. +> Introduced in GitLab 13.7. You can request information about a single project approval rules using the following endpoint: @@ -294,9 +291,10 @@ POST /projects/:id/approval_rules | `id` | integer or string | yes | The ID or [URL-encoded path of a project](index.md#namespaced-path-encoding) | | `name` | string | yes | The name of the approval rule | | `approvals_required` | integer | yes | The number of required approvals for this rule | +| `rule_type` | string | no | The type of rule. `any_approver` is a pre-configured default rule with `approvals_required` at `0`. Other rules are `regular`. | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | -| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by | +| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | ```json { @@ -379,6 +377,23 @@ POST /projects/:id/approval_rules } ``` +You can increase the default number of 0 required approvers like this: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header 'Content-Type: application/json' \ + --data '{"name": "Any name", "rule_type": "any_approver", "approvals_required": 2}' +``` + +Another example is creating an additional, user-specific rule: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header 'Content-Type: application/json' \ + --data '{"name": "Name of your rule", "approvals_required": 3, "user_ids": [123, 456, 789]}' \ + https://gitlab.example.com/api/v4/projects/<project_id>/approval_rules +``` + ### Update project-level rule > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11877) in GitLab 12.3. @@ -402,7 +417,7 @@ PUT /projects/:id/approval_rules/:approval_rule_id | `approvals_required` | integer | yes | The number of required approvals for this rule | | `user_ids` | Array | no | The ids of users as approvers | | `group_ids` | Array | no | The ids of groups as approvers | -| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by | +| `protected_branch_ids` | Array | no | **(PREMIUM)** The ids of protected branches to scope the rule by. To identify the ID, [use the API](protected_branches.md#list-protected-branches). | ```json { @@ -509,8 +524,7 @@ Configuration for approvals on a specific Merge Request. Must be authenticated f ### Get Configuration -> - Introduced in GitLab 8.9. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. You can request information about a merge request's approval status using the following endpoint: @@ -556,8 +570,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals ### Change approval configuration -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/183) in GitLab 10.6. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. If you are allowed to, you can change `approvals_required` using the following endpoint: @@ -937,8 +950,7 @@ These are system generated rules. ## Approve Merge Request -> - Introduced in GitLab 8.9. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. If you are allowed to, you can approve a merge request using the following endpoint: @@ -1001,8 +1013,7 @@ does not match, the response code is `409`. ## Unapprove Merge Request -> - Introduced in GitLab 9.0. -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. If you did approve a merge request, you can unapprove it using the following endpoint: diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index abf9d7af229..8a8a54a753a 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -5,15 +5,15 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# GitLab as an OAuth 2.0 provider **(FREE)** +# OAuth 2.0 identity provider API **(FREE)** -This document covers using the [OAuth2](https://oauth.net/2/) protocol to allow -other services to access GitLab resources on user's behalf. +GitLab provides an API to allow third-party services to access GitLab resources on a user's behalf +with the [OAuth2](https://oauth.net/2/) protocol. -If you want GitLab to be an OAuth authentication service provider to sign into -other services, see the [OAuth2 authentication service provider](../integration/oauth_provider.md) -documentation. This functionality is based on the -[doorkeeper Ruby gem](https://github.com/doorkeeper-gem/doorkeeper). +To configure GitLab for this, see +[Configure GitLab as an OAuth 2.0 authentication identity provider](../integration/oauth_provider.md). + +This functionality is based on the [doorkeeper Ruby gem](https://github.com/doorkeeper-gem/doorkeeper). ## Supported OAuth 2.0 flows @@ -25,7 +25,7 @@ GitLab supports the following authorization flows: - **Authorization code:** Secure and common flow. Recommended option for secure server-side apps. - **Implicit grant:** Originally designed for user-agent only apps, such as - single page web apps running on GitLab Pages). + single page web apps running on GitLab Pages. The [Internet Engineering Task Force (IETF)](https://tools.ietf.org/html/draft-ietf-oauth-security-topics-09#section-2.1.2) recommends against Implicit grant flow. - **Resource owner password credentials:** To be used **only** for securely @@ -412,6 +412,16 @@ prevent breaking changes introduced in [doorkeeper 5.0.2](https://github.com/doo Don't rely on these fields as they are slated for removal in a later release. +## Revoke a token + +To revoke a token, use the `revoke` endpoint. The API returns a 200 response code and an empty +JSON hash to indicate success. + +```ruby +parameters = 'client_id=APP_ID&client_secret=APP_SECRET&token=TOKEN' +RestClient.post 'https://gitlab.example.com/oauth/revoke', parameters +``` + ## OAuth 2.0 tokens and GitLab registries Standard OAuth 2.0 tokens support different degrees of access to GitLab diff --git a/doc/api/packages/debian.md b/doc/api/packages/debian.md index 154c99d7e0a..66377850c49 100644 --- a/doc/api/packages/debian.md +++ b/doc/api/packages/debian.md @@ -78,7 +78,7 @@ GET projects/:id/packages/debian/pool/:distribution/:letter/:package_name/:packa | `letter` | string | yes | The Debian Classification (first-letter or lib-first-letter). | | `package_name` | string | yes | The source package name. | | `package_version` | string | yes | The source package version. | -| `file_name` | string | yes | The file name. | +| `file_name` | string | yes | The filename. | ```shell curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/pool/my-distro/a/my-pkg/1.0.0/example_1.0.0~alpha2_amd64.deb" @@ -92,7 +92,7 @@ curl --header "Private-Token: <personal_access_token>" \ --remote-name ``` -This writes the downloaded file using the remote file name in the current directory. +This writes the downloaded file using the remote filename in the current directory. ## Route prefix @@ -150,7 +150,7 @@ curl --header "Private-Token: <personal_access_token>" \ --remote-name ``` -This writes the downloaded file using the remote file name in the current directory. +This writes the downloaded file using the remote filename in the current directory. ## Download a signed distribution Release file @@ -178,7 +178,7 @@ curl --header "Private-Token: <personal_access_token>" \ --remote-name ``` -This writes the downloaded file using the remote file name in the current directory. +This writes the downloaded file using the remote filename in the current directory. ## Download a release file signature @@ -206,7 +206,7 @@ curl --header "Private-Token: <personal_access_token>" \ --remote-name ``` -This writes the downloaded file using the remote file name in the current directory. +This writes the downloaded file using the remote filename in the current directory. ## Download a binary file's index @@ -236,4 +236,4 @@ curl --header "Private-Token: <personal_access_token>" \ --remote-name ``` -This writes the downloaded file using the remote file name in the current directory. +This writes the downloaded file using the remote filename in the current directory. diff --git a/doc/api/packages/helm.md b/doc/api/packages/helm.md index 8c3b9869368..82b3f5225b0 100644 --- a/doc/api/packages/helm.md +++ b/doc/api/packages/helm.md @@ -63,7 +63,7 @@ GET projects/:id/packages/helm/:channel/charts/:file_name.tgz | ----------- | ------ | -------- | ----------- | | `id` | string | yes | The ID or full path of the project. | | `channel` | string | yes | Helm repository channel. | -| `file_name` | string | yes | Chart file name. | +| `file_name` | string | yes | Chart filename. | ```shell curl --user <username>:<personal_access_token> \ diff --git a/doc/api/pages.md b/doc/api/pages.md index f81a3c3c72b..a115f0b0a0f 100644 --- a/doc/api/pages.md +++ b/doc/api/pages.md @@ -12,7 +12,7 @@ The GitLab Pages feature must be enabled to use these endpoints. Find out more a ## Unpublish pages -Remove pages. The user must have admin privileges. +Remove pages. The user must have the Administrator role. ```plaintext DELETE /projects/:id/pages diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index 47a8df3875e..624bdf29e5d 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -12,7 +12,7 @@ The GitLab Pages feature must be enabled to use these endpoints. Find out more a ## List all Pages domains -Get a list of all Pages domains. The user must have admin permissions. +Get a list of all Pages domains. The user must have the administrator role. ```plaintext GET /pages/domains diff --git a/doc/api/personal_access_tokens.md b/doc/api/personal_access_tokens.md index b96ee81f673..9c9551a5103 100644 --- a/doc/api/personal_access_tokens.md +++ b/doc/api/personal_access_tokens.md @@ -95,6 +95,6 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git - `204: No Content` if successfully revoked. - `400 Bad Request` if not revoked successfully. -## Create a personal access token (admin only) +## Create a personal access token (administrator only) See the [Users API documentation](users.md#create-a-personal-access-token) for information on creating a personal access token. diff --git a/doc/api/pipeline_schedules.md b/doc/api/pipeline_schedules.md index 74f96e5374e..625a92f9b89 100644 --- a/doc/api/pipeline_schedules.md +++ b/doc/api/pipeline_schedules.md @@ -115,7 +115,7 @@ POST /projects/:id/pipeline_schedules | `description` | string | yes | The description of the pipeline schedule. | | `ref` | string | yes | The branch or tag name that is triggered. | | `cron` | string | yes | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | -| `cron_timezone` | string | no | The timezone supported by `ActiveSupport::TimeZone`, for example: `Pacific Time (US & Canada)` (default: `UTC`). | +| `cron_timezone` | string | no | The time zone supported by `ActiveSupport::TimeZone`, for example: `Pacific Time (US & Canada)` (default: `UTC`). | | `active` | boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated (default: `true`). | ```shell @@ -162,7 +162,7 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id | `description` | string | no | The description of the pipeline schedule. | | `ref` | string | no | The branch or tag name that is triggered. | | `cron` | string | no | The [cron](https://en.wikipedia.org/wiki/Cron) schedule, for example: `0 1 * * *`. | -| `cron_timezone` | string | no | The timezone supported by `ActiveSupport::TimeZone` (for example `Pacific Time (US & Canada)`), or `TZInfo::Timezone` (for example `America/Los_Angeles`). | +| `cron_timezone` | string | no | The time zone supported by `ActiveSupport::TimeZone` (for example `Pacific Time (US & Canada)`), or `TZInfo::Timezone` (for example `America/Los_Angeles`). | | `active` | boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated. | ```shell diff --git a/doc/api/plan_limits.md b/doc/api/plan_limits.md index 52152dd6e14..8bd87f5a896 100644 --- a/doc/api/plan_limits.md +++ b/doc/api/plan_limits.md @@ -14,7 +14,7 @@ The existing plans depend on the GitLab edition. In the Community Edition, only is available. In the Enterprise Edition, additional plans are available as well. NOTE: -Administrator access is required to use this API. +The Administrator role is required to use this API. ## Get current plan limits diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 2dcef40aacb..a2c2da9065f 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -15,8 +15,8 @@ Get list of a project's variables. GET /projects/:id/variables ``` -| Attribute | Type | required | Description | -|-----------|---------|----------|---------------------| +| Attribute | Type | required | Description | +| --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | ```shell @@ -26,14 +26,20 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ```json [ { - "key": "TEST_VARIABLE_1", "variable_type": "env_var", - "value": "TEST_1" + "key": "TEST_VARIABLE_1", + "value": "TEST_1", + "protected": false, + "masked": true, + "environment_scope": "*" }, { - "key": "TEST_VARIABLE_2", "variable_type": "env_var", - "value": "TEST_2" + "key": "TEST_VARIABLE_2", + "value": "TEST_2", + "protected": false, + "masked": false, + "environment_scope": "*" } ] ``` @@ -46,11 +52,11 @@ Get the details of a project's specific variable. GET /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | -|-----------|---------|----------|-----------------------| -| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable | -| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | +| Attribute | Type | required | Description | +| --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/TEST_VARIABLE_1" @@ -62,7 +68,8 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "variable_type": "env_var", "value": "TEST_1", "protected": false, - "masked": true + "masked": true, + "environment_scope": "*" } ``` @@ -74,15 +81,15 @@ Create a new variable. POST /projects/:id/variables ``` -| Attribute | Type | required | Description | -|---------------------|---------|----------|-----------------------| -| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | -| `value` | string | yes | The `value` of a variable | -| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | no | Whether the variable is protected. Default: `false` | -| `masked` | boolean | no | Whether the variable is masked. Default: `false` | -| `environment_scope` | string | no | The `environment_scope` of the variable. Default: `*` | +| Attribute | Type | required | Description | +| ------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected. Default: `false` | +| `masked` | boolean | no | Whether the variable is masked. Default: `false` | +| `environment_scope` | string | no | The `environment_scope` of the variable. Default: `*` | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -91,10 +98,10 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ```json { + "variable_type": "env_var", "key": "NEW_VARIABLE", "value": "new value", "protected": false, - "variable_type": "env_var", "masked": false, "environment_scope": "*" } @@ -108,16 +115,16 @@ Update a project's variable. PUT /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | -|---------------------|---------|----------|-------------------------| -| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable | -| `value` | string | yes | The `value` of a variable | -| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | -| `protected` | boolean | no | Whether the variable is protected | -| `masked` | boolean | no | Whether the variable is masked | -| `environment_scope` | string | no | The `environment_scope` of the variable | -| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | +| Attribute | Type | required | Description | +| ------------------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected | +| `masked` | boolean | no | Whether the variable is masked | +| `environment_scope` | string | no | The `environment_scope` of the variable | +| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -126,9 +133,9 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ ```json { + "variable_type": "env_var", "key": "NEW_VARIABLE", "value": "updated value", - "variable_type": "env_var", "protected": true, "masked": false, "environment_scope": "*" @@ -143,11 +150,11 @@ Remove a project's variable. DELETE /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | -|-----------|---------|----------|-------------------------| -| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable | -| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | +| Attribute | Type | required | Description | +| --------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID of a project or [URL-encoded NAMESPACE/PROJECT_NAME of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/VARIABLE_1" diff --git a/doc/api/project_relations_export.md b/doc/api/project_relations_export.md new file mode 100644 index 00000000000..2016dcbd141 --- /dev/null +++ b/doc/api/project_relations_export.md @@ -0,0 +1,111 @@ +--- +stage: Manage +group: Import +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Project Relations Export API **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70330) in GitLab 14.4 behind the `bulk_import` [feature flag](../administration/feature_flags.md), disabled by default. + +FLAG: +On GitLab.com, this feature is available. +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to +[disable the `bulk_import` flag](../administration/feature_flags.md). +The feature is not ready for production use. It is still in experimental stage and might change in the future. + +With the Project Relations Export API, you can partially export project structure. This API is +similar to [project export](project_import_export.md), +but it exports each top-level relation (for example, milestones/boards/labels) as a separate file +instead of one archive. The project relations export API is primarily used in +[group migration](../user/group/import/index.md#enable-or-disable-gitlab-group-migration) +to support group project import. + +## Schedule new export + +Start a new project relations export: + +```plaintext +POST /projects/:id/export_relations +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | ID of the project owned by the authenticated user. | + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/export_relations" +``` + +```json +{ + "message": "202 Accepted" +} +``` + +## Export status + +View the status of the relations export: + +```plaintext +GET /projects/:id/export_relations/status +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | ID of the project owned by the authenticated user. | + +```shell +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/export_relations/status" +``` + +The status can be one of the following: + +- `0`: `started` +- `1`: `finished` +- `-1`: `failed` + +- `0` - `started` +- `1` - `finished` +- `-1` - `failed` + +```json +[ + { + "relation": "project_badges", + "status": 1, + "error": null, + "updated_at": "2021-05-04T11:25:20.423Z" + }, + { + "relation": "boards", + "status": 1, + "error": null, + "updated_at": "2021-05-04T11:25:20.085Z" + } +] +``` + +## Export download + +Download the finished relations export: + +```plaintext +GET /projects/:id/export_relations/download +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ---------------------------------------- | +| `id` | integer/string | yes | ID of the project owned by the authenticated user. | +| `relation` | string | yes | Name of the project top-level relation to download. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name \ + --remote-name "https://gitlab.example.com/api/v4/projects/1/export_relations/download?relation=labels" +``` + +```shell +ls labels.ndjson.gz +labels.ndjson.gz +``` diff --git a/doc/api/project_repository_storage_moves.md b/doc/api/project_repository_storage_moves.md index ebb15e1c1d6..b779a0046c6 100644 --- a/doc/api/project_repository_storage_moves.md +++ b/doc/api/project_repository_storage_moves.md @@ -10,7 +10,7 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31285) in GitLab 13.0. Project repositories including wiki and design repositories can be moved between storages. This can be useful when -[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrate-to-gitaly-cluster), +[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrating-to-gitaly-cluster), for example. As project repository storage moves are processed, they transition through different states. Values diff --git a/doc/api/project_statistics.md b/doc/api/project_statistics.md index a16bcc513f9..c69e41a423e 100644 --- a/doc/api/project_statistics.md +++ b/doc/api/project_statistics.md @@ -8,12 +8,12 @@ type: reference, api # Project statistics API **(FREE)** Every API call to [project](../user/project/index.md) statistics must be authenticated. +Retrieving these statistics requires write access to the repository. -## Get the statistics of the last 30 days +This API retrieves the number of times the project is either cloned or pulled +with the HTTP method. SSH fetches are not included. -Retrieving the statistics requires write access to the repository. -Currently only HTTP fetches statistics are returned. -Fetches statistics includes both clones and pulls count and are HTTP only, SSH fetches are not included. +## Get the statistics of the last 30 days ```plaintext GET /projects/:id/statistics diff --git a/doc/api/projects.md b/doc/api/projects.md index 29e3cdf6cbf..024362f3246 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -53,10 +53,10 @@ GET /projects | `last_activity_before` | datetime | **{dotted-circle}** No | Limit results to projects with last_activity before specified time. Format: ISO 8601 (`YYYY-MM-DDTHH:MM:SSZ`) | | `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | | `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [access level](members.md#valid-access-levels). | -| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for admins. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | +| `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, `last_activity_at`, or `similarity` fields. `repository_size`, `storage_size`, `packages_size` or `wiki_size` fields are only allowed for administrators. `similarity` ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332890) in GitLab 14.1) is only available when searching and is limited to projects that the current user is a member of. Default is `created_at`. | | `owned` | boolean | **{dotted-circle}** No | Limit by projects explicitly owned by the current user. | | `repository_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the repository checksum calculation has failed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6137) in GitLab 11.2). | -| `repository_storage` | string | **{dotted-circle}** No | Limit results to projects stored on `repository_storage`. _(admins only)_ | +| `repository_storage` | string | **{dotted-circle}** No | Limit results to projects stored on `repository_storage`. _(administrators only)_ | | `search_namespaces` | boolean | **{dotted-circle}** No | Include ancestor namespaces when matching search criteria. Default is `false`. | | `search` | string | **{dotted-circle}** No | Return list of projects matching the search criteria. | | `simple` | boolean | **{dotted-circle}** No | Return only limited fields for each project. This is a no-op without authentication as then _only_ simple fields are returned. | @@ -66,7 +66,7 @@ GET /projects | `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | | `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6137) in GitLab 11.2). | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | +| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | | `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | | `with_programming_language` | string | **{dotted-circle}** No | Limit by projects which use the given programming language. | @@ -386,7 +386,7 @@ GET /users/:user_id/projects | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | | `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | +| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | | `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | | `with_programming_language` | string | **{dotted-circle}** No | Limit by projects which use the given programming language. | @@ -621,7 +621,7 @@ GET /users/:user_id/starred_projects | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | | `user_id` | string | **{check-circle}** Yes | The ID or username of the user. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | +| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | | `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | @@ -840,7 +840,7 @@ GET /projects/:id | `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `license` | boolean | **{dotted-circle}** No | Include project license data. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | +| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | ```json { @@ -1218,7 +1218,7 @@ POST /projects | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | | `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(admins only)_ | +| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrator only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | @@ -1237,7 +1237,7 @@ POST /projects ## Create project for user -Creates a new project owned by the specified user. Available only for admins. +Creates a new project owned by the specified user. Available only for administrators. If your HTTP repository isn't publicly accessible, add authentication information to the URL `https://username:password@gitlab.company.com/group/project.git`, @@ -1295,7 +1295,7 @@ POST /projects/user/:user_id | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project-members. | | `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(admins only)_ | +| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `shared_runners_enabled` | boolean | **{dotted-circle}** No | Enable shared runners for this project. | @@ -1360,7 +1360,7 @@ PUT /projects/:id | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | | `mirror_overwrites_diverged_branches` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirror overwrites diverged branches. | | `mirror_trigger_builds` **(PREMIUM)** | boolean | **{dotted-circle}** No | Pull mirroring triggers builds. | -| `mirror_user_id` **(PREMIUM)** | integer | **{dotted-circle}** No | User responsible for all the activity surrounding a pull mirror event. _(admins only)_ | +| `mirror_user_id` **(PREMIUM)** | integer | **{dotted-circle}** No | User responsible for all the activity surrounding a pull mirror event. _(administrators only)_ | | `mirror` **(PREMIUM)** | boolean | **{dotted-circle}** No | Enables pull mirroring in a project. | | `name` | string | **{dotted-circle}** No | The name of the project. | | `operations_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1375,7 +1375,7 @@ PUT /projects/:id | `public_builds` | boolean | **{dotted-circle}** No | If `true`, jobs can be viewed by non-project members. | | `remove_source_branch_after_merge` | boolean | **{dotted-circle}** No | Enable `Delete source branch` option by default for all new merge requests. | | `repository_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(admins only)_ | +| `repository_storage` | string | **{dotted-circle}** No | Which storage shard the repository is on. _(administrators only)_ | | `request_access_enabled` | boolean | **{dotted-circle}** No | Allow users to request member access. | | `resolve_outdated_diff_discussions` | boolean | **{dotted-circle}** No | Automatically resolve merge request diffs discussions on lines changed with a push. | | `service_desk_enabled` | boolean | **{dotted-circle}** No | Enable or disable Service Desk feature. | @@ -1442,7 +1442,7 @@ GET /projects/:id/forks | `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | -| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | +| `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrators only)_ | | `with_issues_enabled` | boolean | **{dotted-circle}** No | Limit by enabled issues feature. | | `with_merge_requests_enabled` | boolean | **{dotted-circle}** No | Limit by enabled merge requests feature. | @@ -2047,7 +2047,7 @@ This endpoint: merge requests). - From [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) on [Premium or higher](https://about.gitlab.com/pricing/) tiers, group - admins can [configure](../user/group/index.md#enable-delayed-project-removal) + administrators can [configure](../user/group/index.md#enable-delayed-project-removal) projects within a group to be deleted after a delayed period. When enabled, actual deletion happens after the number of days specified in the [default deletion delay](../user/admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). @@ -2373,7 +2373,7 @@ is returned. ## Fork relationship Allows modification of the forked relationship between existing projects. -Available only for project owners and admins. +Available only for project owners and administrators. ### Create a forked from/to relation between existing projects @@ -2678,13 +2678,21 @@ Read more in the [Project members](members.md) documentation. > - Introduced in GitLab 11. > - Moved to GitLab Premium in 13.9. -Configure pull mirroring while [creating a new project](#create-project) or [updating an existing project](#edit-project) using the API if the remote repository is publicly accessible or via `username/password` authentication. In case your HTTP repository is not publicly accessible, you can add the authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git`, where password is a [personal access token](../user/profile/personal_access_tokens.md) with the API scope enabled. - -The relevant API parameters to update are: - -- `import_url`: URL of remote repository being mirrored (with `username:password` if needed). -- `mirror`: Enables pull mirroring on project when set to `true`. -- `only_mirror_protected_branches`: Set to `true` for protected branches. +Configure pull mirroring while [creating a new project](#create-project) +or [updating an existing project](#edit-project) using the API +if the remote repository is publicly accessible +or via `username:token` authentication. +In case your HTTP repository is not publicly accessible, +you can add the authentication information to the URL: +`https://username:token@gitlab.company.com/group/project.git`, +where `token` is a [personal access token](../user/profile/personal_access_tokens.md) +with the API scope enabled. + +| Attribute | Type | Required | Description | +|--------------|---------|------------------------|-------------| +| `import_url` | string | **{check-circle}** Yes | URL of remote repository being mirrored (with `user:token` if needed). | +| `mirror` | boolean | **{check-circle}** Yes | Enables pull mirroring on project when set to `true`. | +| `only_mirror_protected_branches`| boolean | **{dotted-circle}** No | Limits mirroring to only protected branches when set to `true`. | ## Start the pull mirroring process for a Project **(PREMIUM)** diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index 72627783947..ded47b24c12 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -729,3 +729,5 @@ Example response: A release with a `released_at` attribute set to a future date is labeled as an **Upcoming Release** [in the UI](../../user/project/releases/index.md#upcoming-releases). + +Additionally, if a [release is requested from the API](#list-releases), for each release with a `release_at` attribute set to a future date, an additional attribute `upcoming_release` (set to true) will be returned as part of the response. diff --git a/doc/api/remote_mirrors.md b/doc/api/remote_mirrors.md index 799763a7da3..8b584285033 100644 --- a/doc/api/remote_mirrors.md +++ b/doc/api/remote_mirrors.md @@ -7,7 +7,7 @@ type: reference, api # Project remote mirrors API **(FREE)** -[Push mirrors](../user/project/repository/repository_mirroring.md#push-to-a-remote-repository) +[Push mirrors](../user/project/repository/mirror/push.md) defined on a project's repository settings are called "remote mirrors", and the state of these mirrors can be queried and modified via the remote mirror API outlined below. @@ -51,11 +51,15 @@ NOTE: For security reasons, the `url` attribute is always scrubbed of username and password information. -## Create a remote mirror +## Create a pull mirror + +Learn how to [configure a pull mirror](projects.md#configure-pull-mirroring-for-a-project) using the Projects API. + +## Create a push mirror > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24189) in GitLab 12.9. -Create a remote mirror for a project. The mirror is disabled by default. You can enable it by including the optional parameter `enabled` when creating it: +Push mirroring is disabled by default. You can enable it by including the optional parameter `enabled` when creating it: ```plaintext POST /projects/:id/remote_mirrors @@ -63,7 +67,7 @@ POST /projects/:id/remote_mirrors | Attribute | Type | Required | Description | | :---------- | :----- | :--------- | :------------ | -| `url` | String | yes | The URL of the remote repository to be mirrored. | +| `url` | String | yes | The target URL to which the repository is mirrored. | | `enabled` | Boolean | no | Determines if the mirror is enabled. | | `only_protected_branches` | Boolean | no | Determines if only protected branches are mirrored. | | `keep_divergent_refs` | Boolean | no | Determines if divergent refs are skipped. | diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 1c9136d22ac..e93ffbc5e72 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -14,6 +14,12 @@ be accessed without authentication if the repository is publicly accessible. This command provides essentially the same functionality as the `git ls-tree` command. For more information, see the section _Tree Objects_ in the [Git internals documentation](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects/#_tree_objects). +WARNING: +This endpoint is changing to keyset-based pagination. Iterating pages of results +with a number (`?page=2`) [is deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67509). +Support for iterating with a number will become unsupported in GitLab 15.0. Use +the new [keyset pagination system](index.md#keyset-based-pagination) instead. + ```plaintext GET /projects/:id/repository/tree ``` @@ -27,6 +33,8 @@ Supported attributes: | `ref` | string | no | The name of a repository branch or tag or if not given the default branch. | | `recursive` | boolean | no | Boolean value used to get a recursive tree (false by default). | | `per_page` | integer | no | Number of results to show per page. If not specified, defaults to `20`. [Learn more on pagination](index.md#pagination). | +| `pagination` | string | no | If set to `keyset`, use the new keyset pagination method. | +| `page_token` | string | no | The tree record ID at which to fetch the next page. Used only with keyset pagination. | ```json [ @@ -118,6 +126,7 @@ Supported attributes: ## Get file archive > Support for [including Git LFS blobs](../topics/git/lfs/index.md#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. +> Support for downloading a subfolder was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28827) in GitLab 14.4. Get an archive of the repository. This endpoint can be accessed without authentication if the repository is publicly accessible. @@ -139,11 +148,12 @@ Supported attributes: |:------------|:---------------|:---------|:----------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `sha` | string | no | The commit SHA to download. A tag, branch reference, or SHA can be used. This defaults to the tip of the default branch if not specified. | +| `path` | string | no | The subpath of the repository to download. This defaults to the whole repository (empty string). | Example request: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/<project_id>/repository/archive?sha=<commit_sha>" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/<project_id>/repository/archive?sha=<commit_sha>&path=<path>" ``` ## Compare branches, tags or commits @@ -640,7 +650,7 @@ In an entry, the following variables are available (here `foo.bar` means that - `author.reference`: a reference to the commit author (for example, `@alice`). - `author.contributor`: a boolean set to `true` when the author is not a project member, otherwise `false`. -- `author.credit`: a boolean set to `true` when `author.contributor` is `true` or +- `author.credit`: a boolean set to `true` when `author.contributor` is `true` or when `include_groups` is configured, and the author is a member of one of the groups. - `merge_request.reference`: a reference to the merge request that first diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 1e33aadbc1b..4fb1a94e294 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -7,9 +7,11 @@ type: reference, api # Repository files API **(FREE)** -**CRUD for repository files** +You can fetch, create, update, and delete files in your repository with this API. +You can also [configure rate limits](../user/admin_area/settings/files_api_rate_limits.md) +for this API. -**Create, read, update, and delete repository files using this API** +## Available scopes for personal access tokens The different scopes available using [personal access tokens](../user/profile/personal_access_tokens.md) are depicted in the following table. @@ -19,8 +21,6 @@ in the following table. | `read_repository` | Allows read-access to the repository files. | | `api` | Allows read-write access to the repository files. | -> `read_repository` scope was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23534) in GitLab 11.6. - ## Get file from repository Allows you to receive information about file in repository like name, size, diff --git a/doc/api/resource_groups.md b/doc/api/resource_groups.md new file mode 100644 index 00000000000..ce4fa33d7f2 --- /dev/null +++ b/doc/api/resource_groups.md @@ -0,0 +1,70 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: concepts, howto +--- + +# Resource Groups API + +You can read more about [controling the job concurrency with resource groups](../ci/resource_groups/index.md). + +## Get a specific resource group + +```plaintext +GET /projects/:id/resource_groups/:key +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The key of the resource group | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/resource_groups/production" +``` + +Example of response + +```json +{ + "id": 3, + "key": "production", + "process_mode": "unordered", + "created_at": "2021-09-01T08:04:59.650Z", + "updated_at": "2021-09-01T08:04:59.650Z" +} +``` + +## Edit an existing resource group + +Updates an existing resource group's properties. + +It returns `200` if the resource group was successfully updated. In case of an error, a status code `400` is returned. + +```plaintext +PUT /projects/:id/resource_groups/:key +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The key of the resource group | +| `process_mode` | string | no | The process mode of the resource group. One of `unordered`, `oldest_first` or `newest_first`. Read [process modes](../ci/resource_groups/index.md#process-modes) for more information. | + +```shell +curl --request PUT --data "process_mode=oldest_first" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/resource_groups/production" +``` + +Example response: + +```json +{ + "id": 3, + "key": "production", + "process_mode": "oldest_first", + "created_at": "2021-09-01T08:04:59.650Z", + "updated_at": "2021-09-01T08:13:38.679Z" +} +``` diff --git a/doc/api/runners.md b/doc/api/runners.md index bfdf2d49100..5e84080ecb5 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -15,7 +15,7 @@ There are two tokens to take into account when connecting a runner with GitLab. | Token | Description | | ----- | ----------- | | Registration token | Token used to [register the runner](https://docs.gitlab.com/runner/register/). It can be [obtained through GitLab](../ci/runners/index.md). | -| Authentication token | Token used to authenticate the runner with the GitLab instance. It is obtained either automatically when [registering a runner](https://docs.gitlab.com/runner/register/), or manually when [registering the runner via the Runner API](#register-a-new-runner). | +| Authentication token | Token used to authenticate the runner with the GitLab instance. It is obtained automatically when you [register a runner](https://docs.gitlab.com/runner/register/) or by the Runner API when you manually [register a runner](#register-a-new-runner) or [reset the authentication token](#reset-runners-authentication-token). | Here's an example of how the two tokens are used in runner registration: @@ -86,7 +86,7 @@ Example response: ## List all runners **(FREE SELF)** Get a list of all runners in the GitLab instance (specific and shared). Access -is restricted to users with `admin` privileges. +is restricted to users with the administrator role. ```plaintext GET /runners/all @@ -720,3 +720,28 @@ POST /groups/:id/runners/reset_registration_token curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/groups/9/runners/reset_registration_token" ``` + +## Reset runner's authentication token + +Resets the runner's authentication token. + +```plaintext +POST /runners/:id/reset_authentication_token +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a runner | + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/runners/1/reset_authentication_token" +``` + +Example response: + +```json +{ + "token": "6337ff461c94fd3fa32ba3b1ff4125" +} +``` diff --git a/doc/api/search.md b/doc/api/search.md index d3a2f9c41b6..d3f0cba9234 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -1161,7 +1161,7 @@ Blobs searches are performed on both filenames and contents. Search results: times in the content. ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation&ref=feature" +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=keyword%20filename:*.py ``` Example response: @@ -1175,7 +1175,7 @@ Example response: "path": "README.md", "filename": "README.md", "id": null, - "ref": "feature", + "ref": "master", "startline": 46, "project_id": 6 } diff --git a/doc/api/services.md b/doc/api/services.md index cf6c5ec511b..7587e53c9db 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -1,1471 +1,9 @@ --- -stage: Ecosystem -group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'integrations.md' +remove_date: '2021-11-09' --- -# Services API **(FREE)** +This file was moved to [another location](integrations.md). -NOTE: -This API requires an access token with the [Maintainer or Owner role](../user/permissions.md). - -## List all active services - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21330) in GitLab 12.7. - -Get a list of all active project services. - -```plaintext -GET /projects/:id/services -``` - -Example response: - -```json -[ - { - "id": 75, - "title": "Jenkins CI", - "slug": "jenkins", - "created_at": "2019-11-20T11:20:25.297Z", - "updated_at": "2019-11-20T12:24:37.498Z", - "active": true, - "commit_events": true, - "push_events": true, - "issues_events": true, - "confidential_issues_events": true, - "merge_requests_events": true, - "tag_push_events": false, - "note_events": true, - "confidential_note_events": true, - "pipeline_events": true, - "wiki_page_events": true, - "job_events": true, - "comment_on_event_enabled": true - }, - { - "id": 76, - "title": "Alerts endpoint", - "slug": "alerts", - "created_at": "2019-11-20T11:20:25.297Z", - "updated_at": "2019-11-20T12:24:37.498Z", - "active": true, - "commit_events": true, - "push_events": true, - "issues_events": true, - "confidential_issues_events": true, - "merge_requests_events": true, - "tag_push_events": true, - "note_events": true, - "confidential_note_events": true, - "pipeline_events": true, - "wiki_page_events": true, - "job_events": true, - "comment_on_event_enabled": true - } -] -``` - -## Asana - -Add commit messages as comments to Asana tasks. - -See also the [Asana service documentation](../user/project/integrations/asana.md). - -### Create/Edit Asana service - -Set Asana service for a project. - -```plaintext -PUT /projects/:id/services/asana -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `api_key` | string | true | User API token. User must have access to task. All comments are attributed to this user. | -| `restrict_to_branch` | string | false | Comma-separated list of branches to be are automatically inspected. Leave blank to include all branches. | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Asana service - -Delete Asana service for a project. - -```plaintext -DELETE /projects/:id/services/asana -``` - -### Get Asana service settings - -Get Asana service settings for a project. - -```plaintext -GET /projects/:id/services/asana -``` - -## Assembla - -Project Management Software (Source Commits Endpoint) - -### Create/Edit Assembla service - -Set Assembla service for a project. - -```plaintext -PUT /projects/:id/services/assembla -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | The authentication token -| `subdomain` | string | false | The subdomain setting | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Assembla service - -Delete Assembla service for a project. - -```plaintext -DELETE /projects/:id/services/assembla -``` - -### Get Assembla service settings - -Get Assembla service settings for a project. - -```plaintext -GET /projects/:id/services/assembla -``` - -## Atlassian Bamboo CI - -A continuous integration and build server - -### Create/Edit Atlassian Bamboo CI service - -Set Atlassian Bamboo CI service for a project. - -> You must set up automatic revision labeling and a repository trigger in Bamboo. - -```plaintext -PUT /projects/:id/services/bamboo -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `bamboo_url` | string | true | Bamboo root URL. For example, `https://bamboo.example.com`. | -| `build_key` | string | true | Bamboo build plan key like KEY | -| `username` | string | true | A user with API access, if applicable | -| `password` | string | true | Password of the user | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Atlassian Bamboo CI service - -Delete Atlassian Bamboo CI service for a project. - -```plaintext -DELETE /projects/:id/services/bamboo -``` - -### Get Atlassian Bamboo CI service settings - -Get Atlassian Bamboo CI service settings for a project. - -```plaintext -GET /projects/:id/services/bamboo -``` - -## Bugzilla - -Bugzilla Issue Tracker - -### Create/Edit Bugzilla service - -Set Bugzilla service for a project. - -```plaintext -PUT /projects/:id/services/bugzilla -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New Issue URL | -| `issues_url` | string | true | Issue URL | -| `project_url` | string | true | Project URL | -| `description` | string | false | Description | -| `title` | string | false | Title | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Bugzilla Service - -Delete Bugzilla service for a project. - -```plaintext -DELETE /projects/:id/services/bugzilla -``` - -### Get Bugzilla Service Settings - -Get Bugzilla service settings for a project. - -```plaintext -GET /projects/:id/services/bugzilla -``` - -## Buildkite - -Continuous integration and deployments - -### Create/Edit Buildkite service - -Set Buildkite service for a project. - -```plaintext -PUT /projects/:id/services/buildkite -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | Buildkite project GitLab token | -| `project_url` | string | true | Pipeline URL. For example, `https://buildkite.com/example/pipeline` | -| `enable_ssl_verification` | boolean | false | DEPRECATED: This parameter has no effect since SSL verification is always enabled | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Buildkite service - -Delete Buildkite service for a project. - -```plaintext -DELETE /projects/:id/services/buildkite -``` - -### Get Buildkite service settings - -Get Buildkite service settings for a project. - -```plaintext -GET /projects/:id/services/buildkite -``` - -## Campfire - -Send notifications about push events to Campfire chat rooms. -[New users can no longer sign up for Campfire](https://basecamp.com/retired/campfire). - -### Create/Edit Campfire service - -Set Campfire service for a project. - -```plaintext -PUT /projects/:id/services/campfire -``` - -Parameters: - -| Parameter | Type | Required | Description | -|---------------|---------|----------|---------------------------------------------------------------------------------------------| -| `token` | string | true | Campfire API token. To find it, log into Campfire and select **My info**. | -| `subdomain` | string | false | Campfire subdomain. Text between `https://` and `.campfirenow.com` when you're logged in. | -| `room` | string | false | Campfire room. The last part of the URL when you're in a room. | -| `push_events` | boolean | false | Enable notifications for push events. | - -### Delete Campfire service - -Delete Campfire service for a project. - -```plaintext -DELETE /projects/:id/services/campfire -``` - -### Get Campfire service settings - -Get Campfire service settings for a project. - -```plaintext -GET /projects/:id/services/campfire -``` - -## Unify Circuit - -Unify Circuit RTC and collaboration tool. - -### Create/Edit Unify Circuit service - -Set Unify Circuit service for a project. - -```plaintext -PUT /projects/:id/services/unify-circuit -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Unify Circuit webhook. For example, `https://circuit.com/rest/v2/webhooks/incoming/...`. | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | - -### Delete Unify Circuit service - -Delete Unify Circuit service for a project. - -```plaintext -DELETE /projects/:id/services/unify-circuit -``` - -### Get Unify Circuit service settings - -Get Unify Circuit service settings for a project. - -```plaintext -GET /projects/:id/services/unify-circuit -``` - -## Webex Teams - -Webex Teams collaboration tool. - -### Create/Edit Webex Teams service - -Set Webex Teams service for a project. - -```plaintext -PUT /projects/:id/services/webex-teams -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Webex Teams webhook. For example, `https://api.ciscospark.com/v1/webhooks/incoming/...`. | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | - -### Delete Webex Teams service - -Delete Webex Teams service for a project. - -```plaintext -DELETE /projects/:id/services/webex-teams -``` - -### Get Webex Teams service settings - -Get Webex Teams service settings for a project. - -```plaintext -GET /projects/:id/services/webex-teams -``` - -## Custom Issue Tracker - -Custom issue tracker - -### Create/Edit Custom Issue Tracker service - -Set Custom Issue Tracker service for a project. - -```plaintext -PUT /projects/:id/services/custom-issue-tracker -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New Issue URL | -| `issues_url` | string | true | Issue URL | -| `project_url` | string | true | Project URL | -| `description` | string | false | Description | -| `title` | string | false | Title | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Custom Issue Tracker service - -Delete Custom Issue Tracker service for a project. - -```plaintext -DELETE /projects/:id/services/custom-issue-tracker -``` - -### Get Custom Issue Tracker service settings - -Get Custom Issue Tracker service settings for a project. - -```plaintext -GET /projects/:id/services/custom-issue-tracker -``` - -## Drone CI - -Drone is a Continuous Integration platform built on Docker, written in Go - -### Create/Edit Drone CI service - -Set Drone CI service for a project. - -```plaintext -PUT /projects/:id/services/drone-ci -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | Drone CI project specific token | -| `drone_url` | string | true | `http://drone.example.com` | -| `enable_ssl_verification` | boolean | false | Enable SSL verification | -| `push_events` | boolean | false | Enable notifications for push events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | - -### Delete Drone CI service - -Delete Drone CI service for a project. - -```plaintext -DELETE /projects/:id/services/drone-ci -``` - -### Get Drone CI service settings - -Get Drone CI service settings for a project. - -```plaintext -GET /projects/:id/services/drone-ci -``` - -## Emails on push - -Email the commits and diff of each push to a list of recipients. - -### Create/Edit Emails on push service - -Set Emails on push service for a project. - -```plaintext -PUT /projects/:id/services/emails-on-push -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `recipients` | string | true | Emails separated by whitespace | -| `disable_diffs` | boolean | false | Disable code diffs | -| `send_from_committer_email` | boolean | false | Send from committer | -| `push_events` | boolean | false | Enable notifications for push events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". Notifications are always fired for tag pushes. The default value is "all" | - -### Delete Emails on push service - -Delete Emails on push service for a project. - -```plaintext -DELETE /projects/:id/services/emails-on-push -``` - -### Get Emails on push service settings - -Get Emails on push service settings for a project. - -```plaintext -GET /projects/:id/services/emails-on-push -``` - -## Confluence service - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220934) in GitLab 13.2. - -Replaces the link to the internal wiki with a link to a Confluence Cloud Workspace. - -### Create/Edit Confluence service - -Set Confluence service for a project. - -```plaintext -PUT /projects/:id/services/confluence -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `confluence_url` | string | true | The URL of the Confluence Cloud Workspace hosted on atlassian.net. | - -### Delete Confluence service - -Delete Confluence service for a project. - -```plaintext -DELETE /projects/:id/services/confluence -``` - -### Get Confluence service settings - -Get Confluence service settings for a project. - -```plaintext -GET /projects/:id/services/confluence -``` - -## External wiki - -Replaces the link to the internal wiki with a link to an external wiki. - -### Create/Edit External wiki service - -Set External wiki service for a project. - -```plaintext -PUT /projects/:id/services/external-wiki -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `external_wiki_url` | string | true | The URL of the external wiki | - -### Delete External wiki service - -Delete External wiki service for a project. - -```plaintext -DELETE /projects/:id/services/external-wiki -``` - -### Get External wiki service settings - -Get External wiki service settings for a project. - -```plaintext -GET /projects/:id/services/external-wiki -``` - -## Flowdock - -Flowdock is a ChatOps application for collaboration in software engineering -companies. You can send notifications from GitLab events to Flowdock flows. -For integration instructions, see the [Flowdock documentation](https://www.flowdock.com/help/gitlab). - -### Create/Edit Flowdock service - -Set Flowdock service for a project. - -```plaintext -PUT /projects/:id/services/flowdock -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | Flowdock Git source token | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Flowdock service - -Delete Flowdock service for a project. - -```plaintext -DELETE /projects/:id/services/flowdock -``` - -### Get Flowdock service settings - -Get Flowdock service settings for a project. - -```plaintext -GET /projects/:id/services/flowdock -``` - -## GitHub **(PREMIUM)** - -Code collaboration software. - -### Create/Edit GitHub service - -Set GitHub service for a project. - -```plaintext -PUT /projects/:id/services/github -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | GitHub API token with `repo:status` OAuth scope | -| `repository_url` | string | true | GitHub repository URL | -| `static_context` | boolean | false | Append instance name instead of branch to [status check name](../user/project/integrations/github.md#static--dynamic-status-check-names) | - -### Delete GitHub service - -Delete GitHub service for a project. - -```plaintext -DELETE /projects/:id/services/github -``` - -### Get GitHub service settings - -Get GitHub service settings for a project. - -```plaintext -GET /projects/:id/services/github -``` - -## Hangouts Chat - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20290) in GitLab 11.2. - -Google Workspace team collaboration tool. - -### Create/Edit Hangouts Chat service - -Set Hangouts Chat service for a project. - -```plaintext -PUT /projects/:id/services/hangouts-chat -``` - -NOTE: -Specific event parameters (for example, `push_events` flag) were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/11435) in GitLab 10.4. - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Hangouts Chat webhook. For example, `https://chat.googleapis.com/v1/spaces...`. | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | - -### Delete Hangouts Chat service - -Delete Hangouts Chat service for a project. - -```plaintext -DELETE /projects/:id/services/hangouts-chat -``` - -### Get Hangouts Chat service settings - -Get Hangouts Chat service settings for a project. - -```plaintext -GET /projects/:id/services/hangouts-chat -``` - -## irker (IRC gateway) - -Send IRC messages, on update, to a list of recipients through an irker gateway. - -For more information, see the [irker integration documentation](../user/project/integrations/irker.md). - -### Create/Edit irker (IRC gateway) service - -Set irker (IRC gateway) service for a project. - -```plaintext -PUT /projects/:id/services/irker -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `recipients` | string | true | Recipients/channels separated by whitespaces | -| `default_irc_uri` | string | false | `irc://irc.network.net:6697/` | -| `server_host` | string | false | localhost | -| `server_port` | integer | false | 6659 | -| `colorize_messages` | boolean | false | Colorize messages | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete irker (IRC gateway) service - -Delete irker (IRC gateway) service for a project. - -```plaintext -DELETE /projects/:id/services/irker -``` - -### Get irker (IRC gateway) service settings - -Get irker (IRC gateway) service settings for a project. - -```plaintext -GET /projects/:id/services/irker -``` - -## Jira - -Jira issue tracker. - -### Get Jira service settings - -Get Jira service settings for a project. - -```plaintext -GET /projects/:id/services/jira -``` - -### Create/Edit Jira service - -Set Jira service for a project. - -> Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and -> `project_url` are replaced by `url`. If you are using an -> older version, [follow this documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/8-13-stable-ee/doc/api/services.md#jira). - -```plaintext -PUT /projects/:id/services/jira -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `url` | string | yes | The URL to the Jira project which is being linked to this GitLab project. For example, `https://jira.example.com`. | -| `api_url` | string | no | The base URL to the Jira instance API. Web URL value is used if not set. For example, `https://jira-api.example.com`. | -| `username` | string | yes | The username of the user created to be used with GitLab/Jira. | -| `password` | string | yes | The password of the user created to be used with GitLab/Jira. | -| `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_automatic` | boolean | no | Enable [automatic issue transitions](../integration/jira/issues.md#automatic-issue-transitions). Takes precedence over `jira_issue_transition_id` if enabled. Defaults to `false` | -| `jira_issue_transition_id` | string | no | The ID of one or more transitions for [custom issue transitions](../integration/jira/issues.md#custom-issue-transitions). Ignored if `jira_issue_transition_automatic` is enabled. Defaults to a blank string, which disables custom transitions. | -| `commit_events` | boolean | false | Enable notifications for commit events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `comment_on_event_enabled` | boolean | false | Enable comments inside Jira issues on each GitLab event (commit / merge request) | - -### Delete Jira service - -Remove all previously Jira settings from a project. - -```plaintext -DELETE /projects/:id/services/jira -``` - -## Slack slash commands - -Ability to receive slash commands from a Slack chat instance. - -### Get Slack slash command service settings - -Get Slack slash command service settings for a project. - -```plaintext -GET /projects/:id/services/slack-slash-commands -``` - -Example response: - -```json -{ - "id": 4, - "title": "Slack slash commands", - "slug": "slack-slash-commands", - "created_at": "2017-06-27T05:51:39-07:00", - "updated_at": "2017-06-27T05:51:39-07:00", - "active": true, - "push_events": true, - "issues_events": true, - "confidential_issues_events": true, - "merge_requests_events": true, - "tag_push_events": true, - "note_events": true, - "job_events": true, - "pipeline_events": true, - "comment_on_event_enabled": false, - "properties": { - "token": "<your_access_token>" - } -} -``` - -### Create/Edit Slack slash command service - -Set Slack slash command for a project. - -```plaintext -PUT /projects/:id/services/slack-slash-commands -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | yes | The Slack token | - -### Delete Slack slash command service - -Delete Slack slash command service for a project. - -```plaintext -DELETE /projects/:id/services/slack-slash-commands -``` - -## Mattermost slash commands - -Ability to receive slash commands from a Mattermost chat instance. - -### Get Mattermost slash command service settings - -Get Mattermost slash command service settings for a project. - -```plaintext -GET /projects/:id/services/mattermost-slash-commands -``` - -### Create/Edit Mattermost slash command service - -Set Mattermost slash command for a project. - -```plaintext -PUT /projects/:id/services/mattermost-slash-commands -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | yes | The Mattermost token | -| `username` | string | no | The username to use to post the message | - -### Delete Mattermost slash command service - -Delete Mattermost slash command service for a project. - -```plaintext -DELETE /projects/:id/services/mattermost-slash-commands -``` - -## Packagist - -Update your project on Packagist (the main Composer repository) when commits or tags are pushed to GitLab. - -### Create/Edit Packagist service - -Set Packagist service for a project. - -```plaintext -PUT /projects/:id/services/packagist -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `username` | string | yes | The username of a Packagist account | -| `token` | string | yes | API token to the Packagist server | -| `server` | boolean | no | URL of the Packagist server. Leave blank for default: <https://packagist.org> | -| `push_events` | boolean | false | Enable notifications for push events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | - -### Delete Packagist service - -Delete Packagist service for a project. - -```plaintext -DELETE /projects/:id/services/packagist -``` - -### Get Packagist service settings - -Get Packagist service settings for a project. - -```plaintext -GET /projects/:id/services/packagist -``` - -## Pipeline-Emails - -Get emails for GitLab CI/CD pipelines. - -### Create/Edit Pipeline-Emails service - -Set Pipeline-Emails service for a project. - -```plaintext -PUT /projects/:id/services/pipelines-email -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `recipients` | string | yes | Comma-separated list of recipient email addresses | -| `add_pusher` | boolean | no | Add pusher to recipients list | -| `notify_only_broken_pipelines` | boolean | no | Notify only broken pipelines | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected. The default value is "default" | -| `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28271)) | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | - -### Delete Pipeline-Emails service - -Delete Pipeline-Emails service for a project. - -```plaintext -DELETE /projects/:id/services/pipelines-email -``` - -### Get Pipeline-Emails service settings - -Get Pipeline-Emails service settings for a project. - -```plaintext -GET /projects/:id/services/pipelines-email -``` - -## Pivotal Tracker - -Add commit messages as comments to Pivotal Tracker stories. - -See also the [Pivotal Tracker service documentation](../user/project/integrations/pivotal_tracker.md). - -### Create/Edit Pivotal Tracker service - -Set Pivotal Tracker service for a project. - -```plaintext -PUT /projects/:id/services/pivotaltracker -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `token` | string | true | The Pivotal Tracker token | -| `restrict_to_branch` | boolean | false | Comma-separated list of branches to automatically inspect. Leave blank to include all branches. | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Pivotal Tracker service - -Delete Pivotal Tracker service for a project. - -```plaintext -DELETE /projects/:id/services/pivotaltracker -``` - -### Get Pivotal Tracker service settings - -Get Pivotal Tracker service settings for a project. - -```plaintext -GET /projects/:id/services/pivotaltracker -``` - -## Prometheus - -Prometheus is a powerful time-series monitoring service. - -### Create/Edit Prometheus service - -Set Prometheus service for a project. - -```plaintext -PUT /projects/:id/services/prometheus -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `api_url` | string | true | Prometheus API Base URL. For example, `http://prometheus.example.com/`. | -| `google_iap_audience_client_id` | string | false | Client ID of the IAP secured resource (looks like IAP_CLIENT_ID.apps.googleusercontent.com) | -| `google_iap_service_account_json` | string | false | `credentials.json` file for your service account, like { "type": "service_account", "project_id": ... } | - -### Delete Prometheus service - -Delete Prometheus service for a project. - -```plaintext -DELETE /projects/:id/services/prometheus -``` - -### Get Prometheus service settings - -Get Prometheus service settings for a project. - -```plaintext -GET /projects/:id/services/prometheus -``` - -## Pushover - -Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop. - -### Create/Edit Pushover service - -Set Pushover service for a project. - -```plaintext -PUT /projects/:id/services/pushover -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `api_key` | string | true | Your application key | -| `user_key` | string | true | Your user key | -| `priority` | string | true | The priority | -| `device` | string | false | Leave blank for all active devices | -| `sound` | string | false | The sound of the notification | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Pushover service - -Delete Pushover service for a project. - -```plaintext -DELETE /projects/:id/services/pushover -``` - -### Get Pushover service settings - -Get Pushover service settings for a project. - -```plaintext -GET /projects/:id/services/pushover -``` - -## Redmine - -Redmine issue tracker - -### Create/Edit Redmine service - -Set Redmine service for a project. - -```plaintext -PUT /projects/:id/services/redmine -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `new_issue_url` | string | true | New Issue URL | -| `project_url` | string | true | Project URL | -| `issues_url` | string | true | Issue URL | -| `description` | string | false | Description | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete Redmine service - -Delete Redmine service for a project. - -```plaintext -DELETE /projects/:id/services/redmine -``` - -### Get Redmine service settings - -Get Redmine service settings for a project. - -```plaintext -GET /projects/:id/services/redmine -``` - -## Slack notifications - -Receive event notifications in Slack - -### Create/Edit Slack service - -Set Slack service for a project. - -```plaintext -PUT /projects/:id/services/slack -``` - -NOTE: -Specific event parameters (for example, `push_events` flag and `push_channel`) were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/11435) in GitLab 10.4. - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | `https://hooks.slack.com/services/...` | -| `username` | string | false | username | -| `channel` | string | false | Default channel to use if others are not configured | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `commit_events` | boolean | false | Enable notifications for commit events | -| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `deployment_channel` | string | false | The name of the channel to receive deployment events notifications | -| `deployment_events` | boolean | false | Enable notifications for deployment events | -| `issue_channel` | string | false | The name of the channel to receive issues events notifications | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `job_events` | boolean | false | Enable notifications for job events | -| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `note_channel` | string | false | The name of the channel to receive note events notifications | -| `note_events` | boolean | false | Enable notifications for note events | -| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `push_channel` | string | false | The name of the channel to receive push events notifications | -| `push_events` | boolean | false | Enable notifications for push events | -| `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -| `vulnerability_channel` | string | false | **(ULTIMATE)** The name of the channel to receive vulnerability event notifications. | -| `vulnerability_events` | boolean | false | **(ULTIMATE)** Enable notifications for vulnerability events | - -### Delete Slack service - -Delete Slack service for a project. - -```plaintext -DELETE /projects/:id/services/slack -``` - -### Get Slack service settings - -Get Slack service settings for a project. - -```plaintext -GET /projects/:id/services/slack -``` - -## Microsoft Teams - -Group Chat Software - -### Create/Edit Microsoft Teams service - -Set Microsoft Teams service for a project. - -```plaintext -PUT /projects/:id/services/microsoft-teams -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Microsoft Teams webhook. For example, `https://outlook.office.com/webhook/...` | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | - -### Delete Microsoft Teams service - -Delete Microsoft Teams service for a project. - -```plaintext -DELETE /projects/:id/services/microsoft-teams -``` - -### Get Microsoft Teams service settings - -Get Microsoft Teams service settings for a project. - -```plaintext -GET /projects/:id/services/microsoft-teams -``` - -## Mattermost notifications - -Receive event notifications in Mattermost - -### Create/Edit Mattermost notifications service - -Set Mattermost service for a project. - -```plaintext -PUT /projects/:id/services/mattermost -``` - -NOTE: -Specific event parameters (for example, `push_events` flag and `push_channel`) were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/11435) in GitLab 10.4. - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `webhook` | string | true | The Mattermost webhook. For example, `http://mattermost_host/hooks/...` | -| `username` | string | false | username | -| `channel` | string | false | Default channel to use if others are not configured | -| `notify_only_broken_pipelines` | boolean | false | Send notifications for broken pipelines | -| `notify_only_default_branch` | boolean | false | DEPRECATED: This parameter has been replaced with `branches_to_be_notified` | -| `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". The default value is "default" | -| `push_events` | boolean | false | Enable notifications for push events | -| `issues_events` | boolean | false | Enable notifications for issue events | -| `confidential_issues_events` | boolean | false | Enable notifications for confidential issue events | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events | -| `tag_push_events` | boolean | false | Enable notifications for tag push events | -| `note_events` | boolean | false | Enable notifications for note events | -| `confidential_note_events` | boolean | false | Enable notifications for confidential note events | -| `pipeline_events` | boolean | false | Enable notifications for pipeline events | -| `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -| `vulnerability_events` | boolean | false | **(ULTIMATE)** Enable notifications for vulnerability events | -| `push_channel` | string | false | The name of the channel to receive push events notifications | -| `issue_channel` | string | false | The name of the channel to receive issues events notifications | -| `confidential_issue_channel` | string | false | The name of the channel to receive confidential issues events notifications | -| `merge_request_channel` | string | false | The name of the channel to receive merge request events notifications | -| `note_channel` | string | false | The name of the channel to receive note events notifications | -| `confidential_note_channel` | string | false | The name of the channel to receive confidential note events notifications | -| `tag_push_channel` | string | false | The name of the channel to receive tag push events notifications | -| `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | -| `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | -| `vulnerability_channel` | string | false | **(ULTIMATE)** The name of the channel to receive vulnerability events notifications | - -### Delete Mattermost notifications service - -Delete Mattermost Notifications service for a project. - -```plaintext -DELETE /projects/:id/services/mattermost -``` - -### Get Mattermost notifications service settings - -Get Mattermost notifications service settings for a project. - -```plaintext -GET /projects/:id/services/mattermost -``` - -## JetBrains TeamCity CI - -A continuous integration and build server - -### Create/Edit JetBrains TeamCity CI service - -Set JetBrains TeamCity CI service for a project. - -> The build configuration in TeamCity must use the build format number `%build.vcs.number%`. Configure monitoring of all branches so merge requests build. That setting is in the VSC root advanced settings. - -```plaintext -PUT /projects/:id/services/teamcity -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `teamcity_url` | string | true | TeamCity root URL. For example, `https://teamcity.example.com` | -| `build_type` | string | true | Build configuration ID | -| `username` | string | true | A user with permissions to trigger a manual build | -| `password` | string | true | The password of the user | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete JetBrains TeamCity CI service - -Delete JetBrains TeamCity CI service for a project. - -```plaintext -DELETE /projects/:id/services/teamcity -``` - -### Get JetBrains TeamCity CI service settings - -Get JetBrains TeamCity CI service settings for a project. - -```plaintext -GET /projects/:id/services/teamcity -``` - -## Jenkins CI - -A continuous integration and build server - -### Create/Edit Jenkins CI service - -Set Jenkins CI service for a project. - -```plaintext -PUT /projects/:id/services/jenkins -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `jenkins_url` | string | true | Jenkins URL like `http://jenkins.example.com`. | -| `project_name` | string | true | The URL-friendly project name. Example: `my_project_name`. | -| `username` | string | false | Username for authentication with the Jenkins server, if authentication is required by the server. | -| `password` | string | false | Password for authentication with the Jenkins server, if authentication is required by the server. | -| `push_events` | boolean | false | Enable notifications for push events. | -| `merge_requests_events` | boolean | false | Enable notifications for merge request events. | -| `tag_push_events` | boolean | false | Enable notifications for tag push events. | - -### Delete Jenkins CI service - -Delete Jenkins CI service for a project. - -```plaintext -DELETE /projects/:id/services/jenkins -``` - -### Get Jenkins CI service settings - -Get Jenkins CI service settings for a project. - -```plaintext -GET /projects/:id/services/jenkins -``` - -## Jenkins CI (Deprecated) Service - -A continuous integration and build server - -NOTE: -This service was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1600) in GitLab 13.0. - -### Create/Edit Jenkins CI (Deprecated) service - -Set Jenkins CI (Deprecated) service for a project. - -```plaintext -PUT /projects/:id/services/jenkins-deprecated -``` - -Parameters: - -- `project_url` (**required**) - Jenkins project URL like `http://jenkins.example.com/job/my-project/` -- `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin -- `pass_unstable` (optional) - Unstable builds are treated as passing - -### Delete Jenkins CI (Deprecated) service - -Delete Jenkins CI (Deprecated) service for a project. - -```plaintext -DELETE /projects/:id/services/jenkins-deprecated -``` - -### Get Jenkins CI (Deprecated) service settings - -Get Jenkins CI (Deprecated) service settings for a project. - -```plaintext -GET /projects/:id/services/jenkins-deprecated -``` - -## MockCI - -Mock an external CI. See [`gitlab-org/gitlab-mock-ci-service`](https://gitlab.com/gitlab-org/gitlab-mock-ci-service) for an example of a companion mock service. - -This service is only available when your environment is set to development. - -### Create/Edit MockCI service - -Set MockCI service for a project. - -```plaintext -PUT /projects/:id/services/mock-ci -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `mock_service_url` | string | true | `http://localhost:4004` | - -### Delete MockCI service - -Delete MockCI service for a project. - -```plaintext -DELETE /projects/:id/services/mock-ci -``` - -### Get MockCI service settings - -Get MockCI service settings for a project. - -```plaintext -GET /projects/:id/services/mock-ci -``` - -## YouTrack - -YouTrack issue tracker - -### Create/Edit YouTrack service - -Set YouTrack service for a project. - -```plaintext -PUT /projects/:id/services/youtrack -``` - -Parameters: - -| Parameter | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `issues_url` | string | true | Issue URL | -| `project_url` | string | true | Project URL | -| `description` | string | false | Description | -| `push_events` | boolean | false | Enable notifications for push events | - -### Delete YouTrack Service - -Delete YouTrack service for a project. - -```plaintext -DELETE /projects/:id/services/youtrack -``` - -### Get YouTrack Service Settings - -Get YouTrack service settings for a project. - -```plaintext -GET /projects/:id/services/youtrack -``` +<!-- This redirect file can be deleted after <2021-11-09>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/api/settings.md b/doc/api/settings.md index 6b1faa0402f..7b8778973f2 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -248,7 +248,7 @@ listed in the descriptions of the relevant settings. | `deactivate_dormant_users` | boolean | no | Enable [automatic deactivation of dormant users](../user/admin_area/moderate_users.md#automatically-deactivate-dormant-users). | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_name` | string | no | [Instance-level custom initial branch name](../user/project/repository/branches/default.md#instance-level-custom-initial-branch-name) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225258) in GitLab 13.2). | -| `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both developers and maintainers can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push, or delete, the branch)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | +| `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both developers and maintainers can push new commits and force push)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no one can force push)_ as a parameter. Default is `2`. | | `default_ci_config_path` | string | no | Default CI/CD configuration file and path for new projects (`.gitlab-ci.yml` if not set). | | `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_project_creation` | integer | no | Default project creation protection. Can take: `0` _(No one)_, `1` _(Maintainers)_ or `2` _(Developers + Maintainers)_| @@ -407,6 +407,7 @@ listed in the descriptions of the relevant settings. | `spam_check_endpoint_enabled` | boolean | no | Enables Spam Check via external API endpoint. Default is `false`. | | `spam_check_endpoint_url` | string | no | URL of the external Spam Check service endpoint. | | `spam_check_api_key` | string | no | The API key used by GitLab for accessing the Spam Check service endpoint. | +| `suggest_pipeline_enabled` | boolean | no | Enable pipeline suggestion banner. | | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | diff --git a/doc/api/snippet_repository_storage_moves.md b/doc/api/snippet_repository_storage_moves.md index a73542c8505..81473fdff91 100644 --- a/doc/api/snippet_repository_storage_moves.md +++ b/doc/api/snippet_repository_storage_moves.md @@ -10,7 +10,7 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49228) in GitLab 13.8. Snippet repositories can be moved between storages. This can be useful when -[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrate-to-gitaly-cluster), for +[migrating to Gitaly Cluster](../administration/gitaly/index.md#migrating-to-gitaly-cluster), for example. As snippet repository storage moves are processed, they transition through different states. Values diff --git a/doc/api/usage_data.md b/doc/api/usage_data.md index f244c681086..4809166f357 100644 --- a/doc/api/usage_data.md +++ b/doc/api/usage_data.md @@ -13,7 +13,7 @@ The Service Data API is associated with [Service Ping](../development/service_pi > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57270) in GitLab 13.11. -Export all metric definitions as a single YAML file, similar to the [Metrics Dictionary](https://gitlab-org.gitlab.io/growth/product-intelligence/metric-dictionary), for easier importing. +Export all metric definitions as a single YAML file, similar to the [Metrics Dictionary](https://metrics.gitlab.com/index.html), for easier importing. ```plaintext GET /usage_data/metric_definitions diff --git a/doc/api/users.md b/doc/api/users.md index 4ec609e62e9..d8effc4d38f 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -987,7 +987,7 @@ error occurs a `400 Bad Request` is returned with a message explaining the error ## Add SSH key for user -Create new key owned by specified user. Available only for admin +Create new key owned by specified user. Available only for administrator. ```plaintext POST /users/:id/keys @@ -1018,7 +1018,7 @@ Parameters: ## Delete SSH key for given user -Deletes key owned by a specified user. Available only for admin. +Deletes key owned by a specified user. Available only for administrator. ```plaintext DELETE /users/:id/keys/:key_id @@ -1165,7 +1165,7 @@ Example response: ## Get a specific GPG key for a given user Get a specific GPG key for a given user. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43693) -in GitLab 13.5, this endpoint can be accessed without admin authentication. +in GitLab 13.5, this endpoint can be accessed without administrator authentication. ```plaintext GET /users/:id/gpg_keys/:key_id @@ -1194,7 +1194,7 @@ Example response: ## Add a GPG key for a given user -Create new GPG key owned by the specified user. Available only for admins. +Create new GPG key owned by the specified user. Available only for administrator. ```plaintext POST /users/:id/gpg_keys @@ -1226,7 +1226,7 @@ Example response: ## Delete a GPG key for a given user -Delete a GPG key owned by a specified user. Available only for admins. +Delete a GPG key owned by a specified user. Available only for administrator. ```plaintext DELETE /users/:id/gpg_keys/:key_id @@ -1276,7 +1276,7 @@ Parameters: ## List emails for user -Get a list of a specified user's emails. Available only for admin +Get a list of a specified user's emails. Available only for administrator NOTE: Due to [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/25077) this endpoint currently @@ -1345,7 +1345,7 @@ error occurs a `400 Bad Request` is returned with a message explaining the error ## Add email for user -Create new email owned by specified user. Available only for admin +Create new email owned by specified user. Available only for administrator ```plaintext POST /users/:id/emails @@ -1372,7 +1372,7 @@ Parameters: ## Delete email for given user -Deletes email owned by a specified user. Available only for admin. +Deletes email owned by a specified user. Available only for administrator. ```plaintext DELETE /users/:id/emails/:email_id @@ -1385,7 +1385,7 @@ Parameters: ## Block user -Blocks the specified user. Available only for admin. +Blocks the specified user. Available only for administrator. ```plaintext POST /users/:id/block @@ -1405,7 +1405,7 @@ Returns: ## Unblock user -Unblocks the specified user. Available only for admin. +Unblocks the specified user. Available only for administrator. ```plaintext POST /users/:id/unblock @@ -1422,7 +1422,7 @@ Returns `201 OK` on success, `404 User Not Found` is user cannot be found or > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22257) in GitLab 12.4. -Deactivates the specified user. Available only for admin. +Deactivates the specified user. Available only for administrator. ```plaintext POST /users/:id/deactivate @@ -1437,7 +1437,7 @@ Returns: - `201 OK` on success. - `404 User Not Found` if user cannot be found. - `403 Forbidden` when trying to deactivate a user: - - Blocked by admin or by LDAP synchronization. + - Blocked by administrator or by LDAP synchronization. - That has any activity in past 90 days. These users cannot be deactivated. - That is internal. @@ -1445,7 +1445,7 @@ Returns: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22257) in GitLab 12.4. -Activates the specified user. Available only for admin. +Activates the specified user. Available only for administrator. ```plaintext POST /users/:id/activate @@ -1465,7 +1465,7 @@ Returns: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327354) in GitLab 14.3. -Bans the specified user. Available only for admin. +Bans the specified user. Available only for administrator. ```plaintext POST /users/:id/ban @@ -1485,7 +1485,7 @@ Returns: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327354) in GitLab 14.3. -Unbans the specified user. Available only for admin. +Unbans the specified user. Available only for administrator. ```plaintext POST /users/:id/unban @@ -1507,7 +1507,7 @@ Please refer to the [Events API documentation](events.md#get-user-contribution-e ## Get all impersonation tokens of a user -> Requires admin permissions. +> Requires administrator permissions. It retrieves every impersonation token of the user. Use the pagination parameters `page` and `per_page` to restrict the list of impersonation tokens. @@ -1639,7 +1639,7 @@ Example Responses: ## Get an impersonation token of a user -> Requires admin permissions. +> Requires administrators permissions. It shows a user's impersonation token. @@ -1678,7 +1678,7 @@ Example response: ## Create an impersonation token -> Requires admin permissions. +> Requires administrator permissions. > Token values are returned once. Make sure you save it - you can't access it again. It creates a new impersonation token. Only administrators can do this. @@ -1723,7 +1723,7 @@ Example response: ## Revoke an impersonation token -> Requires admin permissions. +> Requires administrator permissions. It revokes an impersonation token. @@ -1845,7 +1845,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20532) in GitLab 12.8. -Lists all projects and groups a user is a member of. This endpoint is available for admins only. +Lists all projects and groups a user is a member of. This endpoint is available for administrators only. It returns the `source_id`, `source_name`, `source_type` and `access_level` of a membership. Source can be of type `Namespace` (representing a group) or `Project`. The response represents only direct memberships. Inherited memberships, for example in subgroups, are not included. Access levels are represented by an integer value. For more details, read about the meaning of [access level values](access_requests.md#valid-access-levels). @@ -1865,7 +1865,7 @@ Returns: - `200 OK` on success. - `404 User Not Found` if user can't be found. -- `403 Forbidden` when not requested by an admin. +- `403 Forbidden` when not requested by an administrator. - `400 Bad Request` when requested type is not supported. ```shell diff --git a/doc/api/version.md b/doc/api/version.md index b076993f00e..80269bc5697 100644 --- a/doc/api/version.md +++ b/doc/api/version.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Version API **(FREE)** -> Introduced in GitLab 8.13. - Retrieve version information for this GitLab instance. Responds `200 OK` for authenticated users. diff --git a/doc/api/vulnerability_findings.md b/doc/api/vulnerability_findings.md index dfc6074a1aa..36604ebf87d 100644 --- a/doc/api/vulnerability_findings.md +++ b/doc/api/vulnerability_findings.md @@ -74,77 +74,66 @@ Example response: [ { "id": null, - "report_type": "dependency_scanning", - "name": "Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js", - "severity": "unknown", - "confidence": "undefined", + "report_type": "sast", + "name": "Possible command injection", + "severity": "high", + "confidence": "high", "scanner": { - "external_id": "gemnasium", - "name": "Gemnasium" + "external_id": "brakeman", + "name": "Brakeman", + "vendor": "GitLab" }, "identifiers": [ { - "external_type": "gemnasium", - "external_id": "9952e574-7b5b-46fa-a270-aeb694198a98", - "name": "Gemnasium-9952e574-7b5b-46fa-a270-aeb694198a98", - "url": "https://deps.sec.gitlab.com/packages/npm/saml2-js/versions/1.5.0/advisories" - }, - { - "external_type": "cve", - "external_id": "CVE-2017-11429", - "name": "CVE-2017-11429", - "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11429" + "external_type": "brakeman_warning_code", + "external_id": "14", + "name": "Brakeman Warning Code 14", + "url": "https://brakemanscanner.org/docs/warning_types/command_injection/" } ], - "project_fingerprint": "fa6f5b6c5d240b834ac5e901dc69f9484cef89ec", - "uuid": "31f483bc-bfc0-586d-9b92-f1015c4535b8", - "create_vulnerability_feedback_issue_path": "/tests/yarn-remediation-test/vulnerability_feedback", - "create_vulnerability_feedback_merge_request_path": "/tests/yarn-remediation-test/vulnerability_feedback", - "create_vulnerability_feedback_dismissal_path": "/tests/yarn-remediation-test/vulnerability_feedback", + "project_fingerprint": "ac218b1770af030cfeef967752ab803c55afb36d", + "uuid": "ad5e3be3-a193-55f5-a200-bc12865fb09c", + "create_jira_issue_url": null, + "false_positive": true, + "create_vulnerability_feedback_issue_path": "/root/test-false-positive/-/vulnerability_feedback", + "create_vulnerability_feedback_merge_request_path": "/root/test-false-positive/-/vulnerability_feedback", + "create_vulnerability_feedback_dismissal_path": "/root/test-false-positive/-/vulnerability_feedback", "project": { - "id": 31, - "name": "yarn-remediation-test", - "full_path": "/tests/yarn-remediation-test", - "full_name": "tests / yarn-remediation-test" + "id": 2, + "name": "Test False Positive", + "full_path": "/root/test-false-positive", + "full_name": "Administrator / Test False Positive" }, "dismissal_feedback": null, "issue_feedback": null, "merge_request_feedback": null, - "description": "Some XML DOM traversal and canonicalization APIs may be inconsistent in handling of comments within XML nodes. Incorrect use of these APIs by some SAML libraries results in incorrect parsing of the inner text of XML nodes such that any inner text after the comment is lost prior to cryptographically signing the SAML message. Text after the comment therefore has no impact on the signature on the SAML message.\r\n\r\nA remote attacker can modify SAML content for a SAML service provider without invalidating the cryptographic signature, which may allow attackers to bypass primary authentication for the affected SAML service provider.", - "links": [ - { - "url": "https://github.com/Clever/saml2/commit/3546cb61fd541f219abda364c5b919633609ef3d#diff-af730f9f738de1c9ad87596df3f6de84R279" - }, - { - "url": "https://www.kb.cert.org/vuls/id/475445" - }, - { - "url": "https://github.com/Clever/saml2/issues/127" - } - ], + "description": null, + "links": [], "location": { - "file": "yarn.lock", - "dependency": { - "package": { - "name": "saml2-js" - }, - "version": "1.5.0" - } + "file": "app/controllers/users_controller.rb", + "start_line": 42, + "class": "UsersController", + "method": "list_users" }, - "details": { - "custom_field": { - "name": "URLs", - "type": "list", - "items": [ - { - "type": "url", - "href": "http://site.com/page/1" - } - ] - } + "remediations": [ + null + ], + "solution": null, + "evidence": null, + "request": null, + "response": null, + "evidence_source": null, + "supporting_messages": [], + "assets": [], + "details": {}, + "state": "detected", + "scan": { + "type": "sast", + "status": "success", + "start_time": "2021-09-02T20:55:48", + "end_time": "2021-09-02T20:55:48" }, - "solution": "Upgrade to fixed version.\r\n", - "blob_path": "/tests/yarn-remediation-test/blob/cc6c4a0778460455ae5d16ca7025ca9ca1ca75ac/yarn.lock" + "blob_path": "/root/test-false-positive/-/blob/dfd75607752a839bbc9c7362d111effaa470fecd/app/controllers/users_controller.rb#L42" } ] ``` diff --git a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md index 095eaaec23f..82529a4c199 100644 --- a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md +++ b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md @@ -495,8 +495,8 @@ Pros: Cons: - It is harder to implement GraphQL subscriptions as in case of Sidekiq as we need another way to pass subscriptions -- `api_v4` paths can be used in some services that are used by Sidekiq (e.g. `api_v4_projects_path`) -- url_helpers paths are used in models and services, that could be used by Sidekiq (e.g. `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18)) +- `api_v4` paths can be used in some services that are used by Sidekiq (for example `api_v4_projects_path`) +- url_helpers paths are used in models and services, that could be used by Sidekiq (for example `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18)) #### Example: GraphQL @@ -514,7 +514,7 @@ Today, loading GraphQL requires a bunch of [dependencies](https://gitlab.com/git GraphQL only needs to run in a specific context. If we could limit when it is being loaded we could effectively improve application efficiency, by reducing application load time and required memory. This, for example, is applicable for every size installation. -A potential challenge with GraphQL and Websockets is that at some point we might want to use Action Cable subscriptions and push GraphQL/API payload from Sidekiq to clients. This would likely utilize Redis to pass data through. Where Sidekiq would publish information on Redis and ActionCable Node would pass through that information to connected clients. This way of working is possible in the above model, but we would have to use GraphQL or API (over HTTP endpoint) to calculate what should be sent. +A potential challenge with GraphQL and Websockets is that at some point we might want to use Action Cable subscriptions and push GraphQL/API payload from Sidekiq to clients. This would likely use Redis to pass data through. Where Sidekiq would publish information on Redis and ActionCable Node would pass through that information to connected clients. This way of working is possible in the above model, but we would have to use GraphQL or API (over HTTP endpoint) to calculate what should be sent. An alternative way is to use a notification system that would always make an `ActionCable` node (the one handling WebSockets) generate a payload based on a send query instead of performing passthrough. This could be applicable since `ActionCable` is the one handling a given connection for a client. This could have a downside of having to recalculate the same payload if many clients would be watching the same resource. However, this behavior of system might still be desired for security purposes, as generated payload might be dependent on permission of watching client (we would show different for anonymous, and different for the member of the project). @@ -529,8 +529,8 @@ Grape::API is another example that only needs to run only in a web server contex Potential challenges with Grape API: -- Currently there are some API::API dependencies in the models (e.g. `API::Helpers::Version` dependency in [project model](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project.rb#L2019) or API::API dependency in GeoNode model for [`geo_retrieve_url`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node.rb#L183)) -- `api_v4` paths are used in helpers, presenters, and views (e.g. `api_v4_projects_path` in [PackagesHelper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/packages_helper.rb#L17)) +- Currently there are some API::API dependencies in the models (for example `API::Helpers::Version` dependency in [project model](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project.rb#L2019) or API::API dependency in GeoNode model for [`geo_retrieve_url`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node.rb#L183)) +- `api_v4` paths are used in helpers, presenters, and views (for example `api_v4_projects_path` in [PackagesHelper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/packages_helper.rb#L17)) #### Example: Controllers @@ -544,7 +544,7 @@ Controllers, Serializers, some presenters and some of the Grape:Entities are als Potential challenges with moving Controllers: - We needed to extend `Gitlab::Patch::DrawRoute` in order to support `engines/web_engine/config/routes` and `engines/web_engine/ee/config/routes` in case when `web_engine` is loaded. Here is potential [solution](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720#note_506957398). -- `Gitlab::Routing.url_helpers` paths are used in models and services, that could be used by Sidekiq (e.g. `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18))) +- `Gitlab::Routing.url_helpers` paths are used in models and services, that could be used by Sidekiq (for example `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18))) ### Packwerk diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index fab886808e2..a8d87e5f967 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -32,7 +32,7 @@ they all live on their own. A few more problems with this approach: - Features are coupled to their container. In practice it is not straight forward to decouple a feature from its container. The degree of coupling varies across features. -- Naive duplication of features will result in a more complex and fragile code base. +- Naive duplication of features will result in a more complex and fragile codebase. - Generalizing solutions across groups and projects may degrade system performance. - The range of features span across many teams, and these changes will need to manage development interference. diff --git a/doc/ci/README.md b/doc/ci/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/chatops/index.md b/doc/ci/chatops/index.md index a2d9cf9054d..698b467a813 100644 --- a/doc/ci/chatops/index.md +++ b/doc/ci/chatops/index.md @@ -9,6 +9,7 @@ type: index, concepts, howto > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4466) in GitLab Ultimate 10.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) to GitLab Free in 11.9. +> - `CHAT_USER_ID` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341798) in GitLab 14.4. GitLab ChatOps provides a method to interact with CI/CD jobs through chat services like Slack. Many organizations' discussion, collaboration, and troubleshooting takes @@ -17,7 +18,7 @@ posted back to the channel can significantly augment your team's workflow. ## How GitLab ChatOps works -GitLab ChatOps is built upon [GitLab CI/CD](../README.md) and +GitLab ChatOps is built upon [GitLab CI/CD](../index.md) and [Slack Slash Commands](../../user/project/integrations/slack_slash_commands.md). ChatOps provides a `run` action for [slash commands](../../integration/slash_commands.md) with the following arguments: @@ -30,6 +31,7 @@ to the job: - `CHAT_INPUT` contains any additional arguments. - `CHAT_CHANNEL` is set to the name of channel the action was triggered in. +- `CHAT_USER_ID` is set to the chat service's user ID of the user who triggered the slash command. When executed, ChatOps looks up the specified job name and attempts to match it to a corresponding job in [`.gitlab-ci.yml`](../yaml/index.md). If a matching job diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index e69daf6651a..cf4e846ebb4 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -21,7 +21,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository:  - GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository). + GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/mirror/pull.md). 1. In GitLab, create a [Personal Access Token](../../user/profile/personal_access_tokens.md) diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index 70a9c8fc775..4e4b7420697 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -45,7 +45,7 @@ repositories: GitLab: 1. Imports the project. -1. Enables [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository). +1. Enables [Pull Mirroring](../../user/project/repository/mirror/pull.md). 1. Enables [GitHub project integration](../../user/project/integrations/github.md). 1. Creates a web hook on GitHub to notify GitLab of new commits. diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index 365e2ee31de..fbfcdcbf64f 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -18,7 +18,7 @@ GitLab CI/CD can be used with: Instead of moving your entire project to GitLab, you can connect your external repository to get the benefits of GitLab CI/CD. -Connecting an external repository sets up [repository mirroring](../../user/project/repository/repository_mirroring.md) +Connecting an external repository sets up [repository mirroring](../../user/project/repository/mirror/index.md) and create a lightweight project with issues, merge requests, wiki, and snippets disabled. These features [can be re-enabled later](../../user/project/settings/index.md#sharing-and-permissions). diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 25b87366a30..408d68fb726 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -58,6 +58,9 @@ Some credentials are required to be able to run `aws` commands: | `AWS_SECRET_ACCESS_KEY` | Your Secret access key | | `AWS_DEFAULT_REGION` | Your region code | + NOTE: + When you create a variable it's set to be [protected by default](../variables/index.md#protect-a-cicd-variable). If you want to use the `aws` commands on branches or tags that are not protected, make sure to uncheck the **Protect variable** checkbox. + 1. You can now use `aws` commands in the `.gitlab-ci.yml` file of this project: ```yaml diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index d5adedc611c..9a4290ead4c 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -572,7 +572,7 @@ The configuration is picked up by the `dind` service. ## Authenticate with registry in Docker-in-Docker When you use Docker-in-Docker, the -[standard authentication methods](using_docker_images.md#define-an-image-from-a-private-container-registry) +[standard authentication methods](using_docker_images.md#access-an-image-from-a-private-container-registry) don't work because a fresh Docker daemon is started with the service. ### Option 1: Run `docker login` diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index c2991ce66f9..79c23d73a68 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -214,7 +214,7 @@ Look for the `[runners.docker]` section: The image and services defined this way are added to all jobs run by that runner. -## Define an image from a private Container Registry +## Access an image from a private Container Registry To access private container registries, the GitLab Runner process can use: @@ -224,19 +224,12 @@ To access private container registries, the GitLab Runner process can use: To define which option should be used, the runner process reads the configuration in this order: -- A `DOCKER_AUTH_CONFIG` variable provided as either: - - A [CI/CD variable](../variables/index.md) in the `.gitlab-ci.yml` file. - - A project's variables stored on the project's **Settings > CI/CD** page. -- A `DOCKER_AUTH_CONFIG` variable provided as environment variable in the runner's `config.toml` file. +- A `DOCKER_AUTH_CONFIG` [CI/CD variable](../variables/index.md). +- A `DOCKER_AUTH_CONFIG` environment variable set in the runner's `config.toml` file. - A `config.json` file in `$HOME/.docker` directory of the user running the process. If the `--user` flag is provided to run the child processes as unprivileged user, the home directory of the main runner process user is used. -The runner reads this configuration **only** from the `config.toml` file and ignores it if -it's provided as a CI/CD variable. This is because the runner uses **only** -`config.toml` configuration and does not interpolate **any** CI/CD variables at -runtime. - ### Requirements and limitations - Available for [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html) @@ -253,9 +246,9 @@ private registry. Both require setting the CI/CD variable `DOCKER_AUTH_CONFIG` with appropriate authentication information. 1. Per-job: To configure one job to access a private registry, add - `DOCKER_AUTH_CONFIG` as a job variable. + `DOCKER_AUTH_CONFIG` as a [CI/CD variable](../variables/index.md). 1. Per-runner: To configure a runner so all its jobs can access a - private registry, add `DOCKER_AUTH_CONFIG` to the environment in the + private registry, add `DOCKER_AUTH_CONFIG` as an environment variable in the runner's configuration. See below for examples of each. @@ -274,7 +267,7 @@ Let's also assume that these are the sign-in credentials: | username | `my_username` | | password | `my_password` | -Use one of the following methods to determine the value of `DOCKER_AUTH_CONFIG`: +Use one of the following methods to determine the value for `DOCKER_AUTH_CONFIG`: - Do a `docker login` on your local machine: diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 6886899a54b..69c4557dcbe 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -13,8 +13,8 @@ type: howto container images from a Dockerfile, inside a container or Kubernetes cluster. kaniko solves two problems with using the -[Docker-in-Docker -build](using_docker_build.md#use-the-docker-executor-with-the-docker-image-docker-in-docker) method: +[Docker-in-Docker build](using_docker_build.md#use-the-docker-executor-with-the-docker-image-docker-in-docker) +method: - Docker-in-Docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) to function, which is a significant security concern. @@ -64,7 +64,7 @@ build: entrypoint: [""] script: - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG rules: - if: $CI_COMMIT_TAG @@ -91,7 +91,7 @@ build: - mkdir -p /kaniko/.docker - |- KANIKOPROXYBUILDARGS="" - KANIKOCFG="{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" + KANIKOCFG="{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" if [ "x${http_proxy}" != "x" -o "x${https_proxy}" != "x" ]; then KANIKOCFG="${KANIKOCFG}, \"proxies\": { \"default\": { \"httpProxy\": \"${http_proxy}\", \"httpsProxy\": \"${https_proxy}\", \"noProxy\": \"${no_proxy}\"}}" KANIKOPROXYBUILDARGS="--build-arg http_proxy=${http_proxy} --build-arg https_proxy=${https_proxy} --build-arg no_proxy=${no_proxy}" @@ -120,7 +120,7 @@ store: ```yaml before_script: - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > /kaniko/.docker/config.json + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json - | echo "-----BEGIN CERTIFICATE----- ... diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 1b34b520007..78f30b29e06 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -60,7 +60,7 @@ The improved pipeline flow **after** using the resource group: 1. `deploy` job in Pipeline-A finishes. 1. `deploy` job in Pipeline-B starts running. -For more information, see [`resource_group` keyword in `.gitlab-ci.yml`](../yaml/index.md#resource_group). +For more information, see [Resource Group documentation](../resource_groups/index.md). ## Skip outdated deployment jobs diff --git a/doc/ci/environments/img/environments_list.png b/doc/ci/environments/img/environments_list.png Binary files differdeleted file mode 100644 index 6ddfd858ab6..00000000000 --- a/doc/ci/environments/img/environments_list.png +++ /dev/null diff --git a/doc/ci/environments/img/environments_list_v14_3.png b/doc/ci/environments/img/environments_list_v14_3.png Binary files differnew file mode 100644 index 00000000000..8fdb85338e7 --- /dev/null +++ b/doc/ci/environments/img/environments_list_v14_3.png diff --git a/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png b/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png Binary files differdeleted file mode 100644 index 4a9a4e65d00..00000000000 --- a/doc/ci/environments/img/environments_terminal_button_on_index_v13_10.png +++ /dev/null diff --git a/doc/ci/environments/img/environments_terminal_button_on_index_v14_3.png b/doc/ci/environments/img/environments_terminal_button_on_index_v14_3.png Binary files differnew file mode 100644 index 00000000000..d1faf489277 --- /dev/null +++ b/doc/ci/environments/img/environments_terminal_button_on_index_v14_3.png diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 6bac004fcdf..ca81ad30317 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -19,7 +19,7 @@ GitLab: - Tracks your deployments, so you always know what is deployed on your servers. -If you have a deployment service like [Kubernetes](../../user/project/clusters/index.md) +If you have a deployment service like [Kubernetes](../../user/infrastructure/clusters/index.md) associated with your project, you can use it to assist with your deployments. You can even access a [web terminal](#web-terminals) for your environment from within GitLab. @@ -35,7 +35,7 @@ To view a list of environments and deployments: 1. On the left sidebar, select **Deployments > Environments**. The environments are displayed. -  +  1. To view a list of deployments for an environment, select the environment name, for example, `staging`. @@ -175,13 +175,13 @@ You can find the play button in the pipelines, environments, deployments, and jo > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. -If you are deploying to a [Kubernetes cluster](../../user/project/clusters/index.md) +If you are deploying to a [Kubernetes cluster](../../user/infrastructure/clusters/index.md) associated with your project, you can configure these deployments from your `.gitlab-ci.yml` file. NOTE: Kubernetes configuration isn't supported for Kubernetes clusters that are -[managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). +[managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). To follow progress on support for GitLab-managed clusters, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). @@ -634,7 +634,7 @@ Metric charts can be embedded in GitLab Flavored Markdown. See [Embedding Metric ### Web terminals If you deploy to your environments with the help of a deployment service (for example, -the [Kubernetes integration](../../user/project/clusters/index.md)), GitLab can open +the [Kubernetes integration](../../user/infrastructure/clusters/index.md)), GitLab can open a terminal session to your environment. You can then debug issues without leaving your web browser. The Web terminal is a container-based deployment, which often lack basic tools (like an editor), @@ -646,9 +646,9 @@ Web terminals: - Are available to project Maintainers and Owners only. - Must [be enabled](../../administration/integration/terminal.md). -In the UI, you can view the Web terminal by selecting a **Terminal** button: +In the UI, you can view the Web terminal by selecting **Terminal** from the actions menu: - + You can also access the terminal button from the page for a specific environment: @@ -816,3 +816,62 @@ To ensure the `action: stop` can always run when needed, you can: action: stop when: manual ``` + +### A deployment job failed with "This job could not be executed because it would create an environment with an invalid parameter" error + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21182) in GitLab 14.4. + +FLAG: +On self-managed GitLab, by default this bug fix is not available. To make it available per project or for your entire instance, ask an administrator to [enable the `surface_environment_creation_failure` flag](../../administration/feature_flags.md). On GitLab.com, this bug fix is not available, but will be rolled out shortly. + +If your project is configured to [create a dynamic environment](#create-a-dynamic-environment), +you might encounter this error because the dynamically generated parameter can't be used for creating an environment. + +For example, your project has the following `.gitlab-ci.yml`: + +```yaml +deploy: + script: echo + environment: production/$ENVIRONMENT +``` + +Since `$ENVIRONMENT` variable does not exist in the pipeline, GitLab tries to +create an environment with a name `production/`, which is invalid in +[the environment name constraint](../yaml/index.md). + +To fix this, use one of the following solutions: + +- Remove `environment` keyword from the deployment job. GitLab has already been + ignoring the invalid keyword, therefore your deployment pipelines stay intact + even after the keyword removal. +- Ensure the variable exists in the pipeline. Please note that there is + [a limitation on supported variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file). + +#### If you get this error on Review Apps + +For example, if you have the following in your `.gitlab-ci.yml`: + +```yaml +review: + script: deploy review app + environment: review/$CI_COMMIT_REF_NAME +``` + +When you create a new merge request with a branch name `bug-fix!`, +the `review` job tries to create an environment with `review/bug-fix!`. +However, the `!` is an invalid character for environments, so the +deployment job fails since it was about to run without an environment. + +To fix this, use one of the following solutions: + +- Re-create your feature branch without the invalid characters, + such as `bug-fix`. +- Replace the `CI_COMMIT_REF_NAME` + [predefined variable](../variables/predefined_variables.md) with + `CI_COMMIT_REF_SLUG` which strips any invalid characters: + + ```yaml + review: + script: deploy review app + environment: review/$CI_COMMIT_REF_SLUG + ``` diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index b31e51b35fc..47f93b03136 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -2,26 +2,22 @@ stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: concepts, howto --- # Protected environments **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6303) in GitLab 11.3. +[Environments](../environments/index.md) can be used for both testing and +production reasons. -[Environments](../environments/index.md) can be used for different reasons: +Because deploy jobs can be raised by different users with different roles, it's +important to be able to protect specific environments from the effects of +unauthorized users. -- Some of them are just for testing. -- Others are for production. - -Since deploy jobs can be raised by different users with different roles, it is important that -specific environments are "protected" to prevent unauthorized people from affecting them. - -By default, a protected environment does one thing: it ensures that only people -with the right privileges can deploy to it, thus keeping it safe. +By default, a protected environment ensures that only people with the +appropriate privileges can deploy to it, keeping the environment safe. NOTE: -A GitLab admin is always allowed to use environments, even if they are protected. +GitLab administrators can use all environments, including protected environments. To protect, update, or unprotect an environment, you need to have at least the [Maintainer role](../../user/permissions.md). @@ -157,9 +153,9 @@ For more information, see [Deployment safety](deployment_safety.md). ## Group-level protected environments -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215888) in GitLab 14.0. [Deployed behind the `group_level_protected_environments` flag](../../administration/feature_flags.md), disabled by default. -> - [Feature flag `group_level_protected_environments`](https://gitlab.com/gitlab-org/gitlab/-/issues/331085) removed in GitLab 14.3. -> - [Generally Available](https://gitlab.com/gitlab-org/gitlab/-/issues/331085) on GitLab and on GitLab.com in 14.3. +> - Introduced in GitLab 14.0 [with a flag](https://gitlab.com/gitlab-org/gitlab/-/issues/215888) named `group_level_protected_environments`. Disabled by default. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/331085) in GitLab 14.3. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/331085) in GitLab 14.3. Typically, large enterprise organizations have an explicit permission boundary between [developers and operators](https://about.gitlab.com/topics/devops/). @@ -210,8 +206,8 @@ configured: (or above) to the top-level group. They can maintain CI/CD configurations for the higher environments (such as production) in the group-level settings page, which includes group-level protected environments, - [group-level runners](../runners/runners_scope.md#group-runners), - [group-level clusters](../../user/group/clusters/index.md), etc. Those + [group-level runners](../runners/runners_scope.md#group-runners), and + [group-level clusters](../../user/group/clusters/index.md). Those configurations are inherited to the child projects as read-only entries. This ensures that only operators can configure the organization-wide deployment ruleset. @@ -246,11 +242,11 @@ To protect a group-level environment: 1. Make sure your environments have the correct [`deployment_tier`](index.md#deployment-tier-of-environments) defined in `.gitlab-ci.yml`. -1. Configure the group-level protected environments via the +1. Configure the group-level protected environments by using the [REST API](../../api/group_protected_environments.md). NOTE: -Configuration [via the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) +Configuration [with the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) is scheduled for a later release. <!-- ## Troubleshooting diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/examples/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/deployment/README.md b/doc/ci/examples/deployment/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/examples/deployment/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md index 6817c7cac8e..aa4055c00ea 100644 --- a/doc/ci/examples/deployment/composer-npm-deploy.md +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -7,7 +7,7 @@ type: tutorial # Running Composer and npm scripts with deployment via SCP in GitLab CI/CD **(FREE)** -This guide covers the building of dependencies of a PHP project while compiling assets via an npm script using [GitLab CI/CD](../../README.md). +This guide covers the building of dependencies of a PHP project while compiling assets via an npm script using [GitLab CI/CD](../../index.md). While it is possible to create your own image with custom PHP and Node.js versions, for brevity we use an existing [Docker image](https://hub.docker.com/r/tetraweb/php/) that contains both PHP and Node.js installed. diff --git a/doc/ci/examples/deployment/index.md b/doc/ci/examples/deployment/index.md index 14fb77dc49f..b083dbb8177 100644 --- a/doc/ci/examples/deployment/index.md +++ b/doc/ci/examples/deployment/index.md @@ -56,7 +56,7 @@ To use different provider take a look at long list of [Supported Providers](http ## Using Dpl with Docker In most cases, you configured [GitLab Runner](https://docs.gitlab.com/runner/) to use your server's shell commands. -This means that all commands are run in the context of local user (e.g. `gitlab_runner` or `gitlab_ci_multi_runner`). +This means that all commands are run in the context of local user (for example `gitlab_runner` or `gitlab_ci_multi_runner`). It also means that most probably in your Docker container you don't have the Ruby runtime installed. You must install it: @@ -69,7 +69,7 @@ staging: - gem install dpl - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY only: - - master + - main ``` The first line `apt-get update -yq` updates the list of available packages, @@ -81,7 +81,7 @@ The above example is valid for all Debian-compatible systems. It's pretty common in the development workflow to have staging (development) and production environments -Let's consider the following example: we would like to deploy the `master` +Let's consider the following example: we would like to deploy the `main` branch to `staging` and all tags to the `production` environment. The final `.gitlab-ci.yml` for that setup would look like this: @@ -92,7 +92,7 @@ staging: - gem install dpl - dpl --provider=heroku --app=my-app-staging --api_key=$HEROKU_STAGING_API_KEY only: - - master + - main production: stage: deploy @@ -105,7 +105,7 @@ production: We created two deploy jobs that are executed on different events: -- `staging`: Executed for all commits pushed to the `master` branch +- `staging`: Executed for all commits pushed to the `main` branch - `production`: Executed for all pushed tags We also use two secure variables: diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index 06074d6edc2..a9794afaf10 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -33,7 +33,7 @@ to write such end-to-end tests, and how to set up GitLab CI/CD to automatically against your new code, on a branch-by-branch basis. For the scope of this article, we will walk you through the process of setting up GitLab CI/CD for end-to-end testing JavaScript-based applications with WebdriverIO, but the general strategy should carry over to other languages. -We assume you are familiar with GitLab, [GitLab CI/CD](../../README.md), [Review Apps](../../review_apps/index.md), and running your app locally, e.g., on `localhost:8000`. +We assume you are familiar with GitLab, [GitLab CI/CD](../../index.md), [Review Apps](../../review_apps/index.md), and running your app locally, e.g., on `localhost:8000`. ## What to test @@ -150,8 +150,8 @@ need to do for this: 1. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/index.md#stages) -`confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` [Docker -image](../../docker/using_docker_images.md). However, WebdriverIO fires up actual browsers +`confidence-check` that is executed _after_ the stage that deploys the review app. It uses the `node:latest` +[Docker image](../../docker/using_docker_images.md). However, WebdriverIO fires up actual browsers to interact with your application, so we need to install and run them. Furthermore, WebdriverIO uses Selenium as a common interface to control different browsers, so we need to install and run Selenium as well. Luckily, the Selenium project provides the Docker images diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md index 2c2c6ecd30f..0fc7b06a584 100644 --- a/doc/ci/examples/index.md +++ b/doc/ci/examples/index.md @@ -9,7 +9,7 @@ type: index # GitLab CI/CD Examples **(FREE)** This page contains links to a variety of examples that can help you understand how to -implement [GitLab CI/CD](../README.md) for your specific use case. +implement [GitLab CI/CD](../index.md) for your specific use case. Examples are available in several forms. As a collection of: diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index c511839b3e4..e2e12235eba 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -17,7 +17,7 @@ date: 2017-08-31 GitLab features our applications with Continuous Integration, and it is possible to easily deploy the new code changes to the production server whenever we want. -In this tutorial, we'll show you how to initialize a [Laravel](https://laravel.com) application and set up our [Envoy](https://laravel.com/docs/master/envoy) tasks, then we'll jump into see how to test and deploy it with [GitLab CI/CD](../README.md) via [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/). +In this tutorial, we'll show you how to initialize a [Laravel](https://laravel.com) application and set up our [Envoy](https://laravel.com/docs/master/envoy) tasks, then we'll jump into see how to test and deploy it with [GitLab CI/CD](../index.md) via [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/). We assume you have a basic experience with Laravel, Linux servers, and you know how to use GitLab. @@ -394,7 +394,7 @@ We have our app ready on GitLab, and we also can deploy it manually. But let's take a step forward to do it automatically with [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) method. We need to check every commit with a set of automated tests to become aware of issues at the earliest, and then, we can deploy to the target environment if we are happy with the result of the tests. -[GitLab CI/CD](../../README.md) allows us to use [Docker](https://www.docker.com) engine to handle the process of testing and deploying our app. +[GitLab CI/CD](../../index.md) allows us to use [Docker](https://www.docker.com) engine to handle the process of testing and deploying our app. In case you're not familiar with Docker, refer to [Set up automated builds](https://docs.docker.com/get-started/). To be able to build, test, and deploy our app with GitLab CI/CD, we need to prepare our work environment. diff --git a/doc/ci/index.md b/doc/ci/index.md index 87b259af62a..2f18bd28642 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -32,10 +32,10 @@ For a complete overview of these methodologies and GitLab CI/CD, read the [Introduction to CI/CD with GitLab](introduction/index.md). <div class="video-fallback"> - Video demonstration of GitLab CI/CD: <a href="https://www.youtube.com/watch?v=1iXFbchozdY">Demo: CI/CD with GitLab</a>. + Video demonstration of continuous integration with GitLab CI/CD: <a href="https://www.youtube.com/watch?v=ljth1Q5oJoo">Continuous Integration with GitLab (overview demo)</a>. </div> <figure class="video-container"> - <iframe src="https://www.youtube.com/embed/1iXFbchozdY" frameborder="0" allowfullscreen="true"> </iframe> + <iframe src="https://www.youtube.com/embed/ljth1Q5oJoo" frameborder="0" allowfullscreen="true"> </iframe> </figure> ## GitLab CI/CD concepts @@ -46,7 +46,7 @@ GitLab CI/CD uses a number of concepts to describe and run your build and deploy |:--------------------------------------------------------|:-------------------------------------------------------------------------------| | [Pipelines](pipelines/index.md) | Structure your CI/CD process through pipelines. | | [CI/CD variables](variables/index.md) | Reuse values based on a variable/value key pair. | -| [Environments](environments/index.md) | Deploy your application to different environments (e.g., staging, production). | +| [Environments](environments/index.md) | Deploy your application to different environments (for example, staging, production). | | [Job artifacts](pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | | [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | | [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own runners to execute your scripts. | @@ -65,7 +65,7 @@ GitLab CI/CD supports numerous configuration options: | [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | | [Pipeline triggers](triggers/index.md) | Trigger pipelines through the API. | | [Pipelines for Merge Requests](pipelines/merge_request_pipelines.md) | Design a pipeline structure for running a pipeline in merge requests. | -| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | +| [Integrate with Kubernetes clusters](../user/infrastructure/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | | [Optimize GitLab and GitLab Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repositories. | | [`.gitlab-ci.yml` full reference](yaml/index.md) | All the attributes you can use with GitLab CI/CD. | @@ -97,7 +97,7 @@ GitLab CI/CD features, grouped by DevOps stage, include: | [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | | [Canary Deployments](../user/project/canary_deployments.md) | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | | [Deploy boards](../user/project/deploy_boards.md) | Check the current health and status of each CI/CD environment running on Kubernetes. | -| [Feature Flags](../operations/feature_flags.md) **(PREMIUM)** | Deploy your features behind Feature Flags. | +| [Feature Flags](../operations/feature_flags.md) | Deploy your features behind Feature Flags. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | | [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | @@ -111,7 +111,7 @@ GitLab CI/CD features, grouped by DevOps stage, include: ## GitLab CI/CD examples -See the [CI/CD examples](examples/README.md) page for example project code and tutorials for +See the [CI/CD examples](examples/index.md) page for example project code and tutorials for using GitLab CI/CD with various: - App frameworks diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 70c22d566e5..308f38b22b7 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -12,9 +12,8 @@ When a pipeline job is about to run, GitLab generates a unique token and injects You can use a GitLab CI/CD job token to authenticate with specific API endpoints: - Packages: - - [Package Registry](../../user/packages/package_registry/index.md). To push to the - Package Registry, you can use [deploy tokens](../../user/project/deploy_tokens/index.md). - - [Container Registry](../../user/packages/container_registry/index.md) + - [Package Registry](../../user/packages/package_registry/index.md#use-gitlab-cicd-to-build-packages). + - [Container Registry](../../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) (the `$CI_REGISTRY_PASSWORD` is `$CI_JOB_TOKEN`). - [Container Registry API](../../api/container_registry.md) (scoped to the job's project, when the `ci_job_token_scope` feature flag is enabled). @@ -24,8 +23,8 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - [Release creation](../../api/releases/index.md#create-a-release). - [Terraform plan](../../user/infrastructure/index.md). -The token has the same permissions to access the API as the user that triggers the -pipeline. Therefore, this user must be assigned to [a role that has the required privileges](../../user/permissions.md#gitlab-cicd-permissions). +The token has the same permissions to access the API as the user that executes the +job. Therefore, this user must be assigned to [a role that has the required privileges](../../user/permissions.md#gitlab-cicd-permissions). The token is valid only while the pipeline job runs. After the job finishes, you can't use the token anymore. @@ -60,20 +59,20 @@ tries to steal tokens from other jobs. ## Limit GitLab CI/CD job token access -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. -> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-ci-job-token-scope-limit). **(FREE SELF)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328553) in GitLab 14.1. [Deployed behind the `:ci_scoped_job_token` feature flag](../../user/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/332272) in GitLab 14.4. -This in-development feature might not be available for your use. There can be -[risks when enabling features still in development](../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). -Refer to this feature's version history for more details. +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the `ci_scoped_job_token` flag](../../administration/feature_flags.md). +On GitLab.com, this feature is available. You can limit the access scope of a project's CI/CD job token to increase the job token's security. A job token might give extra permissions that aren't necessary -to access specific private resources. Limiting the job token access scope reduces the risk of a leaked -token being used to access private data that the user associated to the job can access. +to access specific private resources. +If a job token is leaked it could potentially be used to access data that is private +to the job token's user. By limiting the job token access scope, private data cannot +be accessed unless projects are explicitly authorized. Control the job token access scope with an allowlist of other projects authorized to be accessed by authenticating with the current project's job token. By default @@ -87,10 +86,10 @@ setting at all times, and configure the allowlist for cross-project access if ne For example, when the setting is enabled, jobs in a pipeline in project `A` have a `CI_JOB_TOKEN` scope limited to project `A`. If the job needs to use the token to make an API request to a private project `B`, then `B` must be added to the allowlist for `A`. -If project `B` is public or internal, it doesn't need to be added to the allowlist. +If project `B` is public or internal, it's not required to be added to the allowlist. The job token scope is only for controlling access to private projects. -To enable and configure the job token scope limit: +### Configure the job token scope limit 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. @@ -99,31 +98,9 @@ To enable and configure the job token scope limit: 1. (Optional) Add existing projects to the token's access scope. The user adding a project must have the [maintainer role](../../user/permissions.md) in both projects. -If the job token scope limit is disabled, the token can potentially be used to authenticate -API requests to all projects accessible to the user that triggered the job. - There is [a proposal](https://gitlab.com/groups/gitlab-org/-/epics/3559) to improve the feature with more strategic control of the access permissions. -### Enable or disable CI job token scope limit **(FREE SELF)** - -The GitLab CI/CD job token access scope limit is under development and not ready for production -use. It is deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:ci_scoped_job_token) -``` - -To disable it: - -```ruby -Feature.disable(:ci_scoped_job_token) -``` - ## Trigger a multi-project pipeline by using a CI job token > `CI_JOB_TOKEN` for multi-project pipelines was [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) from GitLab Premium to GitLab Free in 12.4. @@ -163,3 +140,50 @@ build_submodule: ``` Read more about the [jobs artifacts API](../../api/job_artifacts.md#download-the-artifacts-archive). + +## Troubleshooting + +CI job token failures are usually shown as responses like `404 Not Found` or similar: + +- Unauthorized Git clone: + + ```plaintext + $ git clone https://gitlab-ci-token:$CI_JOB_TOKEN@gitlab.com/fabiopitino/test2.git + + Cloning into 'test2'... + remote: The project you were looking for could not be found or you don't have permission to view it. + fatal: repository 'https://gitlab-ci-token:[MASKED]@gitlab.com/<namespace>/<project>.git/' not found + ``` + +- Unauthorized package download: + + ```plaintext + $ wget --header="JOB-TOKEN: $CI_JOB_TOKEN" ${CI_API_V4_URL}/projects/1234/packages/generic/my_package/0.0.1/file.txt + + --2021-09-23 11:00:13-- https://gitlab.com/api/v4/projects/1234/packages/generic/my_package/0.0.1/file.txt + Resolving gitlab.com (gitlab.com)... 172.65.251.78, 2606:4700:90:0:f22e:fbec:5bed:a9b9 + Connecting to gitlab.com (gitlab.com)|172.65.251.78|:443... connected. + HTTP request sent, awaiting response... 404 Not Found + 2021-09-23 11:00:13 ERROR 404: Not Found. + ``` + +- Unauthorized API request: + + ```plaintext + $ curl --verbose --request POST --form "token=$CI_JOB_TOKEN" --form ref=master "https://gitlab.com/api/v4/projects/1234/trigger/pipeline" + + < HTTP/2 404 + < date: Thu, 23 Sep 2021 11:00:12 GMT + {"message":"404 Not Found"} + < content-type: application/json + ``` + +While troubleshooting CI/CD job token authentication issues, be aware that: + +- When the [CI/CD job token limit](#limit-gitlab-cicd-job-token-access) is enabled, + and the job token is being used to access a different project: + - The user that executes the job must be a member of the project that is being accessed. + - The user must have the [permissions](../../user/permissions.md) to perform the action. + - The target project must be [allowlisted for the job token scope limit](#configure-the-job-token-scope-limit). +- The CI job token becomes invalid if the job is no longer running, has been erased, + or if the project is in the process of being deleted. diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 54704d5574b..cb3b11ebf99 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -45,8 +45,6 @@ Clicking an individual job shows you its job log, and allows you to: ## See why a job failed -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17782) in GitLab 10.7. - When a pipeline fails or is allowed to fail, there are several places where you can find the reason: @@ -58,8 +56,7 @@ In each place, if you hover over the failed job you can see the reason it failed  -In [GitLab 10.8 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17814), -you can also see the reason it failed on the Job detail page. +You can also see the reason it failed on the Job detail page. ## The order of jobs in a pipeline @@ -87,8 +84,6 @@ For example: ## Group jobs in a pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6242) in GitLab 8.12. - If you have many similar jobs, your [pipeline graph](../pipelines/index.md#visualize-pipelines) becomes long and hard to read. @@ -164,8 +159,6 @@ for a single run of the manual job. ## Delay a job -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. - When you do not want to run a job immediately, you can use the [`when:delayed`](../jobs/job_control.md#run-a-job-after-a-delay) keyword to delay a job's execution for a certain period. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index e01bd6b79ff..45152e5a0df 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -4,7 +4,7 @@ group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Validate `.gitlab-ci.yml` syntax with the CI Lint tool +# Validate `.gitlab-ci.yml` syntax with the CI Lint tool **(FREE)** If you want to test the validity of your GitLab CI/CD configuration before committing the changes, you can use the CI Lint tool. This tool checks for syntax and logical diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md deleted file mode 100644 index 13f4ca428c6..00000000000 --- a/doc/ci/merge_request_pipelines/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../pipelines/merge_request_pipelines.md' -remove_date: '2021-09-29' ---- - -This document was moved to [another location](../pipelines/merge_request_pipelines.md). - -<!-- This redirect file can be deleted after 2021-09-29. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md deleted file mode 100644 index 5b68c4ca931..00000000000 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../../pipelines/pipelines_for_merged_results.md' -remove_date: '2021-09-29' ---- - -This document was moved to [another location](../../pipelines/pipelines_for_merged_results.md). - -<!-- This redirect file can be deleted after 2021-09-29. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md deleted file mode 100644 index c8e8dffbdcd..00000000000 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../../../pipelines/merge_trains.md' -remove_date: '2021-09-29' ---- - -This document was moved to [another location](../../../pipelines/merge_trains.md). - -<!-- This redirect file can be deleted after 2021-09-29. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 9a220121f54..5343af16489 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -7,7 +7,7 @@ type: reference # Metrics Reports **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. Requires GitLab Runner 11.10 and above. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in GitLab 11.10. Requires GitLab Runner 11.10 and above. GitLab provides a lot of great reporting tools for things like [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), and performance tests. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index 925dff8751c..c2c06375d7b 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -42,8 +42,8 @@ can be a great resource. ## Manage organizational transition An important part of transitioning from Jenkins to GitLab is the cultural and organizational -changes that comes with the move, and successfully managing them. There are a few -things we have found that helps this: +changes that come with the move, and successfully managing them. There are a few +things we have found that help this: - Setting and communicating a clear vision of what your migration goals are helps your users understand why the effort is worth it. The value is clear when diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md deleted file mode 100644 index 93e04bf8a0e..00000000000 --- a/doc/ci/multi_project_pipelines.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'pipelines/multi_project_pipelines.md' -remove_date: '2021-09-29' ---- - -This document was moved to [another location](pipelines/multi_project_pipelines.md). - -<!-- This redirect file can be deleted after 2021-09-29. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md deleted file mode 100644 index f2edc263397..00000000000 --- a/doc/ci/parent_child_pipelines.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'pipelines/parent_child_pipelines.md' -remove_date: '2021-09-29' ---- - -This document was moved to [another location](pipelines/parent_child_pipelines.md). - -<!-- This redirect file can be deleted after 2021-09-29. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/pipelines/img/multi_project_pipeline_graph.png b/doc/ci/pipelines/img/multi_project_pipeline_graph.png Binary files differdeleted file mode 100644 index 723a455cb4a..00000000000 --- a/doc/ci/pipelines/img/multi_project_pipeline_graph.png +++ /dev/null diff --git a/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png b/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png Binary files differnew file mode 100644 index 00000000000..aadf8bb0979 --- /dev/null +++ b/doc/ci/pipelines/img/multi_project_pipeline_graph_v14_3.png diff --git a/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png Binary files differdeleted file mode 100644 index db18cc201fc..00000000000 --- a/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png +++ /dev/null diff --git a/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v14_3.png b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v14_3.png Binary files differnew file mode 100644 index 00000000000..206e4eeec05 --- /dev/null +++ b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v14_3.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 1eacc799636..69e974406e2 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -38,7 +38,7 @@ A typical pipeline might consist of four stages, executed in the following order - A `production` stage, with a job called `deploy-to-prod`. NOTE: -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository), +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/mirror/pull.md), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -148,7 +148,7 @@ The pipeline now executes the jobs as configured. #### Prefill variables in manual pipelines -> [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. You can use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) keywords to define @@ -339,7 +339,7 @@ GitLab capitalizes the stages' names in the pipeline graphs. ### View full pipeline graph -> - [Visualization improvements introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276949) in GitLab 13.11. +> - Visualization improvements [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276949) in GitLab 13.11. The [pipeline details page](#view-pipelines) displays the full pipeline graph of all the jobs in the pipeline. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index c6b6f61ef11..7ecee5508ef 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -8,7 +8,7 @@ type: reference, howto # Job artifacts **(FREE)** -> Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675) in GitLab 12.4, artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. Jobs can output an archive of files and directories. This output is known as a job artifact. @@ -111,7 +111,7 @@ the artifact. ## How searching for job artifacts works -In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201784) and later, artifacts +In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201784), artifacts for [parent and child pipelines](parent_child_pipelines.md) are searched in hierarchical order from parent to child. For example, if both parent and child pipelines have a job with the same name, the job artifact from the parent pipeline is returned. diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index b3dfe8753c7..5b40744aa79 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -198,7 +198,7 @@ which helps you get your starting configuration correct. If you are seeing two pipelines when using `only/except`, please see the caveats related to using `only/except` above (or, consider moving to `rules`). -In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, +In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201845), you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines). After a merge request is open on the branch, the pipeline switches to a merge request pipeline. diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index 06c1a6fef44..6074909a887 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -8,8 +8,8 @@ last_update: 2019-07-03 # Merge trains **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. -> - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in GitLab 12.0. +> - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in GitLab 12.6. For more information about why you might want to use merge trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). @@ -59,8 +59,6 @@ to run. If more merge requests are added to the train, they now include the `A` changes that are included in the target branch, and the `C` changes that are from the merge request already in the train. -Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch this video for a demonstration on [how parallel execution of merge trains can prevent commits from breaking the default diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index d31ddcf736e..184961f4c95 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -88,7 +88,7 @@ The keywords available for use in trigger jobs are: - [`only` and `except`](../yaml/index.md#only--except) - [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`) - [`extends`](../yaml/index.md#extends) -- [`needs`](../yaml/index.md#needs) +- [`needs`](../yaml/index.md#needs), but not [cross project artifact downloads with `needs`](../yaml/index.md#cross-project-artifact-downloads-with-needs) #### Specify a downstream pipeline branch @@ -112,7 +112,7 @@ Use: - The `project` keyword to specify the full path to a downstream project. - The `branch` keyword to specify the name of a branch in the project specified by `project`. - [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126) and later, variable expansion is + In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/10126), variable expansion is supported. Pipelines triggered on a protected branch in a downstream project use the [role](../../user/permissions.md) @@ -290,7 +290,7 @@ When using: ## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab Premium 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.8. You can trigger a pipeline in your project whenever a pipeline finishes for a new tag in a different project. @@ -321,7 +321,7 @@ downstream projects. On self-managed instances, an administrator can change this When you configure GitLab CI/CD for your project, you can visualize the stages of your [jobs](index.md#configure-a-pipeline) on a [pipeline graph](index.md#visualize-pipelines). - + In the merge request, on the **Pipelines** tab, multi-project pipeline mini-graphs are displayed. They expand and are shown adjacent to each other when hovering (or tapping on touchscreen devices). diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md index 71f778d81b3..e48728a904a 100644 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -23,7 +23,7 @@ Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The to choose to start sub-pipelines (or not) is a powerful ability, especially if the YAML is dynamically generated. - + Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a set of concurrently running child pipelines, but within the same project: @@ -72,7 +72,7 @@ microservice_a: - template: Security/SAST.gitlab-ci.yml ``` -In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) and later, +In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/205157), you can use [`include:file`](../yaml/index.md#includefile) to trigger child pipelines with a configuration file in a different project: @@ -169,7 +169,7 @@ runner for testing, the path separator for the trigger job would be `/`. Other C configuration for jobs, like scripts, that use the Windows runner would use `\`. In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. -This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). +This is [resolved](https://gitlab.com/gitlab-org/gitlab/-/issues/209070) in GitLab 12.10. ## Nested child pipelines diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 5c3cdd0633e..94cfeff3acc 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -7,7 +7,7 @@ type: reference # Pipeline efficiency **(FREE)** -[CI/CD Pipelines](index.md) are the fundamental building blocks for [GitLab CI/CD](../README.md). +[CI/CD Pipelines](index.md) are the fundamental building blocks for [GitLab CI/CD](../index.md). Making pipelines more efficient helps you save developer time, which: - Speeds up your DevOps processes diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md index 08d7d119787..2acef9be557 100644 --- a/doc/ci/pipelines/pipelines_for_merged_results.md +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -8,7 +8,7 @@ last_update: 2019-07-03 # Pipelines for merged results **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in GitLab 11.10. When you submit a merge request, you are requesting to merge changes from a source branch into a target branch. By default, the CI pipeline runs jobs @@ -49,7 +49,7 @@ To enable pipelines for merge results: - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - You must not be using [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. - To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). + To follow progress, see [#26996](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). - Your repository must be a GitLab repository, not an [external repository](../ci_cd_for_external_repos/index.md). @@ -82,7 +82,7 @@ For more information, read the [documentation on Merge Trains](merge_trains.md). ## Automatic pipeline cancellation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in GitLab 12.3. GitLab CI/CD can detect the presence of redundant pipelines, and cancels them to conserve CI resources. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 9cb600ae551..90494a715ea 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -73,7 +73,7 @@ job:on-schedule: job: rules: - - if: $CI_PIPELINE_SOURCE = "push" + - if: $CI_PIPELINE_SOURCE == "push" script: - make build ``` diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 94d7e317104..e14c1aa621f 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -94,7 +94,7 @@ For more information, see [Deployment safety](../environments/deployment_safety. ## Specify a custom CI/CD configuration file -> [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. +> Support for external `.gitlab-ci.yml` locations [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) in GitLab 12.6. GitLab expects to find the CI/CD configuration file (`.gitlab-ci.yml`) in the project's root directory. However, you can specify an alternate filename path, including locations outside the project. @@ -241,7 +241,7 @@ Use this regex for commonly used test tools. ### View code coverage history > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. -> - [Graph introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. +> - Graph [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. To see the evolution of your project code coverage over time, you can view a graph or download a CSV file with this data. @@ -358,6 +358,29 @@ in your `README.md`:  ``` +#### Test coverage report badge colors and limits + +The default colors and limits for the badge are as follows: + +- 95 up to and including 100% - good (`#4c1`) +- 90 up to 95% - acceptable (`#a3c51c`) +- 75 up to 90% - medium (`#dfb317`) +- 0 up to 75% - low (`#e05d44`) +- no coverage - unknown (`#9f9f9f`) + +NOTE: +*Up to* means up to, but not including, the upper bound. + +You can overwrite the limits by using the following additional parameters ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28317) in GitLab 14.4): + +- `min_good` (default 95, can use any value between 3 and 100) +- `min_acceptable` (default 90, can use any value between 2 and min_good-1) +- `min_medium` (default 75, can use any value between 1 and min_acceptable-1) + +If an invalid boundary is set, GitLab automatically adjusts it to be valid. For example, +if `min_good` is set `80`, and `min_acceptable` is set to `85` (too high), GitLab automatically +sets `min_acceptable` to `79` (`min_good` - `1`). + ### Badge styles Pipeline badges can be rendered in different styles by adding the `style=style_name` parameter to the URL. Two styles are available: diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index e2381238318..4006a1c9c3c 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -152,7 +152,7 @@ The pipeline starts when the commit is committed. - Use the [`rules`](../yaml/index.md#rules) keyword to specify when to run or skip jobs. The `only` and `except` legacy keywords are still supported, but can't be used with `rules` in the same job. - - Keep information across jobs and stages persistent in a pipeline with [`cache`](../yaml/index.md#cache)) + - Keep information across jobs and stages persistent in a pipeline with [`cache`](../yaml/index.md#cache) and [`artifacts`](../yaml/index.md#artifacts). These keywords are ways to store dependencies and job output, even when using ephemeral runners for each job. - For the complete `.gitlab-ci.yml` syntax, see [the full `.gitlab-ci.yml` reference topic](../yaml/index.md). diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md new file mode 100644 index 00000000000..7de3643c0d4 --- /dev/null +++ b/doc/ci/resource_groups/index.md @@ -0,0 +1,203 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: Control the job concurrency in GitLab CI/CD +--- + +# Resource Group **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. + +By default, pipelines in GitLab CI/CD run in parallel. The parallelization is an important factor to improve +the feedback loop in merge requests, however, there are some situations that +you may want to limit the concurrency on deployment +jobs to run them one by one. +Resource Group allows you to strategically control +the concurrency of the jobs for optimizing your continuous deployments workflow with safety. + +## Add a resource group + +Provided that you have the following pipeline configuration (`.gitlab-ci.yml` file in your repository): + +```yaml +build: + stage: build + script: echo "Your build script" + +deploy: + stage: deploy + script: echo "Your deployment script" + environment: production +``` + +Every time you push a new commit to a branch, it runs a new pipeline that has +two jobs `build` and `deploy`. But if you push multiple commits in a short interval, multiple +pipelines start running simultaneously, for example: + +- The first pipeline runs the jobs `build` -> `deploy` +- The second pipeline runs the jobs `build` -> `deploy` + +In this case, the `deploy` jobs across different pipelines could run concurrently +to the `production` environment. Running multiple deployment scripts to the same +infrastructure could harm/confuse the instance and leave it in a corrupted state in the worst case. + +In order to ensure that a `deploy` job runs once at a time, you can specify +[`resource_group` keyword](../yaml/index.md#resource_group) to the concurrency sensitive job: + +```yaml +deploy: + ... + resource_group: production +``` + +With this configuration, the safety on the deployments is assured while you +can still run `build` jobs concurrently for maximizing the pipeline efficency. + +## Requirements + +- The basic knowledge of the [GitLab CI/CD pipelines](../pipelines/index.md) +- The basic knowledge of the [GitLab Environments and Deployments](../environments/index.md) +- [Developer role](../../user/permissions.md) (or above) in the project to configure CI/CD pipelines. + +### Limitations + +Only one resource can be attached to a resource group. + +## Process modes + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202186) in GitLab 14.3. +> - [Feature flag `ci_resource_group_process_modes`](https://gitlab.com/gitlab-org/gitlab/-/issues/340380) removed in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/202186) in GitLab 14.4. + +You can choose a process mode to strategically control the job concurrency for your deployment preferences. +The following modes are supported: + +- **Unordered:** This is the default process mode that limits the concurrency on running jobs. + It's the easiest option to use and useful when you don't care about the execution order + of the jobs. It starts processing the jobs whenever a job ready to run. +- **Oldest first:** This process mode limits the concurrency of the jobs. When a resource is free, + it picks the first job from the list of upcoming jobs (`created`, `scheduled`, or `waiting_for_resource` state) + that are sorted by pipeline ID in ascending order. + + This mode is useful when you want to ensure that the jobs are executed from the oldest pipeline. + This is less efficient compared to the `unordered` mode in terms of the pipeline efficiency, + but safer for continuous deployments. + +- **Newest first:** This process mode limits the concurrency of the jobs. When a resource is free, + it picks the first job from the list of upcoming jobs (`created`, `scheduled` or `waiting_for_resource` state) + that are sorted by pipeline ID in descending order. + + This mode is useful when you want to ensure that the jobs are executed from the newest pipeline and + cancel all of the old deploy jobs with the [skip outdated deployment jobs](../environments/deployment_safety.md#skip-outdated-deployment-jobs) feature. + This is the most efficient option in terms of the pipeline efficiency, but you must ensure that each deployment job is idempotent. + +### Change the process mode + +To change the process mode of a resource group, you need to use the API and +send a request to [edit an existing resource group](../../api/resource_groups.md#edit-an-existing-resource-group) +by specifying the `process_mode`: + +- `unordered` +- `oldest_first` +- `newest_first` + +### An example of difference between the process modes + +Consider the following `.gitlab-ci.yml`, where we have two jobs `build` and `deploy` +each running in their own stage, and the `deploy` job has a resource group set to +`production`: + +```yaml +build: + stage: build + script: echo "Your build script" + +deploy: + stage: deploy + script: echo "Your deployment script" + environment: production + resource_group: production +``` + +If three commits are pushed to the project in a short interval, that means that three +pipelines run almost at the same time: + +- The first pipeline runs the jobs `build` -> `deploy`. Let's call this deployment job `deploy-1`. +- The second pipeline runs the jobs `build` -> `deploy`. Let's call this deployment job `deploy-2`. +- The third pipeline runs the jobs `build` -> `deploy`. Let's call this deployment job `deploy-3`. + +Depending on the process mode of the resource group: + +- If the process mode is set to `unordered`: + - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - There is no guarantee on the job execution order, for example, `deploy-1` could run before or after `deploy-3` runs. +- If the process mode is `oldest_first`: + - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-1` runs first, `deploy-2` runs second, and `deploy-3` runs last. +- If the process mode is `newest_first`: + - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. + - `deploy-3` runs first, `deploy-2` runs second and `deploy-1` runs last. + +## Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines + +See the how to [control the pipeline concurrency in cross-project pipelines](../yaml/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). + +## API + +See the [API documentation](../../api/resource_groups.md). + +## Related features + +Read more how you can use GitLab for [safe deployments](../environments/deployment_safety.md). + +## Troubleshooting + +### Avoid dead locks in pipeline configurations + +Since [`oldest_first` process mode](#process-modes) enforces the jobs to be executed in a pipeline order, +there is a case that it doesn't work well with the other CI features. + +For example, when you run [a child pipeline](../pipelines/parent_child_pipelines.md) +that requires the same resource group with the parent pipeline, +a dead lock could happen. Here is an example of a _bad_ setup: + +```yaml +# BAD +test: + stage: test + trigger: + include: child-pipeline-requires-production-resource-group.yml + strategy: depend + +deploy: + stage: deploy + script: echo + resource_group: production +``` + +In a parent pipeline, it runs the `test` job that subsequently runs a child pipeline, +and the [`strategy: depend` option](../yaml/index.md#linking-pipelines-with-triggerstrategy) makes the `test` job wait until the child pipeline has finished. +The parent pipeline runs the `deploy` job in the next stage, that requires a resource from the `production` resource group. +If the process mode is `oldest_first`, it executes the jobs from the oldest pipelines, meaning the `deploy` job is going to be executed next. + +However, a child pipeline also requires a resource from the `production` resource group. +Since the child pipeline is newer than the parent pipeline, the child pipeline +waits until the `deploy` job is finished, something that will never happen. + +In this case, you should specify the `resource_group` keyword in the parent pipeline configuration instead: + +```yaml +# GOOD +test: + stage: test + trigger: + include: child-pipeline.yml + strategy: depend + resource_group: production # Specify the resource group in the parent pipeline + +deploy: + stage: deploy + script: echo + resource_group: production +``` diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/runners/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/linux_build_cloud.md b/doc/ci/runners/build_cloud/linux_build_cloud.md index 1125d8dbcb4..6cdc93591a7 100644 --- a/doc/ci/runners/build_cloud/linux_build_cloud.md +++ b/doc/ci/runners/build_cloud/linux_build_cloud.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Build Cloud runners for Linux +# Build Cloud runners for Linux **(FREE)** GitLab Build Cloud runners for Linux run in autoscale mode and are powered by Google Cloud Platform. @@ -15,13 +15,13 @@ GitLab offers Ultimate tier capabilities and included CI/CD minutes per group pe All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default region of the VMs is US East1. -Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. +Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), **time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. +project. Check the issues [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. Below are the runners' settings. diff --git a/doc/ci/runners/build_cloud/macos/environment.md b/doc/ci/runners/build_cloud/macos/environment.md index 5336890b931..8a2417186ae 100644 --- a/doc/ci/runners/build_cloud/macos/environment.md +++ b/doc/ci/runners/build_cloud/macos/environment.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# VM instances and images for Build Cloud for macOS +# VM instances and images for Build Cloud for macOS **(FREE)** When you use the Build Cloud for macOS: diff --git a/doc/ci/runners/build_cloud/macos_build_cloud.md b/doc/ci/runners/build_cloud/macos_build_cloud.md index 794bc2e597d..5d55462fb63 100644 --- a/doc/ci/runners/build_cloud/macos_build_cloud.md +++ b/doc/ci/runners/build_cloud/macos_build_cloud.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Build Cloud runners for macOS (Beta) +# Build Cloud runners for macOS (Beta) **(FREE SAAS)** The GitLab Build Cloud for macOS Beta provides on-demand runners integrated with GitLab SaaS [CI/CD](../../../ci/index.md). Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage diff --git a/doc/ci/runners/build_cloud/windows_build_cloud.md b/doc/ci/runners/build_cloud/windows_build_cloud.md index 0004041a3e8..e01de723055 100644 --- a/doc/ci/runners/build_cloud/windows_build_cloud.md +++ b/doc/ci/runners/build_cloud/windows_build_cloud.md @@ -4,7 +4,7 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Build Cloud runners for Windows (beta) +# Build Cloud runners for Windows (beta) **(FREE)** GitLab Build Cloud runners for Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) and shouldn't be used for production workloads. @@ -19,7 +19,7 @@ the Google Cloud Platform. This solution uses an developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). Windows runners execute your CI/CD jobs on `n1-standard-2` instances with 2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). We want to keep iterating to get Windows runners in a stable state and [generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). diff --git a/doc/ci/secrets/index.md b/doc/ci/secrets/index.md index 898d310598f..4d42bc69df8 100644 --- a/doc/ci/secrets/index.md +++ b/doc/ci/secrets/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index 1e761643a10..817263374f1 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -10,7 +10,7 @@ type: tutorial GitLab currently doesn't have built-in support for managing SSH keys in a build environment (where the GitLab Runner runs). -The SSH keys can be useful when: +Use SSH keys when: 1. You want to checkout internal submodules 1. You want to download private packages using your package manager (for example, Bundler) @@ -45,9 +45,9 @@ check the [visibility of your pipelines](../pipelines/settings.md#change-which-u When your CI/CD jobs run inside Docker containers (meaning the environment is contained) and you want to deploy your code in a private server, you need a way -to access it. This is where an SSH key pair comes in handy. +to access it. In this case, you can use an SSH key pair. -1. You first need to create an SSH key pair. For more information, follow +1. You first must create an SSH key pair. For more information, follow the instructions to [generate an SSH key](../../ssh/index.md#generate-an-ssh-key-pair). **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. @@ -101,7 +101,7 @@ to access it. This is where an SSH key pair comes in handy. 1. As a final step, add the _public_ key from the one you created in the first step to the services that you want to have an access to from within the build - environment. If you are accessing a private GitLab repository you need to add + environment. If you are accessing a private GitLab repository you must add it as a [deploy key](../../user/project/deploy_keys/index.md). That's it! You can now have access to private servers or repositories in your @@ -130,7 +130,7 @@ on, and use that key for all projects that are run on this machine. 1. As a final step, add the _public_ key from the one you created earlier to the services that you want to have an access to from within the build environment. - If you are accessing a private GitLab repository you need to add it as a + If you are accessing a private GitLab repository you must add it as a [deploy key](../../user/project/deploy_keys/index.md). After generating the key, try to sign in to the remote server to accept the @@ -163,8 +163,8 @@ ssh-keyscan 1.2.3.4 Create a new [CI/CD variable](../variables/index.md) with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. -If you need to connect to multiple servers, all the server host keys -need to be collected in the **Value** of the variable, one key per line. +If you must connect to multiple servers, all the server host keys +must be collected in the **Value** of the variable, one key per line. NOTE: By using a variable instead of `ssh-keyscan` directly inside @@ -175,7 +175,7 @@ so there's something wrong with the server or the network. Now that the `SSH_KNOWN_HOSTS` variable is created, in addition to the [content of `.gitlab-ci.yml`](#ssh-keys-when-using-the-docker-executor) -above, here's what more you need to add: +above, you must add: ```yaml before_script: @@ -209,5 +209,5 @@ We have set up an [Example SSH Project](https://gitlab.com/gitlab-examples/ssh-p that runs on [GitLab.com](https://gitlab.com) using our publicly available [shared runners](../runners/index.md). -Want to hack on it? Simply fork it, commit and push your changes. Within a few +Want to hack on it? Fork it, commit, and push your changes. In a few moments the changes is picked by a public runner and the job starts. diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/triggers/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 827a89fa99c..994e9294ff6 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -50,7 +50,7 @@ and check if their values are what you expect. ## GitLab CI/CD documentation The [complete `.gitlab-ci.yml` reference](yaml/index.md) contains a full list of -every keyword you may need to use to configure your pipelines. +every keyword you can use to configure your pipelines. You can also look at a large number of pipeline configuration [examples](examples/index.md) and [templates](examples/index.md#cicd-templates). @@ -76,7 +76,7 @@ if you are using that type: ### Troubleshooting Guides for CI/CD features -There are troubleshooting guides available for some CI/CD features and related topics: +Troubleshooting guides are available for some CI/CD features and related topics: - [Container Registry](../user/packages/container_registry/index.md#troubleshooting-the-gitlab-container-registry) - [GitLab Runner](https://docs.gitlab.com/runner/faq/) @@ -118,7 +118,7 @@ Two pipelines can run when pushing a commit to a branch that has an open merge r associated with it. Usually one pipeline is a merge request pipeline, and the other is a branch pipeline. -This is usually caused by the `rules` configuration, and there are several ways to +This situation is usually caused by the `rules` configuration, and there are several ways to [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines). #### A job is not in the pipeline @@ -168,7 +168,7 @@ a branch to its remote repository. To illustrate the problem, suppose you've had 1. A new pipeline starts running on the `example` branch again, however, the previous pipeline (2) fails because of `fatal: reference is not a tree:` error. -This is because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record) +This occurs because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record) from the `example` branch that the commit history has already been overwritten by the force-push. Similarly, [Pipelines for merged results](pipelines/pipelines_for_merged_results.md) might have failed intermittently due to [the same reason](pipelines/pipelines_for_merged_results.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error). @@ -199,6 +199,7 @@ latest commit yet. This might be because: - You are not using CI/CD pipelines in your project. - You are using CI/CD pipelines in your project, but your configuration prevented a pipeline from running on the source branch for your merge request. - The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)). +- The source branch of the merge request is on a private fork. After the pipeline is created, the message updates with the pipeline status. @@ -262,7 +263,7 @@ To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), [`workflow: rules`](yaml/index.md#workflow) or rewrite your rules to control which pipelines can run. -### Console workaround if job using resource_group gets stuck +### Console workaround if job using resource_group gets stuck **(FREE SELF)** ```ruby # find resource group by name diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 7677908e93a..c37d7f27970 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Unit test reports +# Unit test reports **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39737) from JUnit test reports to Unit test reports in GitLab 13.4. @@ -67,9 +67,9 @@ execution time and the error output. ### Number of recent failures -> - [Introduced in Merge Requests](https://gitlab.com/gitlab-org/gitlab/-/issues/241759) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241759) in Merge Requests in GitLab 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268249) in GitLab 13.8. -> - [Introduced in Test Reports](https://gitlab.com/gitlab-org/gitlab/-/issues/235525) in GitLab 13.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235525) in Test Reports in GitLab 13.9. If a test failed in the project's default branch in the last 14 days, a message like `Failed {n} time(s) in {default_branch} in the last 14 days` is displayed for that test. @@ -328,7 +328,7 @@ phpunit: ## Viewing Unit test reports on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. -> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. If JUnit report format XML files are generated and uploaded as part of a pipeline, these reports can be viewed inside the pipelines details page. The **Tests** tab on this page @@ -357,7 +357,7 @@ GitLab does not parse very [large nodes](https://nokogiri.org/tutorials/parsing_ ## Viewing JUnit screenshots on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0 behind the `:junit_pipeline_screenshots_view` feature flag, disabled by default. -> - The feature flag was removed and was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. Upload your screenshots as [artifacts](yaml/index.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/variables/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index fa4bc6e39fb..8650adf0351 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -191,7 +191,7 @@ The output is: ### Add a CI/CD variable to a group -> Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 +> Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. @@ -244,7 +244,7 @@ To add an instance variable: 1. Select the **Add variable** button, and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), + - **Value**: In [GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), 10,000 characters is allowed. This is also bounded by the limits of the selected runner operating system. In GitLab 13.0 to 13.2, 700 characters is allowed. - **Type**: [`File` or `Variable`](#cicd-variable-types). @@ -301,7 +301,7 @@ An alternative to `File` type variables is to: ```shell # Read certificate stored in $KUBE_CA_PEM variable and save it in a new file -echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" +cat "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" # Pass the newly created file to kubectl kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" ``` @@ -346,9 +346,9 @@ The value of the variable must: - Be a single line. - Be 8 characters or longer, consisting only of: - Characters from the Base64 alphabet (RFC4648). - - The `@` and `:` characters ([In GitLab 12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043) and later). - - The `.` character ([In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022) and later). - - The `~` character ([In GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61517) and later). + - The `@` and `:` characters (In [GitLab 12.2 and later](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/63043)). + - The `.` character (In [GitLab 12.10 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29022)). + - The `~` character (In [GitLab 13.12 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61517)). - Not match the name of an existing predefined or custom CI/CD variable. NOTE: diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index c8688d2433e..45fa1994342 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -20,6 +20,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu |------------------------------------------|--------|--------|-------------| | `CHAT_CHANNEL` | 10.6 | all | The Source chat channel that triggered the [ChatOps](../chatops/index.md) command. | | `CHAT_INPUT` | 10.6 | all | The additional arguments passed with the [ChatOps](../chatops/index.md) command. | +| `CHAT_USER_ID` | 14.4 | all | The chat service's user ID of the user who triggered the [ChatOps](../chatops/index.md) command. | | `CI` | all | 0.4 | Available for all jobs executed in CI/CD. `true` when available. | | `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL. | | `CI_BUILDS_DIR` | all | 11.10 | The top-level directory where builds are executed. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 0f9339657fe..9fd1e3eb1a4 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Where variables can be used +# Where variables can be used **(FREE)** As it's described in the [CI/CD variables](index.md) docs, you can define many different variables. Some of them can be used for all GitLab CI/CD @@ -65,9 +65,10 @@ because the expansion is done in GitLab before any runner gets the job. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48627) in GitLab 13.10. [Deployed behind the `variable_inside_variable` feature flag](../../user/feature_flags.md), disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.3. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.4. FLAG: -On self-managed GitLab, by default this feature is disabled. To enable the feature per project or for your entire instance, ask an administrator to [enable the `variable_inside_variable` flag](../../administration/feature_flags.md). +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `variable_inside_variable`. On GitLab.com, this feature is available. GitLab expands job variable values recursively before sending them to the runner. For example: diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ci/yaml/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index fb5748788f7..aa44400fffc 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -10,7 +10,7 @@ type: reference This document lists the configuration options for your GitLab `.gitlab-ci.yml` file. - For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). -- For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). +- For a collection of examples, see [GitLab CI/CD Examples](../examples/index.md). - To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). When you are editing your `.gitlab-ci.yml` file, you can validate it with the @@ -446,15 +446,15 @@ that proposes expanding this feature to support more variables. #### `rules` with `include` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276515) in GitLab 14.2. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3 and is ready for production use. -> - [Enabled with `ci_include_rules` flag](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) for self-managed GitLab in GitLab 14.3 and is ready for production use. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the `ci_include_rules` flag](../../administration/feature_flags.md). On GitLab.com, this feature is available. +> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) GitLab 14.3. +> - [Feature flag `ci_include_rules` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. You can use [`rules`](#rules) with `include` to conditionally include other configuration files. -You can only use `rules:if` in `include` with [certain variables](#variables-with-include). +You can only use [`if` rules](#rulesif) in `include`, and only with [certain variables](#variables-with-include). +`rules` keywords such as `changes` and `exists` are not supported. ```yaml include: @@ -746,9 +746,9 @@ Use `before_script` to define an array of commands that should run before each j ```yaml job: before_script: - - echo "Execute this command before any `script:` commands." + - echo "Execute this command before any 'script:' commands." script: - - echo "This command executes after the job's `before_script` commands." + - echo "This command executes after the job's 'before_script' commands." ``` **Additional details**: @@ -794,7 +794,7 @@ job: Scripts you specify in `after_script` execute in a new shell, separate from any `before_script` or `script` commands. As a result, they: -- Have a current working directory set back to the default. +- Have the current working directory set back to the default (according to the [variables which define how the runner processes Git requests](#configure-runner-behavior-with-variables)). - Don't have access to changes done by commands defined in the `before_script` or `script`, including: - Command aliases and variables exported in `script` scripts. @@ -1178,6 +1178,7 @@ job: all rules. You can't mix `when` at the job-level with `when` in rules. - Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) sections, variables in rules expressions are always formatted as `$VARIABLE`. + - You can use `rules:if` with `include` to [conditionally include other configuration files](#rules-with-include). **Related topics**: @@ -1701,6 +1702,8 @@ same group or namespace, you can omit them from the `project:` keyword. For exam The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility. +You cannot use cross project artifact downloads in the same job as [`trigger`](#trigger). + ##### Artifact downloads between pipelines in the same project Use `needs` to download artifacts from different pipelines in the current project. @@ -2220,7 +2223,7 @@ For more information, see > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. Use the `kubernetes` keyword to configure deployments to a -[Kubernetes cluster](../../user/project/clusters/index.md) that is associated with your project. +[Kubernetes cluster](../../user/infrastructure/clusters/index.md) that is associated with your project. For example: @@ -2243,7 +2246,7 @@ For more information, see NOTE: Kubernetes configuration is not supported for Kubernetes clusters -that are [managed by GitLab](../../user/project/clusters/index.md#gitlab-managed-clusters). +that are [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). To follow progress on support for GitLab-managed clusters, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). @@ -3258,15 +3261,16 @@ job: ### `coverage` -Use `coverage` to configure how code coverage is extracted from the -job output. +Use `coverage` with a custom regular expression to configure how code coverage +is extracted from the job output. The coverage is shown in the UI if at least one +line in the job output matches the regular expression. -Regular expressions are the only valid kind of value expected here. So, using -surrounding `/` is mandatory to consistently and explicitly represent -a regular expression string. You must escape special characters if you want to -match them literally. +To extract the code coverage value in the matching line, GitLab uses this +regular expression: `\d+(\.\d+)?`. -For example: +**Possible inputs**: A regular expression. Must start and end with `/`. + +**Example of `coverage`**: ```yaml job1: @@ -3274,14 +3278,20 @@ job1: coverage: '/Code coverage: \d+\.\d+/' ``` -The coverage is shown in the UI if at least one line in the job output matches the regular expression. -If there is more than one matched line in the job output, the last line is used. -For the matched line, the first occurrence of `\d+(\.\d+)?` is the code coverage. -Leading zeros are removed. +In this example: + +1. GitLab checks the job log for a line that matches the regular expression. A line + like `Code coverage: 67.89` would match. +1. GitLab then checks the line to find a match to `\d+(\.\d+)?`. The sample matching + line above gives a code coverage of `67.89`. + +**Additional details**: -Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) is not recorded -or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) -for more details. +- If there is more than one matched line in the job output, the last line is used. +- Leading zeros are removed. +- Coverage output from [child pipelines](../pipelines/parent_child_pipelines.md) + is not recorded or displayed. Check [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/280818) + for more details. ### `dast_configuration` **(ULTIMATE)** @@ -3331,56 +3341,38 @@ to select a specific site profile and scanner profile. > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5, you can control which failures to retry on. -Use `retry` to configure how many times a job is retried in -case of a failure. - -When a job fails, the job is processed again, -until the limit specified by the `retry` keyword is reached. +Use `retry` to configure how many times a job is retried if it fails. +If not defined, defaults to `0` and jobs do not retry. -If `retry` is set to `2`, and a job succeeds in a second run (first retry), it is not retried. -The `retry` value must be a positive integer, from `0` to `2` -(two retries maximum, three runs in total). +When a job fails, the job is processed up to two more times, until it succeeds or +reaches the maximum number of retries. -The following example retries all failure cases: - -```yaml -test: - script: rspec - retry: 2 -``` +By default, all failure types cause the job to be retried. Use [`retry:when`](#retrywhen) +to select which failures to retry on. -By default, a job is retried on all failure cases. To have better control -over which failures to retry, `retry` can be a hash with the following keys: +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#custom-default-keyword-values). -- `max`: The maximum number of retries. -- `when`: The failure cases to retry. +**Possible inputs**: `0` (default), `1`, or `2`. -To retry only runner system failures at maximum two times: +**Example of `retry`**: ```yaml test: script: rspec - retry: - max: 2 - when: runner_system_failure + retry: 2 ``` -If there is another failure, other than a runner system failure, the job -is not retried. +#### `retry:when` -To retry on multiple failure cases, `when` can also be an array of failures: +Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. +`retry:max` is the maximum number of retries, like [`retry`](#retry), and can be +`0`, `1`, or `2`. -```yaml -test: - script: rspec - retry: - max: 2 - when: - - runner_system_failure - - stuck_or_timeout_failure -``` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#custom-default-keyword-values). -Possible values for `when` are: +**Possible inputs**: A single failure type, or an array of one or more failure types: <!-- If you change any of the values below, make sure to update the `RETRY_WHEN_IN_DOCUMENTATION` @@ -3396,7 +3388,6 @@ Possible values for `when` are: - `api_failure`: Retry on API failure. - `stuck_or_timeout_failure`: Retry when the job got stuck or timed out. - `runner_system_failure`: Retry if there is a runner system failure (for example, job setup failed). -- `missing_dependency_failure`: Retry if a dependency is missing. - `runner_unsupported`: Retry if the runner is unsupported. - `stale_schedule`: Retry if a delayed job could not be executed. - `job_execution_timeout`: Retry if the script exceeded the maximum execution time set for the job. @@ -3405,7 +3396,34 @@ Possible values for `when` are: - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `data_integrity_failure`: Retry if there is a structural integrity problem detected. -You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables. +**Example of `retry:when`** (single failure type): + +```yaml +test: + script: rspec + retry: + max: 2 + when: runner_system_failure +``` + +If there is a failure other than a runner system failure, the job is not retried. + +**Example of `retry:when`** (array of failure types): + +```yaml +test: + script: rspec + retry: + max: 2 + when: + - runner_system_failure + - stuck_or_timeout_failure +``` + +**Related topics**: + +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) +using variables. ### `timeout` @@ -3859,7 +3877,7 @@ can be deployed to, but there can be only one deployment per device at any given The `resource_group` value can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. It can't start or end with `/`. -For more information, see [Deployments Safety](../environments/deployment_safety.md). +For more information, see [Resource Group documentation](../resource_groups/index.md). #### Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines diff --git a/doc/development/README.md b/doc/development/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/development/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 16dd581113c..9ca08ab1dc2 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -275,7 +275,8 @@ You can verify if the MR was deployed to GitLab.com by executing `/chatops run auto_deploy status <merge_sha>`. To verify existence of the index, you can: -- Use a meta-command in #database-lab, such as: `\di <index_name>` +- Use a meta-command in #database-lab, such as: `\d <index_name>` + - Ensure that the index is not [`invalid`](https://www.postgresql.org/docs/12/sql-createindex.html#:~:text=The%20psql%20%5Cd%20command%20will%20report%20such%20an%20index%20as%20INVALID) - Ask someone in #database to check if the index exists - With proper access, you can also verify directly on production or in a production clone diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md new file mode 100644 index 00000000000..c1d7ac9fa0c --- /dev/null +++ b/doc/development/application_slis/index.md @@ -0,0 +1,130 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Application Service Level Indicators (SLIs) + +> [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 + +It is possible to define [Service Level Indicators +(SLIs)](https://en.wikipedia.org/wiki/Service_level_indicator) +directly in the Ruby codebase. This keeps the definition of operations +and their success close to the implementation and allows the people +building features to easily define how these features should be +monitored. + +Defining an SLI causes 2 +[Prometheus +counters](https://prometheus.io/docs/concepts/metric_types/#counter) +to be emitted from the rails application: + +- `gitlab_sli:<sli name>:total`: incremented for each operation. +- `gitlab_sli:<sli_name>:success_total`: incremented for successful + operations. + +## Existing SLIs + +1. [`rails_request_apdex`](rails_request_apdex.md) + +## Defining a new SLI + +An SLI can be defined using the `Gitlab::Metrics::Sli` class. + +Before the first scrape, it is important to have [initialized the SLI +with all possible +label-combinations](https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics). This +avoid confusing results when using these counters in calculations. + +To initialize an SLI, use the `.inilialize_sli` class method, for +example: + +```ruby +Gitlab::Metrics::Sli.initialize_sli(:received_email, [ + { + feature_category: :issue_tracking, + email_type: :create_issue + }, + { + feature_category: :service_desk, + email_type: :service_desk + }, + { + feature_category: :code_review, + email_type: :create_merge_request + } +]) +``` + +Metrics must be initialized before they get +scraped for the first time. This could be done at the start time of the +process that will emit them, in which case we need to pay attention +not to increase application's boot time too much. This is preferable +if possible. + +Alternatively, if initializing would take too long, this can be done +during the first scrape. We need to make sure we don't do it for every +scrape. This can be done as follows: + +```ruby +def initialize_request_slis_if_needed! + return if Gitlab::Metrics::Sli.initialized?(:rails_request_apdex) + Gitlab::Metrics::Sli.initialize_sli(:rails_request_apdex, possible_request_labels) +end +``` + +Also pay attention to do it for the different metrics +endpoints we have. Currently the +[`WebExporter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/metrics/exporter/web_exporter.rb) +and the +[`HealthController`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/health_controller.rb) +for Rails and +[`SidekiqExporter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/metrics/exporter/sidekiq_exporter.rb) +for Sidekiq. + +## Tracking operations for an SLI + +Tracking an operation in the newly defined SLI can be done like this: + +```ruby +Gitlab::Metrics::Sli[:received_email].increment( + labels: { + feature_category: :service_desk, + email_type: :service_desk + }, + success: issue_created? +) +``` + +Calling `#increment` on this SLI will increment the total Prometheus counter + +```prometheus +gitlab_sli:received_email:total{ feature_category='service_desk', email_type='service_desk' } +``` + +If the `success:` argument passed is truthy, then the success counter +will also be incremented: + +```prometheus +gitlab_sli:received_email:success_total{ feature_category='service_desk', email_type='service_desk' } +``` + +## Using the SLI in service monitoring and alerts + +When the application is emitting metrics for the new SLI, those need +to be consumed in the service catalog to result in alerts, and be +included in the error budget for stage groups and GitLab.com's overall +availability. + +This is currently being worked on in [this +project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). As +part of [this +issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1307) +we will update the documentation. + +For any question, please don't hesitate to createan issue in [the +Scalability issue +tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) +or come find us in +[#g_scalability](https://gitlab.slack.com/archives/CMMF8TKR9) on Slack. diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md new file mode 100644 index 00000000000..e1ab5368578 --- /dev/null +++ b/doc/development/application_slis/rails_request_apdex.md @@ -0,0 +1,234 @@ +--- +stage: Platforms +group: Scalability +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Rails request apdex SLI + +> [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 + +NOTE: +This SLI is not yet used in [error budgets for stage +groups](../stage_group_dashboards.md#error-budget) or service +monitoring. This is being worked on in [this +project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). + +The request apdex SLI is [an SLI defined in the application](index.md) +that measures the duration of successful requests as an indicator for +application performance. This includes the REST and GraphQL API, and the +regular controller endpoints. It consists of these counters: + +1. `gitlab_sli:rails_request_apdex:total`: This counter gets + incremented for every request that did not result in a response + with a 5xx status code. This means that slow failures don't get + counted twice: The request is already counted in the error-SLI. + +1. `gitlab_sli:rails_request_apdex:success_total`: This counter gets + incremented for every successful request that performed faster than + the [defined target duration depending on the endpoint's + urgency](#adjusting-request-urgency). + +Both these counters are labeled with: + +1. `endpoint_id`: The identification of the Rails Controller or the + Grape-API endpoint + +1. `feature_category`: The feature category specified for that + controller or API endpoint. + +## Request Apdex SLO + +These counters can be combined into a success ratio, the objective for +this ratio is defined in the service catalog per service: + +1. [Web: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/web.jsonnet#L19) +1. [API: 0.995](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/api.jsonnet#L19) +1. [Git: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/git.jsonnet#L22) + +This means that for this SLI to meet SLO, the ratio recorded needs to +be higher than those defined above. + +For example: for the web-service, we want at least 99.8% of requests +to be faster than their target duration. + +These are the targets we use for alerting and service montoring. So +durations should be set keeping those into account. So we would not +cause alerts. But the goal would be to set the urgency to a target +that users would be satisfied with. + +Both successful measurements and unsuccessful ones have an impact on the +error budget for stage groups. + +## Adjusting request urgency + +Not all endpoints perform the same type of work, so it is possible to +define different urgencies for different endpoints. An endpoint with a +lower urgency can have a longer request duration than endpoints that +are high urgency. + +Long-running requests are more expensive for our +infrastructure: while one request is being served, the thread remains +occupied for the duration of that request. So nothing else can be handled by that +thread. Because of Ruby's Global VM Lock, the thread might keep the +lock and stall other requests handled by the same Puma worker +process. The request is in fact a noisy neighbor for other requests +handled by the worker. This is why the upper bound for a target +duration is capped at 5 seconds. + +## Decreasing the urgency (setting a higher target duration) + +Increasing the urgency on an existing endpoint can be done on +a case-by-case basis. Please take the following into account: + +1. Apdex is about perceived performance, if a user is actively waiting + for the result of a request, waiting 5 seconds might not be + acceptable. While if the endpoint is used by an automation + requiring a lot of data, 5 seconds could be okay. + + A product manager can help to identify how an endpoint is used. + +1. The workload for some endpoints can sometimes differ greatly + depending on the parameters specified by the caller. The urgency + needs to accomodate that. In some cases, it might be interesting to + define a separate [application SLI](index.md#defining-a-new-sli) + for what the endpoint is doing. + + When the endpoints in certain cases turn into no-ops, making them + very fast, we should ignore these fast requests when setting the + target. For example, if the `MergeRequests::DraftsController` is + hit for every merge request being viewed, but doesn't need to + render anything in most cases, then we should pick the target that + would still accomodate the endpoint performing work. + +1. Consider the dependent resources consumed by the endpoint. If the endpoint + loads a lot of data from Gitaly or the database and this is causing + it to not perform satisfactory. It could be better to optimize the + way the data is loaded rather than increasing the target duration + by lowering the urgency. + + In cases like this, it might be appropriate to temporarily decrease + urgency to make the endpoint meet SLO, if this is bearable for the + infrastructure. In such cases, please link an issue from a code + comment. + + If the endpoint consumes a lot of CPU time, we should also consider + this: these kinds of requests are the kind of noisy neighbors we + should try to keep as short as possible. + +1. Traffic characteristics should also be taken into account: if the + trafic to the endpoint is bursty, like CI traffic spinning up a + big batch of jobs hitting the same endpoint, then having these + endpoints take 5s is not acceptable from an infrastructure point of + view. We cannot scale up the fleet fast enough to accomodate for + the incoming slow requests alongside the regular traffic. + +When lowering the urgency for an existing endpoint, please involve a +[Scalability team member](https://about.gitlab.com/handbook/engineering/infrastructure/team/scalability/#team-members) +in the review. We can use request rates and durations available in the +logs to come up with a recommendation. Picking a threshold can be done +using the same process as for [increasing +urgency](#increasing-urgency-setting-a-lower-target-duration), picking +a duration that is higher than the SLO for the service. + +We shouldn't set the longest durations on endpoints in the merge +requests that introduces them, since we don't yet have data to support +the decision. + +## Increasing urgency (setting a lower target duration) + +When decreasing the target duration, we need to make sure the endpoint +still meets SLO for the fleet that handles the request. You can use the +information in the logs to determine this: + +1. Open [this table in + Kibana](https://log.gprd.gitlab.net/goto/bbb6465c68eb83642269e64a467df3df) + +1. The table loads information for the busiest endpoints by + default. You can speed things up by adding a filter for + `json.caller_id.keyword` and adding the identifier you're intersted + in (for example: `Projects::RawController#show`). + +1. Check the [appropriate percentile duration](#request-apdex-slo) for + the service the endpoint is handled by. The overall duration should + be lower than the target you intend to set. + +1. If the overall duration is below the intended targed. Please also + check the peaks over time in [this + graph](https://log.gprd.gitlab.net/goto/9319c4a402461d204d13f3a4924a89fc) + in Kibana. Here, the percentile in question should not peak above + the target duration we want to set. + +Since decreasing a threshold too much could result in alerts for the +apdex degradation, please also involve a Scalability team member in +the merge reqeust. + +## How to adjust the urgency + +The urgency can be specified similar to how endpoints [get a feature +category](../feature_categorization/index.md). + +For endpoints that don't have a specific target, the default urgency (1s duration) will be used. + +The following configurations are available: + +| Urgency | Duration in seconds | Notes | +|----------|---------------------|-----------------------------------------------| +| :high | 0.25s | | +| :medium | 0.5s | | +| :default | 1s | This is the default when nothing is specified | +| :low | 5s | | + +### Rails controller + +An urgency can be specified for all actions in a controller like this: + +```ruby +class Boards::ListsController < ApplicationController + urgency :high +end +``` + +To specify the urgency also for certain actions in a controller, they +can be specified like this: + +```ruby +class Boards::ListsController < ApplicationController + urgency :high, [:index, :show] +end +``` + +### Grape endpoints + +To specify the urgency for an entire API class, this can be done as +follows: + +```ruby +module API + class Issues < ::API::Base + urgency :low + end +end +``` + +To specify the urgency also for certain actions in a API class, they +can be specified like this: + +```ruby +module API + class Issues < ::API::Base + urgency :medium, [ + '/groups/:id/issues', + '/groups/:id/issues_statistics' + ] + end +end +``` + +Or, we can specify the urgency per endpoint: + +```ruby +get 'client/features', urgency: :low do + # endpoint logic +end +``` diff --git a/doc/development/architecture.md b/doc/development/architecture.md index fe2b621da29..9accd4a3595 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -813,7 +813,7 @@ Starting with GitLab 13.0, Puma is the default web server. - [Project page](https://gitlab.com/gitlab-org/gitlab/-/blob/master/README.md) - Configuration: - - [Omnibus](https://docs.gitlab.com/omnibus/settings/puma.html) + - [Omnibus](../administration/operations/puma.md) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - [Source](../install/installation.md#configure-it) - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) diff --git a/doc/development/cascading_settings.md b/doc/development/cascading_settings.md index 0fa0e220ba9..a85fc52d303 100644 --- a/doc/development/cascading_settings.md +++ b/doc/development/cascading_settings.md @@ -135,7 +135,7 @@ Renders the enforcement checkbox. | `attribute` | Name of the setting. For example, `:delayed_project_removal`. | `String` or `Symbol` | `true` | | `group` | Current group. | [`Group`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/group.rb) | `true` | | `form` | [Rails FormBuilder object](https://apidock.com/rails/ActionView/Helpers/FormBuilder). | [`ActionView::Helpers::FormBuilder`](https://apidock.com/rails/ActionView/Helpers/FormBuilder) | `true` | -| `setting_locked` | If the setting is locked by an ancestor group or admin setting. Can be calculated with [`cascading_namespace_setting_locked?`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/helpers/namespaces_helper.rb#L86). | `Boolean` | `true` | +| `setting_locked` | If the setting is locked by an ancestor group or administrator setting. Can be calculated with [`cascading_namespace_setting_locked?`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/helpers/namespaces_helper.rb#L86). | `Boolean` | `true` | | `help_text` | Text shown below the checkbox. | `String` | `false` (Subgroups cannot change this setting.) | [`_setting_label_checkbox.html.haml`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/views/shared/namespaces/cascading_settings/_setting_label_checkbox.html.haml) @@ -147,7 +147,7 @@ Renders the label for a checkbox setting. | `attribute` | Name of the setting. For example, `:delayed_project_removal`. | `String` or `Symbol` | `true` | | `group` | Current group. | [`Group`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/group.rb) | `true` | | `form` | [Rails FormBuilder object](https://apidock.com/rails/ActionView/Helpers/FormBuilder). | [`ActionView::Helpers::FormBuilder`](https://apidock.com/rails/ActionView/Helpers/FormBuilder) | `true` | -| `setting_locked` | If the setting is locked by an ancestor group or admin setting. Can be calculated with [`cascading_namespace_setting_locked?`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/helpers/namespaces_helper.rb#L86). | `Boolean` | `true` | +| `setting_locked` | If the setting is locked by an ancestor group or administrator setting. Can be calculated with [`cascading_namespace_setting_locked?`](https://gitlab.com/gitlab-org/gitlab/-/blob/c2736823b8e922e26fd35df4f0cd77019243c858/app/helpers/namespaces_helper.rb#L86). | `Boolean` | `true` | | `settings_path_helper` | Lambda function that generates a path to the ancestor setting. For example, `settings_path_helper: -> (locked_ancestor) { edit_group_path(locked_ancestor, anchor: 'js-permissions-settings') }` | `Lambda` | `true` | | `help_text` | Text shown below the checkbox. | `String` | `false` (`nil`) | diff --git a/doc/development/changelog.md b/doc/development/changelog.md index be46d61eb4c..2753257c941 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -62,8 +62,8 @@ The value must be the full URL of the merge request. ### GitLab Enterprise changes -If a change is for GitLab Enterprise Edition, you must also add the trailer `EE: -true`: +If a change is exclusively for GitLab Enterprise Edition, **you must add** the +trailer `EE: true`: ```plaintext Update git vendor to gitlab @@ -77,6 +77,8 @@ MR: https://gitlab.com/foo/bar/-/merge_requests/123 EE: true ``` +**Do not** add the trailer for changes that apply to both EE and CE. + ## What warrants a changelog entry? - Any change that introduces a database migration, whether it's regular, post, diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index b4e32066ba8..82fd37eacaf 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -157,7 +157,7 @@ On top of that, we have the following types of jobs: - `Ci::Build` ... The job to be executed by runners. - `Ci::Bridge` ... The job to trigger a downstream pipeline. -- `GenericCommitStatus` ... The job to be executed in an external CI/CD system e.g. Jenkins. +- `GenericCommitStatus` ... The job to be executed in an external CI/CD system, for example Jenkins. When you use the "Job" terminology in codebase, readers would assume that the class/object is any type of above. diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 3fc464e661f..b74a1d0d58a 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -325,8 +325,14 @@ projects on `gitlab.com`: After you're confident the latest template can be moved to stable: 1. Update the stable template with the content of the latest version. +1. Remove the migration template from `Gitlab::Template::GitlabCiYmlTemplate::TEMPLATES_WITH_LATEST_VERSION` const. 1. Remove the corresponding feature flag. +NOTE: +Feature flags are enabled by default in RSpec, so all tests are performed +against the latest templates. You should also test the stable templates +with `stub_feature_flags(redirect_to_latest_template_<name>: false)`. + ### Further reading There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 12cc63ef56d..89516c2168b 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -106,6 +106,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes user-facing changes (*3*), it must be **approved by a [Product Designer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_reviewers_UX)**, based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). + See the [design and user interface guidelines](contributing/design.md) for details. 1. If your merge request includes adding a new JavaScript library (*1*)... - If the library significantly increases the [bundle size](https://gitlab.com/gitlab-org/frontend/playground/webpack-memory-metrics/-/blob/master/doc/report.md), it must @@ -156,7 +157,7 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering 1. I have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or I have decided that this does not apply. 1. I have properly separated EE content from FOSS, or this MR is FOSS only. - [Where should EE code go?](ee_features.md#separation-of-ee-code) -1. If I am introducing a new expectation for existing data, I have confirmed that existing data meets this expectation or I have made this expectation optional rather than required. +1. I have considered that existing data may be surprisingly varied. For example, a new model validation can break existing records. Consider making validation on existing data optional rather than required if you haven't confirmed that existing data will pass validation. ##### Performance, reliability, and availability diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 9e8375fcbdd..e85f5dd8349 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -5,34 +5,119 @@ group: Development info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Implement design & UI elements +# Design and user interface changes -For guidance on UX implementation at GitLab, please refer to our [Design System](https://design.gitlab.com/). +Follow these guidelines when contributing or reviewing design and user interface +(UI) changes. Refer to our [code review guide](../code_review.md) for broader +advice and best practices for code review in general. -The UX team uses labels to manage their workflow. +The basis for most of these guidelines is [Pajamas](https://design.gitlab.com/), +GitLab design system. We encourage you to [contribute to Pajamas](https://design.gitlab.com/get-started/contribute/) +with additions and improvements. -The `~UX` label on an issue is a signal to the UX team that it will need UX attention. -To better understand the priority by which UX tackles issues, see the [UX section](https://about.gitlab.com/handbook/engineering/ux/) of the handbook. +## Merge request reviews -Once an issue has been worked on and is ready for development, a UXer removes the `~UX` label and applies the `~"UX ready"` label to that issue. +As a merge request (MR) author, you must include _Before_ and _After_ +screenshots (or videos) of your changes in the description, as explained in our +[MR workflow](merge_request_workflow.md). These screenshots/videos are very helpful +for all reviewers and can speed up the review process, especially if the changes +are small. -There is a special type label called `~"product discovery"` intended for UX (user experience), -PM (product manager), FE (frontend), and BE (backend). It represents a discovery issue to discuss the problem and -potential solutions. The final output for this issue could be a doc of -requirements, a design artifact, or even a prototype. The solution will be -developed in a subsequent milestone. +## Checklist -`~"product discovery"` issues are like any other issue and should contain a milestone label, `~Deliverable` or `~Stretch`, when scheduled in the current milestone. +Check these aspects both when _designing_ and _reviewing_ UI changes. -The initial issue should be about the problem we are solving. If a separate [product discovery issue](https://about.gitlab.com/handbook/engineering/ux/ux-department-workflow/#how-we-use-labels) -is needed for additional research and design work, it will be created by a PM or UX person. -Assign the `~UX`, `~"product discovery"` and `~Deliverable` labels, add a milestone and -use a title that makes it clear that the scheduled issue is product discovery -(for example, `Product discovery for XYZ`). +### Writing -In order to complete a product discovery issue in a release, you must complete the following: +- Follow [Pajamas](https://design.gitlab.com/content/punctuation/) as the primary + guidelines for UI text and [documentation style guide](../documentation/styleguide/index.md) + as the secondary. +- Use clear and consistent [terminology](https://design.gitlab.com/content/terminology/). +- Check grammar and spelling. +- Consider help content and follow its [guidelines](https://design.gitlab.com/usability/helping-users/). +- Request review from the [appropriate Technical Writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers), + indicating any specific files or lines they should review, and how to preview + or understand the location/context of the text from the user's perspective. -1. UXer removes the `~UX` label, adds the `~"UX ready"` label. -1. Modify the issue description in the product discovery issue to contain the final design. If it makes sense, the original information indicating the need for the design can be moved to a lower "Original Information" section. -1. Copy the design to the description of the delivery issue for which the product discovery issue was created. Do not simply refer to the product discovery issue as a separate source of truth. -1. In some cases, a product discovery issue also identifies future enhancements that will not go into the issue that originated the product discovery issue. For these items, create new issues containing the designs to ensure they are not lost. Put the issues in the backlog if they are agreed upon as good ideas. Otherwise leave them for triage. +### Patterns + +- Consider similar patterns used in the product and justify in the issue when diverging + from them. +- Use appropriate [components](https://design.gitlab.com/components/overview/) + and [data visualizations](https://design.gitlab.com/data-visualization/overview/). + +### Visual design + +Check visual design properties using your browser's _elements inspector_ ([Chrome](https://developer.chrome.com/docs/devtools/css/), +[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Open_the_Inspector)). + +- Use recommended [colors](https://design.gitlab.com/product-foundations/colors/) + and [typography](https://design.gitlab.com/product-foundations/type-fundamentals/). +- Follow [layout guidelines](https://design.gitlab.com/layout/grid/). +- Use existing [icons](http://gitlab-org.gitlab.io/gitlab-svgs/) and [illustrations](http://gitlab-org.gitlab.io/gitlab-svgs/illustrations/) + or propose new ones according to [iconography](https://design.gitlab.com/product-foundations/iconography/) + and [illustration](https://design.gitlab.com/product-foundations/illustration/) + guidelines. +- _Optionally_ consider [dark mode](../../user/profile/preferences.md#dark-mode). [^1] + + [^1]: You're not required to design for [dark mode](../../user/profile/preferences.md#dark-mode) while the feature is in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). The [UX Foundations team](https://about.gitlab.com/direction/ecosystem/foundations/) plans to improve the dark mode in the future. Until we integrate [Pajamas](https://design.gitlab.com/) components into the product and the underlying design strategy is in place to support dark mode, we cannot guarantee that we won't introduce bugs and debt to this mode. At your discretion, evaluate the need to create dark mode patches. + +### States + +Check states using your browser's _styles inspector_ to toggle CSS pseudo-classes +like `:hover` and others ([Chrome](https://developer.chrome.com/docs/devtools/css/reference/#pseudo-class), +[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Page_Inspector/How_to/Examine_and_edit_CSS#viewing_common_pseudo-classes)). + +- Account for all applicable states ([error](https://design.gitlab.com/content/error-messages), + rest, loading, focus, hover, selected, disabled). +- Account for states dependent on data size ([empty](https://design.gitlab.com/regions/empty-states), + some data, and lots of data). +- Account for states dependent on user role, user preferences, and subscription. +- Consider animations and transitions, and follow their [guidelines](https://design.gitlab.com/product-foundations/motion/). + +### Responsive + +Check responsive behavior using your browser's _responsive mode_ ([Chrome](https://developer.chrome.com/docs/devtools/device-mode/#viewport), +[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Responsive_Design_Mode)). + +- Account for resizing, collapsing, moving, or wrapping of elements across + all breakpoints (even if larger viewports are prioritized). +- Provide the same information and actions in all breakpoints. + +### Accessibility + +Check accessibility using your browser's _accessibility inspector_ ([Chrome](https://developer.chrome.com/docs/devtools/accessibility/reference/), +[Firefox](https://developer.mozilla.org/en-US/docs/Tools/Accessibility_inspector#accessing_the_accessibility_inspector)). + +- Conform to level AA of the World Wide Web Consortium (W3C) [Web Content Accessibility Guidelines 2.1](https://www.w3.org/TR/WCAG21/), + according to our [statement of compliance](https://design.gitlab.com/accessibility/a11y/). +- Follow accessibility [best practices](https://design.gitlab.com/accessibility/best-practices/) + and [checklist](../fe_guide/accessibility.md#quick-checklist). + +### Handoff + +When the design is ready, _before_ starting its implementation: + +- Share design specifications in the related issue, preferably through a [Figma link](https://help.figma.com/hc/en-us/articles/360040531773-Share-Files-with-anyone-using-Link-Sharing#Copy_links) + link or [GitLab Designs feature](../../user/project/issues/design_management.md#the-design-management-section). + See [when you should use each tool](https://about.gitlab.com/handbook/engineering/ux/product-designer/#deliver). +- Document user flow and states (for example, using [Mermaid flowcharts in Markdown](../../user/markdown.md#mermaid)). +- Document animations and transitions. +- Document responsive behaviors. +- Document non-evident behaviors (for example, field is auto-focused). +- Document accessibility behaviors (for example, using [accessibility annotations in Figma](https://www.figma.com/file/g7QtDbfxF3pCdWiyskIr0X/Accessibility-bluelines)). +- Contribute new icons or illustrations to the [GitLab SVGs](https://gitlab.com/gitlab-org/gitlab-svgs) + project. + +### Follow-ups + +At any moment, but usually _during_ or _after_ the design's implementation: + +- Contribute [issues to Pajamas](https://design.gitlab.com/get-started/contribute#contribute-an-issue) + for additions or enhancements to the design system. +- Create issues with the [`~UX debt`](issue_workflow.md#technical-and-ux-debt) + label for intentional deviations from the agreed-upon UX requirements due to + time or feasibility challenges, linking back to the corresponding issue(s) or + MR(s). +- Create issues for [feature additions or enhancements](issue_workflow.md#feature-proposals) + outside the agreed-upon UX requirements to avoid scope creep. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 29f6eb57160..d0f107ba98a 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -151,7 +151,7 @@ From the handbook's page: > Categories are high-level capabilities that may be a standalone product at -another company. e.g. Portfolio Management. +another company, such as Portfolio Management, for example. It's highly recommended to add a category label, as it's used by our triage automation to @@ -182,7 +182,7 @@ From the handbook's [Product stages, groups, and categories](https://about.gitlab.com/handbook/product/categories/#hierarchy) page: -> Features: Small, discrete functionalities. e.g. Issue weights. Some common +> Features: Small, discrete functionalities, for example Issue weights. Some common features are listed within parentheses to facilitate finding responsible PMs by keyword. It's highly recommended to add a feature label if no category label applies, as @@ -303,7 +303,7 @@ We automatically add the ~"Accepting merge requests" label to issues that match the [triage policy](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#accepting-merge-requests). We recommend people that have never contributed to any open source project to -look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1) or the `~"Good for new contributors"` [label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) attached to it. +look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1) or the `~"good for new contributors"` [label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) attached to it. More experienced contributors are very welcome to tackle [any of them](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None). @@ -342,19 +342,22 @@ To create a feature proposal, open an issue on the [issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). In order to help track the feature proposals, we have created a -[`feature`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=feature) label. For the time being, users that are not members -of the project cannot add labels. You can instead ask one of the [core team](https://about.gitlab.com/community/core-team/) -members to add the label ~feature to the issue or add the following +[`feature`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=feature) label. +For the time being, users that are not members of the project cannot add labels. +You can instead ask one of the [core team](https://about.gitlab.com/community/core-team/) +members to add the label `~feature` to the issue or add the following code snippet right after your description in a new line: `~feature`. Please keep feature proposals as small and simple as possible, complex ones might be edited to make them small and simple. -Please submit Feature Proposals using the ['Feature Proposal' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md) provided on the issue tracker. +Please submit feature proposals using the ['Feature Proposal' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md) provided on the issue tracker. -For changes in the interface, it is helpful to include a mockup. Issues that add to, or change, the interface should -be given the ~"UX" label. This will allow the UX team to provide input and guidance. You may -need to ask one of the [core team](https://about.gitlab.com/community/core-team/) members to add the label, if you do not have permissions to do it by yourself. +For changes to the user interface (UI), follow our [design and UI guidelines](design.md), +and include a visual example (screenshot, wireframe, or mockup). Such issues should +be given the `~UX"` label for the Product Design team to provide input and guidance. +You may need to ask one of the [core team](https://about.gitlab.com/community/core-team/) +members to add the label, if you do not have permissions to do it by yourself. If you want to create something yourself, consider opening an issue first to discuss whether it is interesting to include this in GitLab. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 25561764bd6..a521d89db2b 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -18,8 +18,8 @@ in order to ensure the work is finished before the release date. If you want to add a new feature that is not labeled, it is best to first create an issue (if there isn't one already) and leave a comment asking for it -to be marked as `Accepting Merge Requests`. Please include screenshots or -wireframes of the proposed feature if it will also change the UI. +to be marked as `Accepting merge requests`. See the [feature proposals](issue_workflow.md#feature-proposals) +section. Merge requests should be submitted to the appropriate project at GitLab.com, for example [GitLab](https://gitlab.com/gitlab-org/gitlab/-/merge_requests), @@ -255,14 +255,14 @@ requirements. 1. The change is tested in a review app where possible and if appropriate. 1. The new feature does not degrade the user experience of the product. 1. The change is evaluated to [limit the impact of far-reaching work](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work). -1. An agreed-upon rollout plan. +1. An agreed-upon [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans/). 1. Merged by a project maintainer. ### Production use 1. Confirmed to be working in staging before implementing the change in production, where possible. 1. Confirmed to be working in the production with no new [Sentry](https://about.gitlab.com/handbook/engineering/#sentry) errors after the contribution is deployed. -1. Confirmed that the rollout plan has been completed. +1. Confirmed that the [rollout plan](https://about.gitlab.com/handbook/engineering/development/processes/rollout-plans) has been completed. 1. If there is a performance risk in the change, I have analyzed the performance of the system before and after the change. 1. *If the merge request uses feature flags, per-project or per-group enablement, and a staged rollout:* - Confirmed to be working on GitLab projects. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index d9b922cb60e..aca37e2182a 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -141,7 +141,7 @@ at GitLab so far: - Their availability: - No "OOO"/"PTO"/"Parental Leave" in their GitLab or Slack status. - No `:red_circle:`/`:palm_tree:`/`:beach:`/`:beach_umbrella:`/`:beach_with_umbrella:` emojis in GitLab or Slack status. - - (Experimental) Their timezone: people for which the local hour is between + - (Experimental) Their time zone: people for which the local hour is between 6 AM and 2 PM are eligible to be picked. This is to ensure they have a good chance to get to perform a review during their current work day. The experimentation is tracked in [this issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/563) diff --git a/doc/development/database/database_migration_pipeline.md b/doc/development/database/database_migration_pipeline.md index 5a8ce89a362..ce7e1801abc 100644 --- a/doc/development/database/database_migration_pipeline.md +++ b/doc/development/database/database_migration_pipeline.md @@ -50,6 +50,6 @@ Some additional information is included at the bottom of the comment: | Result | Description | |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Migrations pending on GitLab.com | A summary of migrations not deployed yet to GitLab.com. This info is useful when testing a migration that was merged but not deployed yet. | +| Migrations pending on GitLab.com | A summary of migrations not deployed yet to GitLab.com. This information is useful when testing a migration that was merged but not deployed yet. | | Clone details | A link to the `Postgres.ai` thin clone created for this testing pipeline, along with information about its expiry. This can be used to further explore the results of running the migration. Only accessible by database maintainers or with an access request. | | Artifacts | A link to the pipeline's artifacts. Full query logs for each migration (ending in `.log`) are available there and only accessible by database maintainers or with an access request. | diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 59653c6dde3..bc18e606f21 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -71,6 +71,7 @@ topics and use cases. The most frequently required during database reviewing are - [Migrations style guide](../migration_style_guide.md) for creating safe SQL migrations. - [Avoiding downtime in migrations](../avoiding_downtime_in_migrations.md). - [SQL guidelines](../sql.md) for working with SQL queries. +- [Guidelines for JiHu contributions with database migrations](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/jihu-database-change-process.html) ## How to apply to become a database maintainer diff --git a/doc/development/database/efficient_in_operator_queries.md b/doc/development/database/efficient_in_operator_queries.md index bc72bce30bf..1e706890f64 100644 --- a/doc/development/database/efficient_in_operator_queries.md +++ b/doc/development/database/efficient_in_operator_queries.md @@ -66,9 +66,9 @@ The execution of the query can be largely broken down into three steps: 1. The database sorts the `issues` rows in memory by `created_at` and returns `LIMIT 20` rows to the end-user. For large groups, this final step requires both large memory and CPU resources. -<details> -<summary>Expand this sentence to see the execution plan for this DB query.</summary> -<pre><code> +Execution plan for this DB query: + +```sql Limit (cost=90170.07..90170.12 rows=20 width=1329) (actual time=967.597..967.607 rows=20 loops=1) Buffers: shared hit=239127 read=3060 I/O Timings: read=336.879 @@ -106,8 +106,7 @@ The execution of the query can be largely broken down into three steps: Planning Time: 7.750 ms Execution Time: 967.973 ms (36 rows) -</code></pre> -</details> +``` The performance of the query depends on the number of rows in the database. On average, we can say the following: @@ -226,7 +225,12 @@ Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( - `finder_query` loads the actual record row from the database. It must also be a lambda, where the order by column expressions is available for locating the record. In this example, the yielded values are `created_at` and `id` SQL expressions. Finding a record is very fast via the - primary key, so we don't use the `created_at` value. + primary key, so we don't use the `created_at` value. Providing the `finder_query` lambda is optional. + If it's not given, the IN operator optimization will only make the ORDER BY columns available to + the end-user and not the full database row. + + If it's not given, the IN operator optimization will only make the ORDER BY columns available to + the end-user and not the full database row. The following database index on the `issues` table must be present to make the query execute efficiently: @@ -235,9 +239,9 @@ to make the query execute efficiently: "idx_issues_on_project_id_and_created_at_and_id" btree (project_id, created_at, id) ``` -<details> -<summary>Expand this sentence to see the SQL query.</summary> -<pre><code> +The SQL query: + +```sql SELECT "issues".* FROM (WITH RECURSIVE "array_cte" AS MATERIALIZED @@ -348,8 +352,7 @@ SELECT (records).* FROM "recursive_keyset_cte" AS "issues" WHERE (COUNT <> 0)) issues -- filtering out the initializer row LIMIT 20 -</code></pre> -</details> +``` ### Using the `IN` query optimization @@ -461,9 +464,9 @@ Gitlab::Pagination::Keyset::InOperatorOptimization::QueryBuilder.new( ).execute.limit(20) ``` -<details> -<summary>Expand this sentence to see the SQL query.</summary> -<pre><code> +The SQL query: + +```sql SELECT "issues".* FROM (WITH RECURSIVE "array_cte" AS MATERIALIZED @@ -581,9 +584,7 @@ FROM FROM "recursive_keyset_cte" AS "issues" WHERE (COUNT <> 0)) issues LIMIT 20 -</code> -</pre> -</details> +``` NOTE: To make the query efficient, the following columns need to be covered with an index: `project_id`, `issue_type`, `created_at`, and `id`. @@ -611,6 +612,32 @@ Gitlab::Pagination::Keyset::Iterator.new(scope: scope, **opts).each_batch(of: 10 end ``` +NOTE: +The query loads complete database rows from the disk. This may cause increased I/O and slower +database queries. Depending on the use case, the primary key is often only +needed for the batch query to invoke additional statements. For example, `UPDATE` or `DELETE`. The +`id` column is included in the `ORDER BY` columns (`created_at` and `id`) and is already +loaded. In this case, you can omit the `finder_query` parameter. + +Example for loading the `ORDER BY` columns only: + +```ruby +scope = Issue.order(:created_at, :id) +array_scope = Group.find(9970).all_projects.select(:id) +array_mapping_scope = -> (id_expression) { Issue.where(Issue.arel_table[:project_id].eq(id_expression)) } + +opts = { + in_operator_optimization_options: { + array_scope: array_scope, + array_mapping_scope: array_mapping_scope + } +} + +Gitlab::Pagination::Keyset::Iterator.new(scope: scope, **opts).each_batch(of: 100) do |records| + puts records.select(:id).map { |r| [r.id] } # only id and created_at are available +end +``` + #### Keyset pagination The optimization works out of the box with GraphQL and the `keyset_paginate` helper method. diff --git a/doc/development/database/keyset_pagination.md b/doc/development/database/keyset_pagination.md index fd62c36b753..4f0b353a37f 100644 --- a/doc/development/database/keyset_pagination.md +++ b/doc/development/database/keyset_pagination.md @@ -169,7 +169,7 @@ Consider the following scope: scope = Issue.where(project_id: 10).order(Gitlab::Database.nulls_last_order('relative_position', 'DESC')) # SELECT "issues".* FROM "issues" WHERE "issues"."project_id" = 10 ORDER BY relative_position DESC NULLS LAST -scope.keyset_paginate # raises: Gitlab::Pagination::Keyset::Paginator::UnsupportedScopeOrder: The order on the scope does not support keyset pagination +scope.keyset_paginate # raises: Gitlab::Pagination::Keyset::UnsupportedScopeOrder: The order on the scope does not support keyset pagination ``` The `keyset_paginate` method raises an error because the order value on the query is a custom SQL string and not an [`Arel`](https://www.rubydoc.info/gems/arel) AST node. The keyset library cannot automatically infer configuration values from these kinds of queries. diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 0fd9f821fab..0ba752ba3a6 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -6,16 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Multiple Databases -In order to scale GitLab, the GitLab application database -will be [decomposed into multiple -databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). +To scale GitLab, the we are +[decomposing the GitLab application database into multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). -## CI Database +## CI/CD Database -Support for configuring the GitLab Rails application to use a distinct -database for CI tables was added in [GitLab -14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289). This -feature is still under development, and is not ready for production use. +> Support for configuring the GitLab Rails application to use a distinct +database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) +in GitLab 14.1. This feature is still under development, and is not ready for production use. By default, GitLab is configured to use only one main database. To opt-in to use a main database, and CI database, modify the @@ -92,8 +90,8 @@ test: &test ### Migrations -Any migrations that affect `Ci::CiDatabaseRecord` models -and their tables must be placed in two directories for now: +Place any migrations that affect `Ci::CiDatabaseRecord` models +and their tables in two directories: - `db/migrate` - `db/ci_migrate` @@ -394,7 +392,8 @@ You can see a real example of using this method for fixing a cross-join in #### Allowlist for existing cross-joins A cross-join across databases can be explicitly allowed by wrapping the code in the -`::Gitlab::Database.allow_cross_joins_across_databases` helper method. +`::Gitlab::Database.allow_cross_joins_across_databases` helper method. Alternative +way is to mark a given relation as `relation.allow_cross_joins_across_databases`. This method should only be used: @@ -405,16 +404,113 @@ This method should only be used: The `allow_cross_joins_across_databases` helper method can be used as follows: ```ruby +# Scope the block executing a object from database ::Gitlab::Database.allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/336590') do subject.perform(1, 4) end ``` +```ruby +# Mark a relation as allowed to cross-join databases +def find_actual_head_pipeline + all_pipelines + .allow_cross_joins_across_databases(url: 'https://gitlab.com/gitlab-org/gitlab/-/issues/336891') + .for_sha_or_source_sha(diff_head_sha) + .first +end +``` + The `url` parameter should point to an issue with a milestone for when we intend to fix the cross-join. If the cross-join is being used in a migration, we do not need to fix the code. See <https://gitlab.com/gitlab-org/gitlab/-/issues/340017> for more details. +### Removing cross-database transactions + +When dealing with multiple databases, it's important to pay close attention to data modification +that affects more than one database. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/339811) GitLab 14.4, an automated check +prevents cross-database modifications. + +When at least two different databases are modified during a transaction initiated on any database +server, the application triggers a cross-database modification error (only in test environment). + +Example: + +```ruby +# Open transaction on Main DB +ApplicationRecord.transaction do + ci_build.update!(updated_at: Time.current) # UPDATE on CI DB + ci_build.project.update!(updated_at: Time.current) # UPDATE on Main DB +end +# raises error: Cross-database data modification of 'main, ci' were detected within +# a transaction modifying the 'ci_build, projects' tables +``` + +The code example above updates the timestamp for two records within a transaction. With the +ongoing work on the CI database decomposition, we cannot ensure the schematics of a database +transaction. +If the second update query fails, the first update query will not be +rolled back because the `ci_build` record is located on a different database server. For +more information, look at the +[transaction guidelines](transaction_guidelines.md#dangerous-example-third-party-api-calls) +page. + +#### Fixing cross-database errors + +##### Removing the transaction block + +Without an open transaction, the cross-database modification check cannot raise an error. +By making this change, we sacrifice consistency. In case of an application failure after the +first `UPDATE` query, the second `UPDATE` query will never execute. + +The same code without the `transaction` block: + +```ruby +ci_build.update!(updated_at: Time.current) # CI DB +ci_build.project.update!(updated_at: Time.current) # Main DB +``` + +##### Async processing + +If we need more guarantee that an operation finishes the work consistently we can execute it +within a background job. A background job is scheduled asynchronously and retried several times +in case of an error. There is still a very small chance of introducing inconsistency. + +Example: + +```ruby +current_time = Time.current + +MyAsyncConsistencyJob.perform_async(cu_build.id) + +ci_build.update!(updated_at: current_time) +ci_build.project.update!(updated_at: current_time) +``` + +The `MyAsyncConsistencyJob` would also attempt to update the timestamp if they differ. + +##### Aiming for perfect consistency + +At this point, we don't have the tooling (we might not even need it) to ensure similar consistency +characteristics as we had with one database. If you think that the code you're working on requires +these properties, then you can disable the cross-database modification check by wrapping to +offending database queries with a block and create a follow-up issue mentioning the sharding group +(`gitlab-org/sharding-group`). + +```ruby +Gitlab::Database.allow_cross_joins_across_databases(url: 'gitlab issue URL') do + ApplicationRecord.transaction do + ci_build.update!(updated_at: Time.current) # UPDATE on CI DB + ci_build.project.update!(updated_at: Time.current) # UPDATE on Main DB + end +end +``` + +Don't hesitate to reach out to the +[sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) +for advice. + ## `config/database.yml` GitLab will support running multiple databases in the future, for example to [separate tables for the continuous integration features](https://gitlab.com/groups/gitlab-org/-/epics/6167) from the main database. In order to prepare for this change, we [validate the structure of the configuration](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67877) in `database.yml` to ensure that only known databases are used. diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index 4c586135015..2806bd217db 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -8,17 +8,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w This document gives a few examples of the usage of database transactions in application code. -For further reference please check PostgreSQL documentation about [transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html). +For further reference, check PostgreSQL documentation about [transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html). ## Database decomposition and sharding -The [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) plans to split the main GitLab database and move some of the database tables to other database servers. +The [sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) plans +to split the main GitLab database and move some of the database tables to other database servers. -The group will start decomposing the `ci_*` related database tables first. To maintain the current application development experience, tooling and static analyzers will be added to the codebase to ensure correct data access and data modification methods. By using the correct form for defining database transactions, we can save significant refactoring work in the future. +We'll start decomposing the `ci_*`-related database tables first. To maintain the current application +development experience, we'll add tooling and static analyzers to the codebase to ensure correct +data access and data modification methods. By using the correct form for defining database transactions, +we can save significant refactoring work in the future. ## The transaction block -The `ActiveRecord` library provides a convenient way to group database statements into a transaction. +The `ActiveRecord` library provides a convenient way to group database statements into a transaction: ```ruby issue = Issue.find(10) @@ -30,16 +34,19 @@ ApplicationRecord.transaction do end ``` -This transaction involves two database tables, in case of an error, each `UPDATE` statement will be rolled back to the previous, consistent state. +This transaction involves two database tables. In case of an error, each `UPDATE` +statement rolls back to the previous consistent state. NOTE: Avoid referencing the `ActiveRecord::Base` class and use `ApplicationRecord` instead. ## Transaction and database locks -When a transaction block is opened, the database will try to acquire the necessary locks on the resources. The type of locks will depend on the actual database statements. +When a transaction block is opened, the database tries to acquire the necessary +locks on the resources. The type of locks depend on the actual database statements. -Consider a concurrent update scenario where the following code is executed at the same time from two different processes: +Consider a concurrent update scenario where the following code is executed at the +same time from two different processes: ```ruby issue = Issue.find(10) @@ -51,15 +58,22 @@ ApplicationRecord.transaction do end ``` -The database will try to acquire the `FOR UPDATE` lock for the referenced `issue` and `project` records. In our case, we have two competing transactions for these locks, one of them will successfully acquire them. The other transaction will have to wait in the lock queue until the first transaction finishes. The execution of the second transaction is blocked at this point. +The database tries to acquire the `FOR UPDATE` lock for the referenced `issue` and +`project` records. In our case, we have two competing transactions for these locks, +and only one of them will successfully acquire them. The other transaction will have +to wait in the lock queue until the first transaction finishes. The execution of the +second transaction is blocked at this point. ## Transaction speed -To prevent lock contention and maintain stable application performance, the transaction block should finish as fast as possible. When a transaction acquires locks, it will hold on to them until the transaction finishes. +To prevent lock contention and maintain stable application performance, the transaction +block should finish as fast as possible. When a transaction acquires locks, it holds +on to them until the transaction finishes. -Apart from application performance, long-running transactions can also affect the application upgrade processes by blocking database migrations. +Apart from application performance, long-running transactions can also affect application +upgrade processes by blocking database migrations. -### Dangerous example: 3rd party API calls +### Dangerous example: third-party API calls Consider the following example: @@ -73,20 +87,29 @@ Member.transaction do end ``` -Here, we ensure that the `notification_email_sent` column is updated only when the `send_notification_email` method succeeds. The `send_notification_email` method executes a network request to an email sending service. If the underlying infrastructure does not specify timeouts or the network call takes too long time, the database transaction will stay open. +Here, we ensure that the `notification_email_sent` column is updated only when the +`send_notification_email` method succeeds. The `send_notification_email` method +executes a network request to an email sending service. If the underlying infrastructure +does not specify timeouts or the network call takes too long time, the database transaction +stays open. Ideally, a transaction should only contain database statements. Avoid doing in a `transaction` block: -- External network requests such as: triggering Sidekiq jobs, sending emails, HTTP API calls and running database statements using a different connection. +- External network requests such as: + - Triggering Sidekiq jobs. + - Sending emails. + - HTTP API calls. + - Running database statements using a different connection. - File system operations. - Long, CPU intensive computation. - Calling `sleep(n)`. ## Explicit model referencing -If a transaction modifies records from the same database table, it's advised to use the `Model.transaction` block: +If a transaction modifies records from the same database table, we advise to use the +`Model.transaction` block: ```ruby build_1 = Ci::Build.find(1) @@ -98,7 +121,8 @@ Ci::Build.transaction do end ``` -The transaction above will use the same database connection for the transaction as the models in the `transaction` block. In a multi-database environment the following example would be dangerous: +The transaction above uses the same database connection for the transaction as the models +in the `transaction` block. In a multi-database environment the following example is dangerous: ```ruby # `ci_builds` table is located on another database @@ -114,4 +138,6 @@ ActiveRecord::Base.transaction do end ``` -The `ActiveRecord::Base` class uses a different database connection than the `Ci::Build` records. The two statements in the transaction block will not be part of the transaction and will not be rolled back in case something goes wrong. They act as 3rd part calls. +The `ActiveRecord::Base` class uses a different database connection than the `Ci::Build` records. +The two statements in the transaction block will not be part of the transaction and will not be +rolled back in case something goes wrong. They act as 3rd part calls. diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index b1c8508c884..7c17a39746e 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -72,8 +72,8 @@ bundle exec rails db -e development ## Access the database with a GUI -Most GUIs (DataGrid, RubyMine, DBeaver) require a TCP connection to the database, but by default -the database runs on a UNIX socket. To be able to access the database from these tools, some steps +Most GUIs (DataGrid, RubyMine, DBeaver) require a TCP connection to the database, but by default +the database runs on a UNIX socket. To be able to access the database from these tools, some steps are needed: 1. On the GDK root directory, run: diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 42bfa656a61..dcd5baab177 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -38,13 +38,24 @@ migration only. ### Required -The following artifacts are required prior to submitting for a ~database review. +You must provide the following artifacts when you request a ~database review. If your merge request description does not include these items, the review will be reassigned back to the author. +#### Migrations + If new migrations are introduced, in the MR **you are required to provide**: - The output of both migrating (`db:migrate`) and rolling back (`db:rollback`) for all migrations. +Note that we have automated tooling for +[GitLab](https://gitlab.com/gitlab-org/gitlab) (provided by the +`db:check-migrations` pipeline job) that provides this output for migrations on +~database merge requests. You do not need to provide this information manually +if the bot can do it for you. The bot also checks that migrations are correctly +reversible. + +#### Queries + If new queries have been introduced or existing queries have been updated, **you are required to provide**: - [Query plans](#query-plans) for each raw SQL query included in the merge request along with the link to the query plan following each raw SQL snippet. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index f8184a562ec..1e85abf585c 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -57,7 +57,7 @@ on non-Go GitLab subsystems. ## Enabling distributed tracing GitLab uses the `GITLAB_TRACING` environment variable to configure distributed tracing. The same -configuration is used for all components (e.g., Workhorse, Rails, etc). +configuration is used for all components (for example, Workhorse, Rails, etc). When `GITLAB_TRACING` is not set, the application isn't instrumented, meaning that there is no overhead at all. diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index 5a4d365ed20..c169af1958e 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -37,14 +37,14 @@ FLAG: | If the feature is... | Use this text | |--------------------------|---------------| -| Available | `On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Unavailable | `On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Available, per-group | `On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Unavailable, per-group | `On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Available, per-project | `On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Unavailable, per-project | `On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Available, per-user | `On self-managed GitLab, by default this feature is available. To hide the feature per user, ask an administrator to [disable the <flag name> flag](<path to>/administration/feature_flags.md).` | -| Unavailable, per-user | `On self-managed GitLab, by default this feature is not available. To make it available per user, ask an administrator to [enable the <flag name> flag](<path to>/administration/feature_flags.md).` | +| Available | `On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Unavailable | `On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Available, per-group | `On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Unavailable, per-group | `On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Available, per-project | `On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to [disable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Unavailable, per-project | `On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Available, per-user | `On self-managed GitLab, by default this feature is available. To hide the feature per user, ask an administrator to [disable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | +| Unavailable, per-user | `On self-managed GitLab, by default this feature is not available. To make it available per user, ask an administrator to [enable the feature flag](<path to>/administration/feature_flags.md) named <flag name>.` | ### GitLab.com availability information @@ -67,7 +67,7 @@ When the state of a flag changes (for example, disabled by default to enabled by Possible version history entries are: ```markdown -> - [Introduced](issue-link) in GitLab X.X. [Deployed behind the <flag name> flag](../../administration/feature_flags.md), disabled by default. +> - [Introduced](issue-link) in GitLab X.X [with a flag](../../administration/feature_flags.md) named <flag name>. Disabled by default. > - [Enabled on GitLab.com](issue-link) in GitLab X.X. > - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. > - [Enabled on self-managed](issue-link) in GitLab X.X. @@ -80,31 +80,31 @@ Possible version history entries are: The following examples show the progression of a feature flag. ```markdown -> Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. +> Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the `forti_token_cloud` flag](../administration/feature_flags.md).` +ask an administrator to [enable the featured flag](../administration/feature_flags.md) named `forti_token_cloud`. The feature is not ready for production use. ``` When the feature is enabled in production, you can update the version history: ```markdown -> - Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. +> - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. FLAG: On self-managed GitLab, by default this feature is available. To hide the feature per user, -ask an administrator to [disable the `forti_token_cloud` flag](../administration/feature_flags.md). +ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `forti_token_cloud`. ``` And, when the feature is done and fully available to all users: ```markdown -> - Introduced in GitLab 13.7. [Deployed behind the `forti_token_cloud` flag](../../administration/feature_flags.md), disabled by default. +> - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. -> - [Feature flag `forti_token_cloud`](https://gitlab.com/issue/etc) removed in GitLab 14.0. +> - [Feature flag removed](https://gitlab.com/issue/etc) in GitLab 14.0. > - [Generally available](issue-link) in GitLab 14.0. ``` diff --git a/doc/development/documentation/img/manual_build_docs.png b/doc/development/documentation/img/manual_build_docs_v14_3.png Binary files differindex e366a2f7ec4..e366a2f7ec4 100644 --- a/doc/development/documentation/img/manual_build_docs.png +++ b/doc/development/documentation/img/manual_build_docs_v14_3.png diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index a597ea512c6..75538fe1fe7 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -108,23 +108,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w If you need help determining the correct stage, read [Ask for help](workflow.md#ask-for-help). -### Document type metadata - -Originally discussed in [this epic](https://gitlab.com/groups/gitlab-org/-/epics/1280), -each page should have a metadata tag called `type`. It can be one or more of the -following: - -- `index`: It consists mostly of a list of links to other pages. - [Example page](../../index.md). -- `concepts`: The background or context of a subject. - [Example page](../../topics/autodevops/index.md). -- `howto`: Specific use case instructions. - [Example page](../../ssh/index.md). -- `tutorial`: Learn a process/concept by doing. - [Example page](../../gitlab-basics/start-using-git.md). -- `reference`: A collection of information used as a reference to use a feature - or a functionality. [Example page](../../ci/yaml/index.md). - ### Redirection metadata The following metadata should be added when a page is moved to another location: @@ -154,7 +137,12 @@ Each page can have additional, optional metadata (set in the [default.html](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fc3577921343173d589dfa43d837b4307e4e620f/layouts/default.html#L30-52) Nanoc layout), which is displayed at the top of the page if defined. -## Move or rename a page +### Deprecated metadata + +The `type` metadata parameter is deprecated but still exists in documentation +pages. You can safely remove the `type` metadata parameter and its values. + +## Move, rename, or delete a page See [redirects](redirects.md). @@ -214,8 +202,10 @@ with the following conventions: The help text follows the [Pajamas guidelines](https://design.gitlab.com/usability/helping-users/#formatting-help-content). -Use the following special cases depending on the context, ensuring all links -are inside `_()` so they can be translated: +#### Linking to `/help` in HAML + +Use the following special cases depending on the context, ensuring all link text +is inside `_()` so it can be translated: - Linking to a doc page. In its most basic form, the HAML code to generate a link to the `/help` page is: @@ -260,6 +250,27 @@ helpPagePath('user/permissions', { anchor: 'anchor-link' }) This is preferred over static paths, as the helper also works on instances installed under a [relative URL](../../install/relative_url.md). +#### Linking to `/help` in Ruby + +To link to the documentation from within Ruby code, use the following code block as a guide, ensuring all link text is inside `_()` so it can +be translated: + +```ruby +docs_link = link_to _('Learn more.'), help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' +_('This is a text describing the option/feature in a sentence. %{docs_link}').html_safe % { docs_link: docs_link.html_safe } +``` + +In cases where you need to generate a link from outside of views/helpers, where the `link_to` and `help_page_url` methods are not available, use the following code block +as a guide where the methods are fully qualified: + +```ruby +docs_link = ActionController::Base.helpers.link_to _('Learn more.'), Rails.application.routes.url_helpers.help_page_url('user/permissions', anchor: 'anchor-link'), target: '_blank', rel: 'noopener noreferrer' +_('This is a text describing the option/feature in a sentence. %{docs_link}').html_safe % { docs_link: docs_link.html_safe } +``` + +Do not use `include ActionView::Helpers::UrlHelper` just to make the `link_to` method available as you might see in some existing code. Read more in +[issue 340567](https://gitlab.com/gitlab-org/gitlab/-/issues/340567). + ### GitLab `/help` tests Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/features/help_pages_spec.rb) diff --git a/doc/development/documentation/redirects.md b/doc/development/documentation/redirects.md index eb6878f5870..ef94c33a276 100644 --- a/doc/development/documentation/redirects.md +++ b/doc/development/documentation/redirects.md @@ -15,13 +15,26 @@ description: Learn how to contribute to GitLab Documentation. # Redirects in GitLab documentation -Moving or renaming a document is the same as changing its location. Be sure -to assign a technical writer to any merge request that renames or moves a page. -Technical Writers can help with any questions and can review your change. +When you move, rename, or delete a page, you must add a redirect. Redirects reduce +how often users get 404s when visiting the documentation site from out-of-date links, like: + +- Bookmarks +- Links from external sites +- Links from old blog posts +- Links in the documentation site global navigation + +Add a redirect to ensure: -When moving or renaming a page, you must redirect browsers to the new page. -This ensures users find the new page, and have the opportunity to update their -bookmarks. +- Users see the new page and can update or delete their bookmark. +- External sites can update their links, especially sites that have automation that + check for redirecting links. +- The documentation site global navigation does not link to a missing page. + + The links in the global navigation are already tested in the `gitlab-docs` project. + If the redirect is missing, the `gitlab-docs` project's `main` branch might break. + +Be sure to assign a technical writer to any merge request that moves, renames, or deletes a page. +Technical Writers can help with any questions and can review your change. There are two types of redirects: diff --git a/doc/development/documentation/review_apps.md b/doc/development/documentation/review_apps.md index 2b8c412f165..a5094ea87f0 100644 --- a/doc/development/documentation/review_apps.md +++ b/doc/development/documentation/review_apps.md @@ -26,7 +26,7 @@ to render and preview the documentation locally. If a merge request has documentation changes, use the `review-docs-deploy` manual job to deploy the documentation review app for your merge request. - + The `review-docs-deploy*` job triggers a cross project pipeline and builds the docs site with your changes. When the pipeline finishes, the review app URL diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index cd69154217c..d1736e10000 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -230,7 +230,7 @@ reports. ## Monthly release process (versions) The docs website supports versions and each month we add the latest one to the list. -For more information, read about the [monthly release process](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases). +For more information, read about the [monthly release process](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md). ## Review Apps for documentation merge requests diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index a9b93997906..229dbb077fe 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -149,6 +149,8 @@ For the heading: If you do not put the full error in the title, include it in the body text. +If multiple causes or workarounds exist, consider putting them into a table format. + ## Other types of content There are other types of content in the GitLab documentation that don't diff --git a/doc/development/documentation/styleguide/img/admin_access_level.png b/doc/development/documentation/styleguide/img/admin_access_level.png Binary files differnew file mode 100644 index 00000000000..191ba78cd6c --- /dev/null +++ b/doc/development/documentation/styleguide/img/admin_access_level.png diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 2cbecc91b20..72491ab3a33 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -1091,47 +1091,51 @@ However, they should be used sparingly because: - They are difficult and expensive to localize. - They cannot be read by screen readers. -If you do include an image in the documentation, ensure it provides value. -Don't use `lorem ipsum` text. Try to replicate how the feature would be -used in a real-world scenario, and [use realistic text](#fake-user-information). +When needed, use images to help the reader understand: -### Capture the image +- Where they are in a complicated process. +- How they should interact with the application. -Use images to help the reader understand where they are in a process, or how -they need to interact with the application. +### Capture the image When you take screenshots: -- **Capture the most relevant area of the page.** Don't include unnecessary white - space or areas of the page that don't help illustrate the point. The left - sidebar of the GitLab user interface can change, so don't include the sidebar - if it's not necessary. +- **Ensure it provides value.** Don't use `lorem ipsum` text. + Try to replicate how the feature would be used in a real-world scenario, and + [use realistic text](#fake-user-information). +- **Capture only the relevant UI.** Don't include unnecessary white + space or areas of the UI that don't help illustrate the point. The + sidebars in GitLab can change, so don't include + them in screenshots unless absolutely necessary. - **Keep it small.** If you don't need to show the full width of the screen, don't. - A value of 1000 pixels is a good maximum width for your screenshot image. + Reduce the size of your browser window as much as possible to keep elements close + together and reduce empty space. Try to keep the screenshot dimensions as small as possible. +- **Review how the image renders on the page.** Preview the image locally or use the +review app in the merge request. Make sure the image isn't blurry or overwhelming. - **Be consistent.** Coordinate screenshots with the other screenshots already on - a documentation page. For example, if other screenshots include the left - sidebar, include the sidebar in all screenshots. + a documentation page for a consistent reading experience. ### Save the image +- Resize any wide or tall screenshots if needed, but make sure the screenshot is + still clear after being resized and compressed. +- All images **must** be [compressed](#compress-images) to 100KB or less. + In many cases, 25-50KB or less is often possible without reducing image quality. - Save the image with a lowercase filename that's descriptive of the feature - or concept in the image. If the image is of the GitLab interface, append the - GitLab version to the filename, based on this format: - `image_name_vX_Y.png`. For example, for a screenshot taken from the pipelines - page of GitLab 11.1, a valid name is `pipelines_v11_1.png`. If you're adding an - illustration that doesn't include parts of the user interface, add the release - number corresponding to the release the image was added to; for an MR added to - 11.1's milestone, a valid name for an illustration is `devops_diagram_v11_1.png`. + or concept in the image: + - If the image is of the GitLab interface, append the GitLab version to the filename, + based on this format: `image_name_vX_Y.png`. For example, for a screenshot taken + from the pipelines page of GitLab 11.1, a valid name is `pipelines_v11_1.png`. + - If you're adding an illustration that doesn't include parts of the user interface, + add the release number corresponding to the release the image was added to. + For an MR added to 11.1's milestone, a valid name for an illustration is `devops_diagram_v11_1.png`. - Place images in a separate directory named `img/` in the same directory where the `.md` document that you're working on is located. - Consider using PNG images instead of JPEG. -- [Compress all PNG images](#compress-images). - Compress GIFs with <https://ezgif.com/optimize> or similar tool. - Images should be used (only when necessary) to illustrate the description of a process, not to replace it. -- Max image size: 100KB (GIFs included). -- See also how to link and embed [videos](#videos) to illustrate the - documentation. +- See also how to link and embed [videos](#videos) to illustrate the documentation. ### Add the image link to content @@ -1152,8 +1156,11 @@ known tool is [`pngquant`](https://pngquant.org/), which is cross-platform and open source. Install it by visiting the official website and following the instructions for your OS. +If you use macOS and want all screenshots to be compressed automatically, read +[One simple trick to make your screenshots 80% smaller](https://about.gitlab.com/blog/2020/01/30/simple-trick-for-smaller-screenshots/). + GitLab has a [Ruby script](https://gitlab.com/gitlab-org/gitlab/-/blob/master/bin/pngquant) -that you can use to automate the process. In the root directory of your local +that you can use to simplify the manual process. In the root directory of your local copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: - Before compressing, if you want, check that all documentation PNG images have @@ -1360,13 +1367,15 @@ Do not use words to describe the icon: ## Alert boxes -Use alert boxes to call attention to information. +Use alert boxes to call attention to information. Use them sparingly, and never have an alert box immediately follow another alert box. Alert boxes are generated when one of these words is followed by a line break: - `FLAG:` - `NOTE:` - `WARNING:` +- `INFO:` (Marketing only) +- `DISCLAIMER:` For example: @@ -1423,6 +1432,58 @@ It renders on the GitLab documentation site as: WARNING: This is something to be warned about. +### Info + +The Marketing team uses the `INFO` alert to add information relating +to sales and marketing efforts. + +The text in an `INFO:` alert always renders in a floating text box to the right of the text around it. +To view the rendered GitLab docs site, check the review app in the MR. You might need to move the text up or down +in the surrounding text, depending on where you'd like to floating box to appear. + +For example, if your page has text like this: + +```markdown +This is an introductory paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +INFO: +Here is some information. This information is an important addition to how you +work with GitLab and you might want to consider it. + +And here is another paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +And here is another paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. +``` + +It renders on the GitLab documentation site as: + +This is an introductory paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +INFO: +Here is some information. This information is an important addition to how you +work with GitLab and you might want to consider it. + +And here is another paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +And here is another paragraph. GitLab uses the SSH protocol to securely communicate with Git. +When you use SSH keys to authenticate to the GitLab remote server, +you don't need to supply your username and password each time. + +### Disclaimer + +Use to describe future functionality only. +For more information, see [Legal disclaimer for future features](#legal-disclaimer-for-future-features). + ## Blockquotes For highlighting a text inside a blockquote, use this format: @@ -1616,6 +1677,34 @@ For example: You can say that we plan to remove a feature. +#### Legal disclaimer for future features + +If you **must** write about features we have not yet delivered, put this exact disclaimer near the content it applies to. + +```markdown +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. +``` + +It renders on the GitLab documentation site as: + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +If all of the content on the page is not available, use the disclaimer once at the top of the page. + +If the content in a topic is not ready, use the disclaimer in the topic. + ### Removing versions after each major release Whenever a major GitLab release occurs, we remove all version references diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index eafe0e7a1c2..f1e6a147571 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -29,9 +29,39 @@ Try to avoid using **above** when referring to an example or table in a document - In the previous example, the dog had fleas. -## admin, admin area +Do not use **above** when referring to versions of the product. Use [**later**](#later) instead. -Use **administration**, **administrator**, **administer**, or **Admin Area** instead. ([Vale](../testing.md#vale) rule: [`Admin.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Admin.yml)) +- Do: In GitLab 14.4 and later... +- Do not: In GitLab 14.4 and above... +- Do not: In GitLab 14.4 and higher... + +## access level + +Access levels are different than [roles](#roles) or [permissions](#permissions). +When you create a user, you choose an access level: **Regular**, **Auditor**, or **Admin**. + +Capitalize these words when you refer to the UI. Otherwise use lowercase. + +## administrator + +Use **administrator** instead of **admin** when talking about a user's access level. +Use lowercase unless you are referring to the **Admin** access level you select in the UI. + +To view the administrator access type, in the GitLab UI, go to the Admin Area and select +**Users**. Then select **New user**. + + + +An **administrator** is not a [role](#roles) or [permission](#permissions). + +- Do: To do this thing, you must be an administrator. +- Do: To do this thing, you must have the administrator access level. +- Do not: To do this thing, you must have the Admin role. + +## Admin Area + +Use title case **Admin Area** to refer to the area of the UI that you access when you select **Menu > Admin**. +This area of the UI says **Admin Area** at the top of the page and on the menu. ## allow, enable @@ -54,9 +84,14 @@ in the handbook when writing about Alpha features. Instead of **and/or**, use **or** or rewrite the sentence to spell out both options. +## and so on + +Do not use **and so on**. Instead, be more specific. For details, see +[the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/and-so-on). + ## area -Use [**section**](#section) instead of **area**. The only exception is [the Admin Area](#admin-admin-area). +Use [**section**](#section) instead of **area**. The only exception is [the Admin Area](#admin-area). ## below @@ -150,8 +185,8 @@ When writing about the Developer role: - Do not use the phrase, **if you are a developer** to mean someone who is assigned the Developer role. Instead, write it out. For example, **if you are assigned the Developer role**. - To describe a situation where the Developer role is the minimum required: - - Avoid: the Developer role or higher - - Use instead: at least the Developer role + - Do: at least the Developer role + - Do not: the Developer role or higher Do not use **Developer permissions**. A user who is assigned the Developer role has a set of associated permissions. @@ -207,7 +242,8 @@ Use lowercase for **epic board**. ## etc. -Try to avoid **etc.**. Be as specific as you can. Do not use **and so on** as a replacement. +Try to avoid **etc.**. Be as specific as you can. Do not use +[**and so on**](#and-so-on) as a replacement. - Do: You can update objects, like merge requests and issues. - Do not: You can update objects, like merge requests, issues, etc. @@ -220,8 +256,8 @@ Use **expand** instead of **open** when you are talking about expanding or colla Use **box** instead of **field** or **text box**. -- Avoid: In the **Variable name** field, enter `my text`. -- Use instead: In the **Variable name** box, enter `my text`. +- Do: In the **Variable name** box, enter `my text`. +- Do not: In the **Variable name** field, enter `my text`. ## foo @@ -265,8 +301,8 @@ When writing about the Guest role: - Do not use the phrase, **if you are a guest** to mean someone who is assigned the Guest role. Instead, write it out. For example, **if you are assigned the Guest role**. - To describe a situation where the Guest role is the minimum required: - - Avoid: the Guest role or higher - - Use instead: at least the Guest role + - Do: at least the Guest role + - Do not: the Guest role or higher Do not use **Guest permissions**. A user who is assigned the Guest role has a set of associated permissions. @@ -282,15 +318,16 @@ Do not use **high availability** or **HA**. Instead, direct readers to the GitLa Do not use **higher** when talking about version numbers. -- Do: In GitLab 14.1 and later. -- Do not: In GitLab 14.1 and higher. +- Do: In GitLab 14.4 and later... +- Do not: In GitLab 14.4 and higher... +- Do not: In GitLab 14.4 and above... ## hit Don't use **hit** to mean **press**. -- Avoid: Hit the **ENTER** button. -- Use instead: Press **ENTER**. +- Do: Press **ENTER**. +- Do not: Hit the **ENTER** button. ## I @@ -326,8 +363,9 @@ If you want to use **CI** with the word **job**, use **CI/CD job** rather than * Use **later** when talking about version numbers. -- Avoid: In GitLab 14.1 and higher. -- Use instead: In GitLab 14.1 and later. +- Do: In GitLab 14.1 and later... +- Do not: In GitLab 14.1 and higher... +- Do not: In GitLab 14.1 and above... ## list @@ -354,8 +392,8 @@ When writing about the Maintainer role: - Do not use the phrase, **if you are a maintainer** to mean someone who is assigned the Maintainer role. Instead, write it out. For example, **if you are assigned the Maintainer role**. - To describe a situation where the Maintainer role is the minimum required: - - Avoid: the Maintainer role or higher - - Use instead: at least the Maintainer role + - Do: at least the Maintainer role + - Do not: the Maintainer role or higher Do not use **Maintainer permissions**. A user who is assigned the Maintainer role has a set of associated permissions. @@ -416,6 +454,13 @@ Do not use **note that** because it's wordy. - Do: You can change the settings. - Do not: Note that you can change the settings. +## once + +The word **once** means **one time**. Don't use it to mean **after** or **when**. + +- Do: When the process is complete... +- Do not: Once the process is complete... + ## Owner When writing about the Owner role: @@ -429,7 +474,9 @@ Do not use **Owner permissions**. A user who is assigned the Owner role has a se ## permissions -Do not use **roles** and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. +Do not use [**roles**](#roles) and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. + +Permissions are not the same as [**access levels**](#access-level). ## please @@ -454,8 +501,8 @@ When writing about the Reporter role: - Do not use the phrase, **if you are a reporter** to mean someone who is assigned the Reporter role. Instead, write it out. For example, **if you are assigned the Reporter role**. - To describe a situation where the Reporter role is the minimum required: - - Avoid: the Reporter role or higher - - Use instead: at least the Reporter role + - Do: at least the Reporter role + - Do not: the Reporter role or higher Do not use **Reporter permissions**. A user who is assigned the Reporter role has a set of associated permissions. @@ -465,12 +512,23 @@ Use title case for **Repository Mirroring**. ## roles -Do not use **roles** and **permissions** interchangeably. Each user is assigned a role. Each role includes a set of permissions. +Do not use **roles** and [**permissions**](#permissions) interchangeably. Each user is assigned a role. Each role includes a set of permissions. + +Roles are not the same as [**access levels**](#access-level). ## runner, runners Use lowercase for **runners**. These are the agents that run CI/CD jobs. See also [GitLab Runner](#gitlab-runner) and [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/233529). +## (s) + +Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: + +Do: Select the jobs you want. +Do not: Select the job(s) you want. + +If you can select multiples of something, then write the word as plural. + ## sanity check Do not use **sanity check**. Use **check for completeness** instead. ([Vale](../testing.md#vale) rule: [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml)) @@ -514,6 +572,13 @@ You can use **single sign-on**. Do not use **simply** or **simple**. If the user doesn't find the process to be simple, we lose their trust. ([Vale](../testing.md#vale) rule: [`Simplicity.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Simplicity.yml)) +## since + +The word **since** indicates a timeframe. For example, **Since 1984, Bon Jovi has existed**. Don't use **since** to mean **because**. + +- Do: Because you have the Developer role, you can delete the widget. +- Do not: Since you have the Developer role, you can delete the widget. + ## slashes Instead of **and/or**, use **or** or re-write the sentence. This rule also applies to other slashes, like **follow/unfollow**. Some exceptions (like **CI/CD**) are allowed. diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index dfa2f3ed55a..13648a7c710 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -206,7 +206,6 @@ Vale returns three types of results: `suggestion`, `warning`, and `error`: (after the Technical Writing team completes its cleanup). Warnings don't break CI. See a list of [warning-level rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Warning%3A&group_id=9970&project_id=278964). - **Error**-level results are Style Guide violations, and should contain clear explanations - about how to resolve the error. Errors break CI and are displayed in CI job output. of how to resolve the error. Errors break CI and are displayed in CI job output. See a list of [error-level rules](https://gitlab.com/search?utf8=✓&snippets=false&scope=&repository_ref=master&search=path%3Adoc%2F.vale%2Fgitlab+Error%3A&group_id=9970&project_id=278964). diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 31c38bc1446..90c1137e5c5 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -99,7 +99,7 @@ The process involves the following: - Primary Reviewer. Review by a [code reviewer](https://about.gitlab.com/handbook/engineering/projects/) or other appropriate colleague to confirm accuracy, clarity, and completeness. This can be skipped for minor fixes without substantive content changes. -- Technical Writer (Optional). If not completed for a merge request prior to merging, must be scheduled +- Technical Writer (Optional). If not completed for a merge request before merging, must be scheduled post-merge. Schedule post-merge reviews only if an urgent merge is required. To request a: - Pre-merge review, assign the Technical Writer listed for the applicable [DevOps stage group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments). @@ -111,7 +111,7 @@ The process involves the following: - Ensure the appropriate labels are applied, including any required to pick a merge request into a release. - Ensure that, if there has not been a Technical Writer review completed or scheduled, they - [create the required issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review), assign to the Technical Writer of the given stage group, + [create the required issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review), assign it to the Technical Writer of the given stage group, and link it from the merge request. The process is reflected in the **Documentation** @@ -130,10 +130,10 @@ immediately after merge by the developer or maintainer. For this, create an issue using the [Doc Review description template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Doc%20Review) and link to it from the merged merge request that introduced the documentation change. -Circumstances where a regular pre-merge Technical Writer review might be skipped include: +Circumstances, where a regular pre-merge Technical Writer review might be skipped, include: -- There is a short amount of time left before the milestone release. If there are less than three days - remaining, seek a post-merge review and ping the writer via Slack to ensure the review is +- There is a short amount of time left before the milestone release. If less than three + days are remaining, seek a post-merge review and ping the writer via Slack to ensure the review is completed as soon as possible. - The size of the change is small and you have a high degree of confidence that early users of the feature (for example, GitLab.com users) can easily @@ -156,15 +156,15 @@ Remember: Ensure the following if skipping an initial Technical Writer review: -- That [product badges](styleguide/index.md#product-tier-badges) are applied. -- That the GitLab [version](styleguide/index.md#gitlab-versions) that - introduced the feature has been included. -- That changes to headings don't affect in-app hyperlinks. +- [Product badges](styleguide/index.md#product-tier-badges) are applied. +- The GitLab [version](styleguide/index.md#gitlab-versions) that + introduced the feature is included. +- Changes to headings don't affect in-app hyperlinks. - Specific [user permissions](../../user/permissions.md) are documented. -- That new documents are linked from higher-level indexes, for discoverability. -- Style guide is followed: +- New documents are linked from higher-level indexes, for discoverability. +- The style guide is followed: - For [directories and files](styleguide/index.md#work-with-directories-and-files). - For [images](styleguide/index.md#images). Merge requests that change the location of documentation must always be reviewed by a Technical -Writer prior to merging. +Writer before merging. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 42fb9fd42fc..7f74d9660e9 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -40,7 +40,7 @@ By default, merge request pipelines for development run in an EE-context only. I developing features that differ between FOSS and EE, you may wish to run pipelines in a FOSS context as well. -To run pipelines in both contexts, include `RUN AS-IF-FOSS` in the merge request title. +To run pipelines in both contexts, add the `~"pipeline:run-as-if-foss"` label to the merge request. See the [As-if-FOSS jobs](pipelines.md#as-if-foss-jobs) pipelines documentation for more information. diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index bba4e1cda23..68d8b424331 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -233,11 +233,11 @@ Any data or index cleanup needed to support migration retries should be handled will re-enqueue itself with a delay which is set using the `throttle_delay` option described below. The batching must be handled within the `migrate` method, this setting controls the re-enqueuing only. -- `batch_size` - Sets the number of documents modified during a `batched!` migration run. This size should be set to a value which allows the updates - enough time to finish. This can be tuned in combination with the `throttle_delay` option described below. The batching - must be handled within a custom `migrate` method or by using the [`Elastic::MigrationBackfillHelper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/concerns/elastic/migration_backfill_helper.rb) - `migrate` method which uses this setting. Default value is 1000 documents. - +- `batch_size` - Sets the number of documents modified during a `batched!` migration run. This size should be set to a value which allows the updates +enough time to finish. This can be tuned in combination with the `throttle_delay` option described below. The batching +must be handled within a custom `migrate` method or by using the [`Elastic::MigrationBackfillHelper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/concerns/elastic/migration_backfill_helper.rb) +`migrate` method which uses this setting. Default value is 1000 documents. + - `throttle_delay` - Sets the wait time in between batch runs. This time should be set high enough to allow each migration batch enough time to finish. Additionally, the time should be less than 30 minutes since that is how often the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 4de272fec20..1b1f756d4c0 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -61,7 +61,7 @@ Therefore, you should postpone this effort until the [experiment cleanup process We recommend the following workflow: -1. Review the Pajamas guidelines for [icons](https://design.gitlab.com/product-foundations/iconography) and [illustrations](https://design.gitlab.com/product-foundations/illustration). +1. Review the Pajamas guidelines for [icons](https://design.gitlab.com/product-foundations/iconography/) and [illustrations](https://design.gitlab.com/product-foundations/illustration/). 1. Add an icon or illustration as an `.svg` file in the `/app/assets/images` (or EE) path in the GitLab repository. 1. Use `image_tag` or `image_path` to render it via the asset pipeline. 1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index 7c870de9a6c..c4ebef4c289 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -334,7 +334,7 @@ Keep in mind that: - When you add `:hover` styles, in most cases you should add `:focus` styles too so that the styling is applied for both mouse **and** keyboard users. - If you remove an interactive element's `outline`, make sure you maintain visual focus state in another way such as with `box-shadow`. -See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/keyboard-only) for more detail. +See the [Pajamas Keyboard-only page](https://design.gitlab.com/accessibility-audits/keyboard-only/) for more detail. ## Tabindex diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md index 956e7d0d56e..139825655e9 100644 --- a/doc/development/fe_guide/content_editor.md +++ b/doc/development/fe_guide/content_editor.md @@ -11,7 +11,7 @@ experience for [GitLab Flavored Markdown](../../user/markdown.md) in the GitLab It also serves as the foundation for implementing Markdown-focused editors that target other engines, like static site generators. -We use [tiptap 2.0](https://www.tiptap.dev/) and [ProseMirror](https://prosemirror.net/) +We use [tiptap 2.0](https://tiptap.dev/) and [ProseMirror](https://prosemirror.net/) to build the Content Editor. These frameworks provide a level of abstraction on top of the native [`contenteditable`](https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Editable_content) web technology. @@ -143,7 +143,7 @@ The Content Editor is composed of three main layers: ### Editing tools UI The editing tools UI are Vue components that display the editor's state and -dispatch [commands](https://www.tiptap.dev/api/commands/#commands) to mutate it. +dispatch [commands](https://tiptap.dev/api/commands/#commands) to mutate it. They are located in the `~/content_editor/components` directory. For example, the **Bold** toolbar button displays the editor's state by becoming active when the user selects bold text. This button also dispatches the `toggleBold` command @@ -159,7 +159,7 @@ sequenceDiagram #### Node views -We implement [node views](https://www.tiptap.dev/guide/node-views/vue/#node-views-with-vue) +We implement [node views](https://tiptap.dev/guide/node-views/vue/#node-views-with-vue) to provide inline editing tools for some content types, like tables and images. Node views allow separating the presentation of a content type from its [model](https://prosemirror.net/docs/guide/#doc.data_structures). Using a Vue component in @@ -209,7 +209,7 @@ the following events: - `blur` - `error`. -Learn more about these events in [Tiptap's event guide](https://www.tiptap.dev/api/events/). +Learn more about these events in [Tiptap's event guide](https://tiptap.dev/api/events/). ```html <script> @@ -246,7 +246,7 @@ export default { ### The Tiptap editor object -The Tiptap [Editor](https://www.tiptap.dev/api/editor) class manages +The Tiptap [Editor](https://tiptap.dev/api/editor) class manages the editor's state and encapsulates all the business logic that powers the Content Editor. The Content Editor constructs a new instance of this class and provides all the necessary extensions to support @@ -255,9 +255,9 @@ provides all the necessary extensions to support #### Implement new extensions Extensions are the building blocks of the Content Editor. You can learn how to implement -new ones by reading [Tiptap's guide](https://www.tiptap.dev/guide/custom-extensions). -We recommend checking the list of built-in [nodes](https://www.tiptap.dev/api/nodes) and -[marks](https://www.tiptap.dev/api/marks) before implementing a new extension +new ones by reading [Tiptap's guide](https://tiptap.dev/guide/custom-extensions). +We recommend checking the list of built-in [nodes](https://tiptap.dev/api/nodes) and +[marks](https://tiptap.dev/api/marks) before implementing a new extension from scratch. Store the Content Editor extensions in the `~/content_editor/extensions` directory. @@ -326,8 +326,8 @@ sequenceDiagram ``` Deserializers live in the extension modules. Read Tiptap's -[parseHTML](https://www.tiptap.dev/guide/custom-extensions#parse-html) and -[addAttributes](https://www.tiptap.dev/guide/custom-extensions#attributes) documentation to +[parseHTML](https://tiptap.dev/guide/custom-extensions#parse-html) and +[addAttributes](https://tiptap.dev/guide/custom-extensions#attributes) documentation to learn how to implement them. Titap's API is a wrapper around ProseMirror's [schema spec API](https://prosemirror.net/docs/ref/#model.SchemaSpec). diff --git a/doc/development/fe_guide/droplab/droplab.md b/doc/development/fe_guide/droplab/droplab.md deleted file mode 100644 index 8f1ecc115fe..00000000000 --- a/doc/development/fe_guide/droplab/droplab.md +++ /dev/null @@ -1,281 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# DropLab - -A generic dropdown for all of your custom dropdown needs. - -## Usage - -DropLab can be used by adding a `data-dropdown-trigger` HTML attribute. This -attribute allows us to find the "trigger" _(toggle)_ for the dropdown, whether -it's a button, link or input. - -The value of the `data-dropdown-trigger` should be a CSS selector that DropLab -can use to find the trigger's dropdown list. - -You should also add the `data-dropdown` attribute to declare the dropdown list. -The value is irrelevant. - -The DropLab class has no side effects, so you must always call `.init` when the -DOM is ready. `DropLab.prototype.init` takes the same arguments as `DropLab.prototype.addHook`. -If you don't provide any arguments, it globally queries and instantiates all -DropLab-compatible dropdowns. - -```html -<a href="#" data-dropdown-trigger="#list">Toggle</a> - -<ul id="list" data-dropdown> - <!-- ... --> -<ul> -``` - -```javascript -const droplab = new DropLab(); -droplab.init(); -``` - -As noted, we have a "Toggle" link that's declared as a trigger. It provides a -selector to find the dropdown list it should control. - -### Static data - -You can add static list items. - -```html -<a href="#" data-dropdown-trigger="#list">Toggle</a> - -<ul id="list" data-dropdown> - <li>Static value 1</li> - <li>Static value 2</li> -<ul> -``` - -```javascript -const droplab = new DropLab(); -droplab.init(); -``` - -### Explicit instantiation - -You can pass the trigger and list elements as constructor arguments to return a -non-global instance of DropLab using the `DropLab.prototype.init` method. - -```html -<a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> - -<ul id="list" data-dropdown> - <!-- ... --> -<ul> -``` - -```javascript -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -const droplab = new DropLab(); -droplab.init(trigger, list); -``` - -You can also add hooks to an existing DropLab instance using `DropLab.prototype.addHook`. - -```html -<a href="#" data-dropdown-trigger="#auto-dropdown">Toggle</a> -<ul id="auto-dropdown" data-dropdown><!-- ... --><ul> - -<a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> -<ul id="list" data-dropdown><!-- ... --><ul> -``` - -```javascript -const droplab = new DropLab(); - -droplab.init(); - -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -droplab.addHook(trigger, list); -``` - -### Dynamic data - -Adding `data-dynamic` to your dropdown element enables dynamic list -rendering. - -You can template a list item using the keys of the data object provided. Use the -handlebars syntax `{{ value }}` to HTML escape the value. Use the `<%= value %>` -syntax to interpolate the value. Use the `<%= value %>` syntax to evaluate the -value. - -Passing an array of objects to `DropLab.prototype.addData` renders that data -for all `data-dynamic` dropdown lists tracked by that DropLab instance. - -```html -<a href="#" data-dropdown-trigger="#list">Toggle</a> - -<ul id="list" data-dropdown data-dynamic> - <li><a href="#" data-id="{{id}}">{{text}}</a></li> -</ul> -``` - -```javascript -const droplab = new DropLab(); - -droplab.init().addData([{ - id: 0, - text: 'Jacob', -}, { - id: 1, - text: 'Jeff', -}]); -``` - -Alternatively, you can specify a specific dropdown to add this data to by -passing the data as the second argument and the `id` of the trigger element as -the first argument. - -```html -<a href="#" data-dropdown-trigger="#list" id="trigger">Toggle</a> - -<ul id="list" data-dropdown data-dynamic> - <li><a href="#" data-id="{{id}}">{{text}}</a></li> -</ul> -``` - -```javascript -const droplab = new DropLab(); - -droplab.init().addData('trigger', [{ - id: 0, - text: 'Jacob', -}, { - id: 1, - text: 'Jeff', -}]); -``` - -This allows you to mix static and dynamic content, even with one trigger. - -Note the use of scoping regarding the `data-dropdown` attribute to capture both -dropdown lists, one of which is dynamic. - -```html -<input id="trigger" data-dropdown-trigger="#list"> -<div id="list" data-dropdown> - <ul> - <li><a href="#">Static item 1</a></li> - <li><a href="#">Static item 2</a></li> - </ul> - <ul data-dynamic> - <li><a href="#" data-id="{{id}}">{{text}}</a></li> - </ul> -</div> -``` - -```javascript -const droplab = new DropLab(); - -droplab.init().addData('trigger', [{ - id: 0, - text: 'Jacob', -}, { - id: 1, - text: 'Jeff', -}]); -``` - -## Internal selectors - -DropLab adds some CSS classes to help lower the barrier to integration. - -For example: - -- The `droplab-item-selected` CSS class is added to items that have been - selected either by a mouse click or by enter key selection. -- The `droplab-item-active` CSS class is added to items that have been selected - using arrow key navigation. -- You can add the `droplab-item-ignore` CSS class to any item that you don't - want to be selectable. For example, an `<li class="divider"></li>` list - divider element that shouldn't be interactive. - -## Internal events - -DropLab uses some custom events to help lower the barrier to integration. - -For example: - -- The `click.dl` event is fired when an `li` list item has been clicked. It's - also fired when a list item has been selected with the keyboard. It's also - fired when a `HookButton` button is clicked (a registered `button` tag or `a` - tag trigger). -- The `input.dl` event is fired when a `HookInput` (a registered `input` tag - trigger) triggers an `input` event. -- The `mousedown.dl` event is fired when a `HookInput` triggers a `mousedown` - event. -- The `keyup.dl` event is fired when a `HookInput` triggers a `keyup` event. -- The `keydown.dl` event is fired when a `HookInput` triggers a `keydown` event. - -These custom events add a `detail` object to the vanilla `Event` object that -provides some potentially useful data. - -## Plugins - -Plugins are objects that are registered to be executed when a hook is added (when -a DropLab trigger and dropdown are instantiated). - -If no modules API is detected, the library falls back as it does with -`window.DropLab` and adds `window.DropLab.plugins.PluginName`. - -### Usage - -To use plugins, you can pass them in an array as the third argument of -`DropLab.prototype.init` or `DropLab.prototype.addHook`. Some plugins require -configuration values; the configuration object can be passed as the fourth argument. - -```html -<a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> -<ul id="list" data-dropdown><!-- ... --><ul> -``` - -```javascript -const droplab = new DropLab(); - -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -droplab.init(trigger, list, [droplabAjax], { - droplabAjax: { - endpoint: '/some-endpoint', - method: 'setData', - }, -}); -``` - -### Documentation - -Refer to the list of available [DropLab plugins](plugins/index.md) for -information about their use. - -### Development - -When plugins are initialised for a DropLab trigger+dropdown, DropLab calls the -plugins' `init` function, so this must be implemented in the plugin. - -```javascript -class MyPlugin { - static init() { - this.someProp = 'someProp'; - this.someMethod(); - } - - static someMethod() { - this.otherProp = 'otherProp'; - } -} - -export default MyPlugin; -``` diff --git a/doc/development/fe_guide/droplab/plugins/ajax.md b/doc/development/fe_guide/droplab/plugins/ajax.md deleted file mode 100644 index f12f8f260c7..00000000000 --- a/doc/development/fe_guide/droplab/plugins/ajax.md +++ /dev/null @@ -1,46 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Ajax plugin - -`Ajax` is a DropLab plugin that allows for retrieving and rendering list data -from a server. - -## Usage - -Add the `Ajax` object to the plugins array of a `DropLab.prototype.init` or -`DropLab.prototype.addHook` call. - -`Ajax` requires 2 configuration values: the `endpoint` and `method`. - -- `endpoint`: Should be a URL to the request endpoint. -- `method`: Should be `setData` or `addData`. -- `setData`: Completely replaces the dropdown with the response data. -- `addData`: Appends the response data to the current dropdown list. - -```html -<a href="#" id="trigger" data-dropdown-trigger="#list">Toggle</a> -<ul id="list" data-dropdown><!-- ... --><ul> -``` - -```javascript -const droplab = new DropLab(); - -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -droplab.addHook(trigger, list, [Ajax], { - Ajax: { - endpoint: '/some-endpoint', - method: 'setData', - }, -}); -``` - -Optionally, you can set `loadingTemplate` to a HTML string. This HTML string -replaces the dropdown list while the request is pending. - -Additionally, you can set `onError` to a function to catch any XHR errors. diff --git a/doc/development/fe_guide/droplab/plugins/filter.md b/doc/development/fe_guide/droplab/plugins/filter.md deleted file mode 100644 index 79f10cdb6c1..00000000000 --- a/doc/development/fe_guide/droplab/plugins/filter.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Filter plugin - -`Filter` is a DropLab plugin that allows for filtering data that has been added -to the dropdown using a simple fuzzy string search of an input value. - -## Usage - -Add the `Filter` object to the plugins array of a `DropLab.prototype.init` or -`DropLab.prototype.addHook` call. - -- `Filter`: Requires a configuration value for `template`. -- `template`: Should be the key of the objects within your data array that you - want to compare to the user input string, for filtering. - -```html -<input href="#" id="trigger" data-dropdown-trigger="#list"> -<ul id="list" data-dropdown data-dynamic> - <li><a href="#" data-id="{{id}}">{{text}}</a></li> -<ul> -``` - -```javascript -const droplab = new DropLab(); - -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -droplab.init(trigger, list, [Filter], { - Filter: { - template: 'text', - }, -}); - -droplab.addData('trigger', [{ - id: 0, - text: 'Jacob', -}, { - id: 1, - text: 'Jeff', -}]); -``` - -In the previous code, the input string is compared against the `test` key of the -passed data objects. - -Optionally you can set `filterFunction` to a function. This function is then -used instead of `Filter`'s built-in string search. `filterFunction` is passed -two arguments: the first is one of the data objects, and the second is the -current input value. diff --git a/doc/development/fe_guide/droplab/plugins/index.md b/doc/development/fe_guide/droplab/plugins/index.md deleted file mode 100644 index c7a2865ca83..00000000000 --- a/doc/development/fe_guide/droplab/plugins/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -description: A list of DropLab plugins. ---- - -# DropLab plugins - -The following plugins are available for use with [DropLab](../droplab.md): - -- [Ajax plugin](ajax.md) -- [Filter plugin](filter.md) -- [InputSetter plugin](input_setter.md) diff --git a/doc/development/fe_guide/droplab/plugins/input_setter.md b/doc/development/fe_guide/droplab/plugins/input_setter.md deleted file mode 100644 index a3c073520cb..00000000000 --- a/doc/development/fe_guide/droplab/plugins/input_setter.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# InputSetter plugin - -`InputSetter` is a DropLab plugin that allows for updating DOM out of the scope -of DropLab when a list item is clicked. - -## Usage - -Add the `InputSetter` object to the plugins array of a `DropLab.prototype.init` -or `DropLab.prototype.addHook` call. - -- `InputSetter`: Requires a configuration value for `input` and `valueAttribute`. -- `input`: The DOM element that you want to manipulate. -- `valueAttribute`: A string that's the name of an attribute on your list items - that's used to get the value to update the `input` element with. - -You can also set the `InputSetter` configuration to an array of objects, which -allows you to update multiple elements. - -```html -<input id="input" value=""> -<div id="div" data-selected-id=""></div> - -<input href="#" id="trigger" data-dropdown-trigger="#list"> -<ul id="list" data-dropdown data-dynamic> - <li><a href="#" data-id="{{id}}">{{text}}</a></li> -<ul> -``` - -```javascript -const droplab = new DropLab(); - -const trigger = document.getElementById('trigger'); -const list = document.getElementById('list'); - -const input = document.getElementById('input'); -const div = document.getElementById('div'); - -droplab.init(trigger, list, [InputSetter], { - InputSetter: [{ - input: input, - valueAttribute: 'data-id', - } { - input: div, - valueAttribute: 'data-id', - inputAttribute: 'data-selected-id', - }], -}); - -droplab.addData('trigger', [{ - id: 0, - text: 'Jacob', -}, { - id: 1, - text: 'Jeff', -}]); -``` - -In the previous code, if the second list item was clicked, it would update the -`#input` element to have a `value` of `1`, it would also update the `#div` -element's `data-selected-id` to `1`. - -Optionally, you can set `inputAttribute` to a string that's the name of an -attribute on your `input` element that you want to update. If you don't provide -an `inputAttribute`, `InputSetter` updates the `value` of the `input` -element if it's an `INPUT` element, or the `textContent` of the `input` element -if it isn't an `INPUT` element. diff --git a/doc/development/fe_guide/editor_lite.md b/doc/development/fe_guide/editor_lite.md deleted file mode 100644 index 5020bf9eeeb..00000000000 --- a/doc/development/fe_guide/editor_lite.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'source_editor.md' -remove_date: '2021-09-19' ---- - -This document was moved to [another location](source_editor.md). - -<!-- This redirect file can be deleted after <2021-09-19>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index 8f501007755..f905fdad77e 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -57,7 +57,7 @@ For example: When using the GitLab UI form builder, the following components are available for use in HAML. NOTE: -Currently only `gitlab_ui_checkbox_component` is available but more components are planned. +Currently only the listed components are available but more components are planned. #### gitlab_ui_checkbox_component @@ -72,3 +72,16 @@ Currently only `gitlab_ui_checkbox_component` is available but more components a | `checked_value` | Value when checkbox is checked. | `String` | `false` (`'1'`) | | `unchecked_value` | Value when checkbox is unchecked. | `String` | `false` (`'0'`) | | `label_options` | Options that are passed to [Rails `label` method](https://api.rubyonrails.org/classes/ActionView/Helpers/FormBuilder.html#method-i-label). | `Hash` | `false` (`{}`) | + +#### gitlab_ui_radio_component + +[GitLab UI Docs](https://gitlab-org.gitlab.io/gitlab-ui/?path=/story/base-form-form-radio--default) + +| Argument | Description | Type | Required (default value) | +|---|---|---|---| +| `method` | Attribute on the object passed to `gitlab_ui_form_for`. | `Symbol` | `true` | +| `value` | The value of the radio tag. | `Symbol` | `true` | +| `label` | Radio label. | `String` | `true` | +| `help_text` | Help text displayed below the radio button. | `String` | `false` (`nil`) | +| `radio_options` | Options that are passed to [Rails `radio_button` method](https://api.rubyonrails.org/classes/ActionView/Helpers/FormBuilder.html#method-i-radio_button). | `Hash` | `false` (`{}`) | +| `label_options` | Options that are passed to [Rails `label` method](https://api.rubyonrails.org/classes/ActionView/Helpers/FormBuilder.html#method-i-label). | `Hash` | `false` (`{}`) | diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index a6b49394733..9ef4375d795 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -133,9 +133,13 @@ Best practices for monitoring and maximizing frontend performance. Frontend security practices. -## [Accessibility](accessibility.md) +## Accessibility -Our accessibility standards and resources. +Our [accessibility standards and resources](accessibility.md). + +## Logging + +Best practices for [client-side logging](logging.md) for GitLab frontend development. ## [Internationalization (i18n) and Translations](../i18n/externalization.md) diff --git a/doc/development/fe_guide/logging.md b/doc/development/fe_guide/logging.md new file mode 100644 index 00000000000..26633eade43 --- /dev/null +++ b/doc/development/fe_guide/logging.md @@ -0,0 +1,86 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Client-side logging for frontend development + +This guide contains the best practices for client-side logging for GitLab +frontend development. + +## When to log to the browser console + +We do not want to log unnecessarily to the browser console, as excessively +noisy console logs are not easy to read, parse, or process. We **do** want to +give visibility to unintended events in the system. If a possible but unexpected +exception occurs during runtime, we want to log the details of this exception. +These logs can give significantly helpful context to end users creating issues, or +contributors diagnosing problems. + +Whenever a `catch(e)` exists, and `e` is something unexpected, log the details. + +### What makes an error unexpected? + +Sometimes a caught exception can be part of normal operations. For instance, third-party +libraries might throw an exception based on certain inputs. If we can gracefully +handle these exceptions, then they are expected. Don't log them noisily. +For example: + +```javascript +try { + // Here, we call a method based on some user input. + // `doAThing` will throw an exception if the input is invalid. + const userInput = getUserInput(); + doAThing(userInput); +} catch (e) { + if (e instanceof FooSyntaxError) { + // To handle a `FooSyntaxError`, we just need to instruct the user to change their input. + // This isn't unexpected, and is part of normal operations. + setUserMessage(`Try writing better code. ${e.message}`); + } else { + // We're not sure what `e` is, so something unexpected and bad happened... + logError(e); + setUserMessage('Something unexpected happened...'); + } +} +``` + +## How to log an error + +We have a helpful `~/lib/logger` module which encapsulates how we can +consistently log runtime errors in GitLab. Import `logError` from this +module, and use it as you normally would `console.error`. Pass the actual `Error` +object, so the stack trace and other details can be captured in the log: + +```javascript +// 1. Import the logger module. +import { logError } from '~/lib/logger'; + +export const doThing = () => { + return foo() + .then(() => { + // ... + }) + .catch(e => { + // 2. Use `logError` like you would `console.error`. + logError('An unexpected error occurred while doing the thing', e); + + // We may or may not want to present that something bad happened to the end user. + showThingFailed(); + }); +}; +``` + +## Relation to frontend observability + +Client-side logging is strongly related to +[Frontend observability](https://about.gitlab.com/company/team/structure/working-groups/frontend-observability/). +We want unexpected errors to be observed by our monitoring systems, so +we can quickly react to user-facing issues. For a number of reasons, it is +unfeasible to send every log to the monitoring system. Don't shy away from using +`~/lib/logger`, but consider controlling which messages passed to `~/lib/logger` +are actually sent to the monitoring systems. + +A cohesive logging module helps us control these side effects consistently +across the various entry points. diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index 15225cc1deb..a46157d2cad 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -33,19 +33,25 @@ Stories can be added for any Vue component in the `gitlab` repository. To add a story: 1. Create a new `.stories.js` file in the same directory as the Vue component. - The file name should have the same prefix as the Vue component. + The filename should have the same prefix as the Vue component. ```txt vue_shared/ ├─ components/ │ ├─ sidebar - │ │ ├─ todo_button.vue - │ │ ├─ todo_button.stories.js + │ | ├─ todo_toggle + │ | | ├─ todo_button.vue + │ │ | ├─ todo_button.stories.js ``` 1. Write the story as per the [official Storybook instructions](https://storybook.js.org/docs/vue/writing-stories/introduction/) Notes: - Specify the `title` field of the story as the component's file path from the `javascripts/` directory, - e.g. if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the `title` as - `vue_shared/components/To-do Button`. This will ensure the Storybook navigation maps closely to our internal directory structure. + e.g. if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure. + +## Mock backend APIs + +GitLab’s Storybook uses [MirajeJS](https://miragejs.com/) to mock REST and GraphQL APIs. Storybook shares the MirajeJS server +with the [frontend integration tests](../testing_guide/testing_levels.md#frontend-integration-tests). You can find the MirajeJS +configuration files in `spec/frontend_integration/mock_server`. diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index 6d9bbdd3f2d..ffaaa3e87c7 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -45,7 +45,7 @@ result (such as `ml-1` becoming `gl-ml-2`). If a class you need has not been added to GitLab UI, you get to add it! Follow the naming patterns documented in the [utility files](https://gitlab.com/gitlab-org/gitlab-ui/-/tree/main/src/scss/utility-mixins) and refer to [GitLab UI's CSS documentation](https://gitlab.com/gitlab-org/gitlab-ui/-/blob/main/doc/contributing/adding_css.md#adding-utility-mixins) for more details, especially about adding responsive and stateful rules. -If it is not possible to wait for a GitLab UI update (generally one day), add the class to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/stylesheets/utilities.scss) following the same naming conventions documented in GitLab UI. A follow—up issue to backport the class to GitLab UI and delete it from GitLab should be opened. +If it is not possible to wait for a GitLab UI update (generally one day), add the class to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/stylesheets/utilities.scss) following the same naming conventions documented in GitLab UI. A follow-up issue to backport the class to GitLab UI and delete it from GitLab should be opened. #### When should I create component classes? diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 2f0f8101b53..20325facc75 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -72,6 +72,11 @@ class SomeCrossCuttingConcernWorker end ``` +When possible, workers marked as "not owned" use their caller's +category (worker or HTTP endpoint) in metrics and logs. +For instance, `ReactiveCachingWorker` can have multiple feature +categories in metrics and logs. + ## Rails controllers Specifying feature categories on controller actions can be done using diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 1962d5262ce..987ff7c9fe8 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -21,7 +21,7 @@ All newly-introduced feature flags should be [disabled by default](https://about NOTE: This document is the subject of continued work as part of an epic to [improve internal usage of Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). Raise any suggestions as new issues and attach them to the epic. -For an [overview of the feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), or if you need help deciding [if you should use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) or not, please see the [feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle) handbook page. +For an [overview of the feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle), or if you need help deciding [if you should use a feature flag](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags) or not, please see the [feature flag lifecycle](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/) handbook page. ## When to use feature flags diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 71fc81a6ea3..d161206f44d 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -28,6 +28,8 @@ There are many places where file uploading is used, according to contexts: - LFS Objects - Merge request diffs - Design Management design thumbnails +- Topic + - Topic avatars ## Disk storage @@ -42,6 +44,7 @@ they are still not 100% standardized. You can see them below: | User avatars | yes | `uploads/-/system/user/avatar/:id/:filename` | `AvatarUploader` | User | | User snippet attachments | yes | `uploads/-/system/personal_snippet/:id/:random_hex/:filename` | `PersonalFileUploader` | Snippet | | Project avatars | yes | `uploads/-/system/project/avatar/:id/:filename` | `AvatarUploader` | Project | +| Topic avatars | yes | `uploads/-/system/projects/topic/avatar/:id/:filename` | `AvatarUploader` | Topic | | Issues/MR/Notes Markdown attachments | yes | `uploads/:project_path_with_namespace/:random_hex/:filename` | `FileUploader` | Project | | Issues/MR/Notes Legacy Markdown attachments | no | `uploads/-/system/note/attachment/:id/:filename` | `AttachmentUploader` | Note | | Design Management design thumbnails | yes | `uploads/-/system/design_management/action/image_v432x230/:id/:filename` | `DesignManagement::DesignV432x230Uploader` | DesignManagement::Action | diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index 2b9c7efc087..6f9811f7e05 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -82,6 +82,19 @@ AND (EXISTS ( While this worked without schema changes, and did improve readability somewhat, it did not improve query performance. +### Attempt A2: use label IDs in the WHERE EXISTS clause + +In [merge request #34503](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34503), we followed a similar approach to A1. But this time, we +did a separate query to fetch the IDs of the labels used in the filter so that we avoid the `JOIN` in the `EXISTS` clause and filter directly by +`label_links.label_id`. We also added a new index on `label_links` for the `target_id`, `label_id`, and `target_type` columns to speed up this query. + +Finding the label IDs wasn't straightforward because there could be multiple labels with the same title within a single root namespace. We solved +this by grouping the label IDs by title and then using the array of IDs in the `EXISTS` clauses. + +This resulted in a significant performance improvement. However, this optimization could not be applied to the dashboard pages +where we do not have a project or group context. We could not easily search for the label IDs here because that would mean searching across all +projects and groups that the user has access to. + ## Attempt B: Denormalize using an array column Having [removed MySQL support in GitLab 12.1](https://about.gitlab.com/blog/2019/06/27/removing-mysql-support/), @@ -159,9 +172,8 @@ However, at present, the disadvantages outweigh the advantages. ## Conclusion -We have yet to find a method that is demonstrably better than the current -method, when considering: +We found a method A2 that does not need denormalization and improves the query performance significantly. This +did not apply to all cases, but we were able to apply method A1 to the rest of the cases so that we remove the +`GROUP BY` and `HAVING` clauses in all scenarios. -1. Query performance. -1. Readability. -1. Ease of maintaining schema consistency. +This simplified the query and improved the performance in the most common cases. diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md new file mode 100644 index 00000000000..75a093a55ac --- /dev/null +++ b/doc/development/go_guide/go_upgrade.md @@ -0,0 +1,187 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Managing Go versions + +## Overview + +All Go binaries, with the exception of +[GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) and [Security Projects](https://gitlab.com/gitlab-org/security-products), are built in +projects managed by the [Distribution team](https://about.gitlab.com/handbook/product/categories/#distribution-group). + +The [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) project creates a +single, monolithic operating system package containing all the binaries, while +the [Cloud-Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG) project +publishes a set of Docker images deployed and configured by Helm Charts or +the GitLab Operator. + +Testing matrices for all projects using Go must include the version shipped +by Distribution: + +- [Check the Go version shipping with Omnibus GitLab](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/master/docker/VERSIONS#L6). +- [Check the Go version shipping with Cloud-Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/cng/blob/master/ci_files/variables.yml#L12). + +## Supporting multiple Go versions + +Individual Golang projects need to support multiple Go versions because: + +- When a new version of Go is released, we should start integrating it into the CI pipelines to verify compatibility with the new compiler. +- We must support the [official Omnibus GitLab Go version](#updating-go-version), which may be behind the latest minor release. +- When Omnibus switches Go version, we still may need to support the old one for security backports. + +These 3 requirements may easily be satisfied by keeping support for the [3 latest minor versions of Go](https://golang.org/dl/). + +It is ok to drop support for the oldest Go version and support only the 2 latest releases, +if this is enough to support backports to the last 3 minor GitLab releases. + +For example, if we want to drop support for `go 1.11` in GitLab `12.10`, we need +to verify which Go versions we are using in `12.9`, `12.8`, and `12.7`. We do not +consider the active milestone, `12.10`, because a backport for `12.7` is required +in case of a critical security release. + +- If both [Omnibus GitLab and Cloud-Native GitLab (CNG)](#updating-go-version) were using Go `1.12` in GitLab `12.7` and later, + then we can safely drop support for `1.11`. +- If Omnibus GitLab or Cloud-Native GitLab (CNG) were using `1.11` in GitLab `12.7`, then we still need to keep + support for Go `1.11` for easier backporting of security fixes. + +## Updating Go version + +We should always: + +- Use the same Go version for Omnibus GitLab and Cloud Native GitLab. +- Use a [supported version](https://golang.org/doc/devel/release#policy). +- Use the most recent patch-level for that version to keep up with security fixes. + +Changing the version affects every project being compiled, so it's important to +ensure that all projects have been updated to test against the new Go version +before changing the package builders to use it. Despite [Go's compatibility promise](https://golang.org/doc/go1compat), +changes between minor versions can expose bugs or cause problems in our projects. + +### Upgrade process + +The upgrade process involves several key steps: + +- [Track component updates and validation](#tracking-work). +- [Track component integration for release](#tracking-work). +- [Communication with stakeholders](#communication-plan). + +#### Tracking work + +Use [the product categories page](https://about.gitlab.com/handbook/product/categories/) +if you need help finding the correct person or labels: + +1. Create the epic in `gitlab-org` group: + - Title the epic `Update Go version to <VERSION_NUMBER>`. + - Ping the engineering managers responsible for [the projects listed below](#known-dependencies-using-go). + +1. Create an upgrade issue for each dependency in the [location indicated below](#known-dependencies-using-go) + titled `Support building with Go <VERSION_NUMBER>`. Add the proper label to each issue for easier triage. + + NOTE: + The upgrade issues must include [upgrade validation items](#upgrade-validation) + in their definition of done. Creating a second [performance testing issue](#upgrade-validation) + titled `Validate operation and performance at scale with Go <VERSION_NUMBER>` + is strongly recommended to help with scheduling tasks and managing workloads. + +1. Schedule an update with the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues): + - Title the issue `Support using Go version <VERSION_NUMBER>`. + - Set the issue as related to every issue created in the previous step. +1. Schedule one issue per Secure Stage team and add the `devops::secure` label to each: + - [Static Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). + - [Composition Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). + - [Container Security tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). +1. Schedule builder updates with Distribution projects: + - Dependency and GitLab Development Kit issues created in previous steps should be set as blockers. + - Each issue should have the title `Support building with Go <VERSION_NUMBER>` and description as noted: + - [Cloud-Native GitLab](https://gitlab.com/gitlab-org/charts/gitlab/-/issues) + + ```plaintext + Update the `GO_VERSION` in `ci_files/variables.yml`. + ``` + + - [Omnibus GitLab Builder](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/issues) + + ```plaintext + Update `GO_VERSION` in `docker/VERSIONS`. + ``` + + - [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues) + + ```plaintext + Update `BUILDER_IMAGE_REVISION` in `.gitlab-ci.yml` to match tag from builder. + ``` + + NOTE: + If the component is not automatically upgraded for [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues) + and [Cloud Native GitLab](https://gitlab.com/gitlab-org/charts/gitlab/-/issues), + issues should be opened in their respective trackers titled `Updated bundled version of COMPONENT_NAME` + and set as blocked by the component's upgrade issue. + +#### Known dependencies using Go + +| Component Name | Where to track work | +|-------------------------------|---------------------| +| [Alertmanager](https://github.com/prometheus/alertmanager) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| Docker Distribution Pruner | [Issue Tracker](https://gitlab.com/gitlab-org/docker-distribution-pruner) | +| Gitaly | [Issue Tracker](https://gitlab.com/gitlab-org/gitaly/-/issues) | +| GitLab Compose Kit | [Issuer Tracker](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/issues) | +| GitLab Container Registry | [Issue Tracker](https://gitlab.com/gitlab-org/container-registry) | +| GitLab Elasticsearch Indexer | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer/-/issues) | +| GitLab Kubernetes Agent (KAS) | [Issue Tracker](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues) | +| GitLab Pages | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-pages/-/issues) | +| GitLab Quality Images | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-build-images/-/issues) | +| GitLab Shell | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab-shell/-/issues) | +| GitLab Workhorse | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| Labkit | [Issue Tracker](https://gitlab.com/gitlab-org/labkit/-/issues) | +| [Node Exporter](https://github.com/prometheus/node_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| [PgBouncer Exporter](https://github.com/prometheus-community/pgbouncer_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| [Postgres Exporter](https://github.com/prometheus-community/postgres_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| [Prometheus](https://github.com/prometheus/prometheus) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | +| [Redis Exporter](https://github.com/oliver006/redis_exporter) | [Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues) | + +#### Communication plan + +Communication is required at several key points throughout the process and should +be included in the relevant issues as part of the definition of done: + +1. Immediately after creating the epic, it should be posted to Slack. Community members must ask the pinged engineering managers for assistance with this step. The responsible GitLab team member should share a link to the epic in the following Slack channels: + - `#backend` + - `#development` +1. Immediately after merging the GitLab Development Kit Update, the same maintainer should add an entry to the engineering week-in-review sync and + announce the change in the following Slack channels: + - `#backend` + - `#development` +1. Immediately upon merge of the updated Go versions in + [Cloud-Native GitLab](https://gitlab.com/gitlab-org/build/CNG) and + [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) add the + change to the engineering-week-in-review sync and announce in the following + Slack channels: + - `#backend` + - `#development` + - `#releases` + +#### Upgrade validation + +Upstream component maintainers must validate their Go-based projects using: + +- Established unit tests in the codebase. +- Procedures established in [Merge Request Performance Guidelines](../merge_request_performance_guidelines.md). +- Procedures established in [Performance, Reliability, and Availability guidelines](../code_review.md#performance-reliability-and-availability). + +Upstream component maintainers should consider validating their Go-based +projects with: + +- Isolated component operation performance tests. + + Integration tests are costly and should be testing inter-component + operational issues. Isolated component testing reduces mean time to + feedback on updates and decreases resource burn across the organization. + +- Components should have end-to-end test coverage in the GitLab Performance Test tool. +- Integration validation through installation of fresh packages **_and_** upgrade from previous versions for: + - Single GitLab Node + - Reference Architecture Deployment + - Geo Deployment diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 0ee73da48db..44f8924be04 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -23,6 +23,16 @@ experiences. Several projects were started with different standards and they can still have specifics. They are described in their respective `README.md` or `PROCESS.md` files. +## Go language versions + +The Go upgrade documentation [provides an overview](go_upgrade.md#overview) +of how GitLab manages and ships Go binary support. + +If a GitLab component requires a newer version of Go, please +follow the [upgrade process](go_upgrade.md#updating-go-version) to ensure no customer, team, or component is adversely impacted. + +Sometimes, individual projects must also [manage builds with multiple versions of Go](go_upgrade.md#supporting-multiple-go-versions). + ## Dependency Management Go uses a source-based strategy for dependency management. Dependencies are @@ -417,70 +427,6 @@ Generated Docker images should have the program at their `Entrypoint` to create portable commands. That way, anyone can run the image, and without parameters it displays its help message (if `cli` has been used). -## Distributing Go binaries - -With the exception of [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner), -which publishes its own binaries, our Go binaries are created by projects -managed by the [Distribution group](https://about.gitlab.com/handbook/product/categories/#distribution-group). - -The [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) project creates a -single, monolithic operating system package containing all the binaries, while -the [Cloud-Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG) project -publishes a set of Docker images and Helm charts to glue them together. - -Both approaches use the same version of Go for all projects, so it's important -to ensure all our Go-using projects have at least one Go version in common in -their test matrices. You can check the version of Go currently being used by -[Omnibus](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/blob/master/docker/Dockerfile_debian_10#L59), -and the version being used for [CNG](https://gitlab.com/gitlab-org/build/cng/blob/master/ci_files/variables.yml#L12). - -### Updating Go version - -We should always use a [supported version](https://golang.org/doc/devel/release#policy) -of Go, that is, one of the three most recent minor releases, and should always use -the most recent patch-level for that version, as it may contain security fixes. - -Changing the version affects every project being compiled, so it's important to -ensure that all projects have been updated to test against the new Go version -before changing the package builders to use it. Despite [Go's compatibility promise](https://golang.org/doc/go1compat), -changes between minor versions can expose bugs or cause problems in our projects. - -Once you've picked a new Go version to use, the steps to update Omnibus and CNG -are: - -- [Create a merge request in the CNG project](https://gitlab.com/gitlab-org/build/CNG/-/edit/master/ci_files/variables.yml?branch_name=update-go-version), - update the `GO_VERSION` in `ci_files/variables.yml`. -- [Create a merge request in the `gitlab-omnibus-builder` project](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/edit/master/docker/VERSIONS?branch_name=update-go-version), - update the `GO_VERSION` in `docker/VERSIONS`. -- Tag a new release of `gitlab-omnibus-builder` containing the change. -- [Create a merge request in the `omnibus-gitlab` project](https://gitlab.com/gitlab-org/omnibus-gitlab/edit/master/.gitlab-ci.yml?branch_name=update-gitlab-omnibus-builder-version), - update the `BUILDER_IMAGE_REVISION` to match the newly-created tag. - -To reduce unnecessary differences between two distribution methods, Omnibus and -CNG **should always use the same Go version**. - -### Supporting multiple Go versions - -Individual Golang-projects need to support multiple Go versions for the following reasons: - -1. When a new Go release is out, we should start integrating it into the CI pipelines to verify compatibility with the new compiler. -1. We must support the [Omnibus official Go version](#updating-go-version), which may be behind the latest minor release. -1. When Omnibus switches Go version, we still may need to support the old one for security backports. - -These 3 requirements may easily be satisfied by keeping support for the 3 latest minor versions of Go. - -It's ok to drop support for the oldest Go version and support only 2 latest releases, -if this is enough to support backports to the last 3 GitLab minor releases. - -Example: - -In case we want to drop support for `go 1.11` in GitLab `12.10`, we need to verify which Go versions we are using in `12.9`, `12.8`, and `12.7`. - -We do not consider the active milestone, `12.10`, because a backport for `12.7` is required in case of a critical security release. - -1. If both [Omnibus and CNG](#updating-go-version) were using Go `1.12` in GitLab `12.7` and later, then we safely drop support for `1.11`. -1. If Omnibus or CNG were using `1.11` in GitLab `12.7`, then we still need to keep support for Go `1.11` for easier backporting of security fixes. - ## Secure Team standards and style guidelines The following are some style guidelines that are specific to the Secure Team. diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index a37c47f1b11..1f40a605cfe 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -338,9 +338,9 @@ describe 'sorting and pagination' do let(:ordered_issues) { issues.sort_by(&:weight) } it_behaves_like 'sorted paginated query' do - let(:sort_param) { :WEIGHT_ASC } - let(:first_param) { 2 } - let(:expected_results) { ordered_issues.map(&:iid) } + let(:sort_param) { :WEIGHT_ASC } + let(:first_param) { 2 } + let(:all_records) { ordered_issues.map(&:iid) } end end end diff --git a/doc/development/image_scaling.md b/doc/development/image_scaling.md index 79687b66711..82ca8cf8e83 100644 --- a/doc/development/image_scaling.md +++ b/doc/development/image_scaling.md @@ -31,8 +31,8 @@ The hard-coded rules only permit: Furthermore, configuration in Workhorse can lead to the image scaler rejecting a request if: -- The image file is too large (controlled by [`max_filesize`](- we only rescale images that do not exceed a configured size in bytes (see [`max_filesize`](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/67ab3a2985d2097392f93523ae1cffe0dbf01b31/config.toml.example#L17)))). -- Too many image scalers are already running (controlled by [`max_scaler_procs`](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/67ab3a2985d2097392f93523ae1cffe0dbf01b31/config.toml.example#L16)). +- The image file is too large (controlled by [`max_filesize`](- we only rescale images that do not exceed a configured size in bytes (see [`max_filesize`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/config.toml.example#L22)))). +- Too many image scalers are already running (controlled by [`max_scaler_procs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/config.toml.example#L21)). For instance, here are two different URLs that serve the GitLab project avatar both in its original size and scaled down to 64 pixels. Only the second request will trigger the image scaler: @@ -73,7 +73,7 @@ we simply follow the path we take to serve any ordinary upload. ### Workhorse Assuming Rails decided the request to be valid, Workhorse will take over. Upon receiving the `send-scaled-image` -instruction through the Rails response, a [special response injector](https://gitlab.com/gitlab-org/gitlab-workhorse/-/blob/master/internal/imageresizer/image_resizer.go) +instruction through the Rails response, a [special response injector](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/internal/imageresizer/image_resizer.go) will be invoked that knows how to rescale images. The only inputs it requires are the location of the image (a path if the image resides in block storage, or a URL to remote storage otherwise) and the desired width. Workhorse will handle the location transparently so Rails does not need to be concerned with where the image diff --git a/doc/development/import_project.md b/doc/development/import_project.md index d021126c8eb..9872aa239dc 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -111,9 +111,9 @@ public folder (for example `/tmp/`) fixes the issue. ##### `Name can contain only letters, digits, emojis ...` ```plaintext -Name can contain only letters, digits, emojis, '_', '.', dash, space. It must start with letter, -digit, emoji or '_'. and Path can contain only letters, digits, '_', '-' and '.'. Cannot start -with '-', end in '.git' or end in '.atom' +Name can contain only letters, digits, emojis, '_', '.', '+', dashes, or spaces. It must start with a letter, +digit, emoji, or '_', and Path can contain only letters, digits, '_', '-', or '.'. It cannot start +with '-', end in '.git', or end in '.atom'. ``` The project name specified in `project_path` is not valid for one of the specified reasons. @@ -216,6 +216,6 @@ This is due to a [n+1 calls limit being set for development setups](gitaly.md#to Many of the tests also require a GitLab Personal Access Token. This is due to numerous endpoints themselves requiring authentication. -[The official GitLab docs detail how to create this token](../user/profile/personal_access_tokens.md#create-a-personal-access-token). The tests require that the token is generated by an admin user and that it has the `API` and `read_repository` permissions. +[The official GitLab docs detail how to create this token](../user/profile/personal_access_tokens.md#create-a-personal-access-token). The tests require that the token is generated by an administrator and that it has the `API` and `read_repository` permissions. Details on how to use the Access Token with each type of test are found in their respective documentation. diff --git a/doc/development/index.md b/doc/development/index.md index e8e7369f6c5..3780c986367 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -18,7 +18,7 @@ the [Handbook](https://about.gitlab.com/handbook/). For information on using GitLab to work on your own software projects, see the [GitLab user documentation](../user/index.md). -For information on working with the GitLab APIs, see the [API documentation](../api/README.md). +For information on working with the GitLab APIs, see the [API documentation](../api/index.md). For information about how to install, configure, update, and upgrade your own GitLab instance, see the [administration documentation](../administration/index.md). @@ -173,6 +173,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Windows Development on GCP](windows.md) - [FIPS compliance](fips_compliance.md) - [`Gemfile` guidelines](gemfile.md) +- [Ruby upgrade guidelines](ruby_upgrade.md) ### Things to be aware of @@ -189,6 +190,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Issuable-like Rails models](issuable-like-models.md) - [Issue types vs first-class types](issue_types.md) - [DeclarativePolicy framework](policies.md) +- [Rails update guidelines](rails_update.md) ### Debugging @@ -215,6 +217,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [How to dump production data to staging](db_dump.md) - [Geo development](geo.md) - [Redis guidelines](redis.md) + - [Adding a new Redis instance](redis/new_redis_instance.md) - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Elasticsearch integration docs](elasticsearch.md) @@ -253,8 +256,6 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev ## Performance guides -- [Instrumentation](instrumentation.md) for Ruby code running in production - environments. - [Performance guidelines](performance.md) for writing code, benchmarks, and certain patterns to avoid. - [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment. @@ -334,6 +335,7 @@ See [database guidelines](database/index.md). - [Features inside `.gitlab/`](features_inside_dot_gitlab.md) - [Dashboards for stage groups](stage_group_dashboards.md) - [Preventing transient bugs](transient/prevention-patterns.md) +- [GitLab Application SLIs](application_slis/index.md) ## Other GitLab Development Kit (GDK) guides diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md deleted file mode 100644 index 83e7444bb1f..00000000000 --- a/doc/development/instrumentation.md +++ /dev/null @@ -1,161 +0,0 @@ ---- -stage: Monitor -group: Monitor -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Instrumenting Ruby code **(FREE)** - -[GitLab Performance Monitoring](../administration/monitoring/performance/index.md) allows instrumenting of both methods and custom -blocks of Ruby code. Method instrumentation is the primary form of -instrumentation with block-based instrumentation only being used when we want to -drill down to specific regions of code within a method. - -Please refer to [Product Intelligence](https://about.gitlab.com/handbook/product/product-intelligence-guide/) if you are tracking product usage patterns. - -## Instrumenting Methods - -Instrumenting methods is done by using the `Gitlab::Metrics::Instrumentation` -module. This module offers a few different methods that can be used to -instrument code: - -- `instrument_method`: Instruments a single class method. -- `instrument_instance_method`: Instruments a single instance method. -- `instrument_class_hierarchy`: Given a Class, this method recursively - instruments all sub-classes (both class and instance methods). -- `instrument_methods`: Instruments all public and private class methods of a - Module. -- `instrument_instance_methods`: Instruments all public and private instance - methods of a Module. - -To remove the need for typing the full `Gitlab::Metrics::Instrumentation` -namespace you can use the `configure` class method. This method simply yields -the supplied block while passing `Gitlab::Metrics::Instrumentation` as its -argument. An example: - -```ruby -Gitlab::Metrics::Instrumentation.configure do |conf| - conf.instrument_method(Foo, :bar) - conf.instrument_method(Foo, :baz) -end -``` - -Using this method is in general preferred over directly calling the various -instrumentation methods. - -Method instrumentation should be added in the initializer -`config/initializers/zz_metrics.rb`. - -### Examples - -Instrumenting a single method: - -```ruby -Gitlab::Metrics::Instrumentation.configure do |conf| - conf.instrument_method(User, :find_by) -end -``` - -Instrumenting an entire class hierarchy: - -```ruby -Gitlab::Metrics::Instrumentation.configure do |conf| - conf.instrument_class_hierarchy(ActiveRecord::Base) -end -``` - -Instrumenting all public class methods: - -```ruby -Gitlab::Metrics::Instrumentation.configure do |conf| - conf.instrument_methods(User) -end -``` - -### Checking Instrumented Methods - -The easiest way to check if a method has been instrumented is to check its -source location. For example: - -```ruby -method = Banzai::Renderer.method(:render) - -method.source_location -``` - -If the source location points to `lib/gitlab/metrics/instrumentation.rb` you -know the method has been instrumented. - -If you're using Pry you can use the `$` command to display the source code of a -method (along with its source location), this is easier than running the above -Ruby code. In case of the above snippet you'd run the following: - -- `$ Banzai::Renderer.render` - -This prints a result similar to: - -```plaintext -From: /path/to/your/gitlab/lib/gitlab/metrics/instrumentation.rb @ line 148: -Owner: #<Module:0x0055f0865c6d50> -Visibility: public -Number of lines: 21 - -def #{name}(#{args_signature}) - if trans = Gitlab::Metrics::Instrumentation.transaction - trans.measure_method(#{label.inspect}) { super } - else - super - end -end -``` - -## Instrumenting Ruby Blocks - -Measuring blocks of Ruby code is done by calling `Gitlab::Metrics.measure` and -passing it a block. For example: - -```ruby -Gitlab::Metrics.measure(:foo) do - ... -end -``` - -The block is executed and the execution time is stored as a set of fields in the -currently running transaction. If no transaction is present the block is yielded -without measuring anything. - -Three values are measured for a block: - -- The real time elapsed, stored in `NAME_real_time`. -- The CPU time elapsed, stored in `NAME_cpu_time`. -- The call count, stored in `NAME_call_count`. - -Both the real and CPU timings are measured in milliseconds. - -Multiple calls to the same block results in the final values being the sum -of all individual values. Take this code for example: - -```ruby -3.times do - Gitlab::Metrics.measure(:sleep) do - sleep 1 - end -end -``` - -Here, the final value of `sleep_real_time` is `3`, and not `1`. - -## Tracking Custom Events - -Besides instrumenting code GitLab Performance Monitoring also supports tracking -of custom events. This is primarily intended to be used for tracking business -metrics such as the number of Git pushes, repository imports, and so on. - -To track a custom event simply call `Gitlab::Metrics.add_event` passing it an -event name and a custom set of (optional) tags. For example: - -```ruby -Gitlab::Metrics.add_event(:user_login, email: current_user.email) -``` - -Event names should be verbs such as `push_repository` and `remove_branch`. diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index d37ce29e353..34293845d17 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -534,15 +534,24 @@ affecting version `2.50.3-2+deb9u1` of Debian package `glib2.0`: }, "version": "2.50.3-2+deb9u1", "operating_system": "debian:9", - "image": "index.docker.io/library/nginx:1.18" + "image": "index.docker.io/library/nginx:1.18", + "kubernetes_resource": { + "namespace": "production", + "kind": "Deployment", + "name": "nginx-ingress", + "container_name": "nginx", + "agent_id": "1" + } } ``` -The affected package is found when scanning the image of the pod `index.docker.io/library/nginx:1.18`. +The affected package is found when scanning a deployment using the `index.docker.io/library/nginx:1.18` image. The location fingerprint of a Cluster Image Scanning vulnerability combines the -`operating_system` and the package `name`, so these attributes are mandatory. The `image` is also -mandatory. All other attributes are optional. +`namespace`, `kind`, `name`, and `container_name` fields from the `kubernetes_resource`, +as well as the package `name`, so these fields are required. The `image` field is also mandatory. +The `cluster_id` and `agent_id` are mutually exclusive, and one of them must be present. +All other fields are optional. #### SAST diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 9e67227ec7f..45c94019c63 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Kubernetes integration - development guidelines **(FREE)** This document provides various guidelines when developing for the GitLab -[Kubernetes integration](../user/project/clusters/index.md). +[Kubernetes integration](../user/infrastructure/clusters/index.md). ## Development diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index d87b7bcb5af..cbf3c09b28b 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -377,16 +377,6 @@ comment. Instead of always rendering these kind of elements they should only be rendered when actually needed. This ensures we don't spend time generating Haml/HTML when it's not used. -## Instrumenting New Code - -**Summary:** always add instrumentation for new classes, modules, and methods. - -Newly added classes, modules, and methods must be instrumented. This ensures -we can track the performance of this code over time. - -For more information see [Instrumentation](instrumentation.md). This guide -describes how to add instrumentation and where to add it. - ## Use of Caching **Summary:** cache data in memory or in Redis when it's needed multiple times in diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index ce564551fbf..e03b96a0e14 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -879,7 +879,7 @@ See the [text data type](database/strings_and_the_text_data_type.md) style guide ## Timestamp column type By default, Rails uses the `timestamp` data type that stores timestamp data -without timezone information. The `timestamp` data type is used by calling +without time zone information. The `timestamp` data type is used by calling either the `add_timestamps` or the `timestamps` method. Also, Rails converts the `:datetime` data type to the `timestamp` one. @@ -904,15 +904,15 @@ end ``` Instead of using these methods, one should use the following methods to store -timestamps with timezones: +timestamps with time zones: - `add_timestamps_with_timezone` - `timestamps_with_timezone` - `datetime_with_timezone` This ensures all timestamps have a time zone specified. This, in turn, means -existing timestamps don't suddenly use a different timezone when the system's -timezone changes. It also makes it very clear which timezone was used in the +existing timestamps don't suddenly use a different time zone when the system's +time zone changes. It also makes it very clear which time zone was used in the first place. ## Storing JSON in database diff --git a/doc/development/packages.md b/doc/development/packages.md index 869a1755d8f..38c1b941eaf 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -30,9 +30,9 @@ The existing database model requires the following: ### API endpoints -Package systems work with GitLab via API. For example `lib/api/npm_packages.rb` +Package systems work with GitLab via API. For example `lib/api/npm_project_packages.rb` implements API endpoints to work with npm clients. So, the first thing to do is to -add a new `lib/api/your_name_packages.rb` file with API endpoints that are +add a new `lib/api/your_name_project_packages.rb` file with API endpoints that are necessary to make the package system client to work. Usually that means having endpoints like: @@ -48,7 +48,7 @@ GET https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ PUT https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ ``` -Group-level and instance-level endpoints are good to have but are optional. +Group-level and instance-level endpoints should only be considered after the project-level endpoint is available in production. #### Remote hierarchy @@ -168,7 +168,7 @@ The implementation of the different Merge Requests varies between different pack The MVC must support [Personal Access Tokens](../user/profile/personal_access_tokens.md) right from the start. We currently support two options for these tokens: OAuth and Basic Access. -OAuth authentication is already supported. You can see an example in the [npm API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/npm_packages.rb). +OAuth authentication is already supported. You can see an example in the [npm API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/npm_project_packages.rb). [Basic Access authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication) support is done by overriding a specific function in the API helpers, like diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index dd45091a31b..45982d6075b 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pipelines for the GitLab project -Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) and [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss) (as well as the -`dev` instance's mirrors) are configured in the usual +Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) (as well as the `dev` instance's) is configured in the usual [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml) which itself includes files under [`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) @@ -17,29 +16,206 @@ We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfo GitLab [CI/CD features and best-practices](../ci/yaml/index.md) as much as possible. -## Overview +## Minimal test jobs before a merge request is approved -Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../ci/yaml/index.md#workflow) -feature of the GitLab CI/CD. +**To reduce the pipeline cost and shorten the job duration, before a merge request is approved, the pipeline will run a minimal set of RSpec & Jest tests that are related to the merge request changes.** -Pipelines are always created for the following scenarios: +After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests +have been run before a merge request is merged. -- `main` branch, including on schedules, pushes, merges, and so on. -- Merge requests. -- Tags. -- Stable, `auto-deploy`, and security branches. +### RSpec minimal jobs -Pipeline creation is also affected by the following CI/CD variables: +#### Determining related RSpec test files in a merge request -- If `$FORCE_GITLAB_CI` is set, pipelines are created. -- If `$GITLAB_INTERNAL` is not set, pipelines are not created. +To identify the minimal set of tests needed, we use the [`test_file_finder` gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder), with two strategies: -No pipeline is created in any other cases (for example, when pushing a branch with no -MR for it). +- dynamic mapping from test coverage tracing (generated via the [Crystalball gem](https://github.com/toptal/crystalball)) + ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L15)) +- static mapping maintained in the [`tests.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/tests.yml) for special cases that cannot + be mapped via coverage tracing ([see where it's used](https://gitlab.com/gitlab-org/gitlab/-/blob/47d507c93779675d73a05002e2ec9c3c467cd698/tooling/bin/find_tests#L12)) -The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). +The test mappings contain a map of each source files to a list of test files which is dependent of the source file. + +In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current merge request. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full RSpec tests: + +- when the `pipeline:run-all-rspec` label is set on the merge request +- when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +### Jest minimal jobs + +#### Determining related Jest test files in a merge request + +To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. +In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. + +#### Exceptional cases + +In addition, there are a few circumstances where we would always run the full Jest tests: + +- when the `pipeline:run-all-rspec` label is set on the merge request +- when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) +- when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) +- when any vendored JavaScript file is changed (i.e. `vendor/assets/javascripts/**/*`) +- when any backend file is changed ([see the patterns list for details](https://gitlab.com/gitlab-org/gitlab/-/blob/3616946936c1adbd9e754c1bd06f86ba670796d8/.gitlab/ci/rules.gitlab-ci.yml#L205-216)) + +## Fail-fast job in merge request pipelines + +To provide faster feedback when a merge request breaks existing tests, we are experimenting with a +fail-fast mechanism. + +An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a merge +request pipeline. This job runs the tests that are directly related to the changes +in the merge request. + +If any of these tests fail, the `rspec fail-fast` job fails, triggering a +`fail-pipeline-early` job to run. The `fail-pipeline-early` job: + +- Cancels the currently running pipeline and all in-progress jobs. +- Sets pipeline to have status `failed`. + +For example: + +```mermaid +graph LR + subgraph "prepare stage"; + A["detect-tests"] + end + + subgraph "test stage"; + B["jest"]; + C["rspec migration"]; + D["rspec unit"]; + E["rspec integration"]; + F["rspec system"]; + G["rspec fail-fast"]; + end + + subgraph "post-test stage"; + Z["fail-pipeline-early"]; + end + + A --"artifact: list of test files"--> G + G --"on failure"--> Z +``` + +The `rspec fail-fast` is a no-op if there are more than 10 test files related to the +merge request. This prevents `rspec fail-fast` duration from exceeding the average +`rspec` job duration and defeating its purpose. + +This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. + +## Test jobs + +We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the +changes made in your merge request. +If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. + +WARNING: +Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors + +### Test suite parallelization + +Our current RSpec tests parallelization setup is as follows: + +1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a + `knapsack/report-master.json` file: + - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` + (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. +1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with + `knapsack rspec` and should have an evenly distributed share of tests: + - It works because the jobs have access to the `knapsack/report-master.json` + since the "artifacts from all previous stages are passed by default". + - the jobs set their own report path to + `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`. + - if knapsack is doing its job, test files that are run should be listed under + `Report specs`, not under `Leftover specs`. +1. The `update-tests-metadata` job (which only runs on scheduled pipelines for + [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the + `knapsack/rspec*_pg_*.json` files and merge them all together into a single + `knapsack/report-master.json` file that is saved as artifact. + +After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. + +### Monitoring + +The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch +that includes `rspec-profile` in their name. + +### Logging + +- Rails logging to `log/test.log` is disabled by default in CI [for + performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). To override this setting, provide the + `RAILS_ENABLE_TEST_LOG` environment variable. + +## Review app jobs + +Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. + +## As-if-FOSS jobs + +The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context +of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: + +- when the `pipeline:run-as-if-foss` label is set on the merge request +- when the merge request is created in the `gitlab-org/security/gitlab` project +- when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) + +The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable +set and get the `ee/` folder removed before the tests start running. + +The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to +the `gitlab-org/gitlab-foss` project. + +## As-if-JH jobs + +The `* as-if-jh` jobs run the GitLab test suite "as if JiHu", meaning as if the jobs would run in the context +of [the `gitlab-jh/gitlab` project](jh_features_review.md). These jobs are only created in the following cases: + +- when the `pipeline:run-as-if-jh` label is set on the merge request +- when the `pipeline:run-all-rspec` label is set on the merge request +- when any code or backstage file is changed +- when any startup CSS file is changed + +The `* as-if-jh` jobs are run in addition to the regular EE-context jobs. The `jh/` folder is added before the tests start running. + +The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to +the `gitlab-jh/gitlab` project. + +## PostgreSQL versions testing + +Our test suite runs against PG12 as GitLab.com runs on PG12 and +[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md). + +We do run our test suite against PG11 and PG13 on nightly scheduled pipelines. + +We also run our test suite against PG11 upon specific database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). + +### Current versions testing + +| Where? | PostgreSQL version | +| ------ | ------------------ | +| MRs | 12, 11 for DB library changes | +| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | +| 2-hourly scheduled pipelines | 12, 11 for DB library changes | +| `nightly` scheduled pipelines | 12, 11, 13 | + +### Long-term plan + +We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): + +| PostgreSQL version | 14.1 (July 2021) | 14.2 (August 2021) | 14.3 (September 2021) | 14.4 (October 2021) | 14.5 (November 2021) | 14.6 (December 2021) | +| -------------------| ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | ---------------------- | +| PG12 | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | +| PG11 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | +| PG13 | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | `nightly` | -### Pipelines for Merge Requests +## Pipelines types for merge requests In general, pipelines for an MR fall into one or more of the following types, depending on the changes made in the MR: @@ -53,7 +229,7 @@ We use the [`rules:`](../ci/yaml/index.md#rules) and [`needs:`](../ci/yaml/index to determine the jobs that need to be run in a pipeline. Note that an MR that includes multiple types of changes would have a pipelines that include jobs from multiple types (for example, a combination of docs-only and code-only pipelines). -#### Documentation only MR pipeline +### Documentation only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/-/pipelines/250546928): @@ -71,7 +247,7 @@ graph LR end ``` -#### Code-only MR pipeline +### Code-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/136295694) @@ -102,7 +278,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -123,7 +298,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; + 2_2-2["rspec-all frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -155,7 +330,7 @@ graph RL; 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; + subgraph "Needs `rspec-all frontend_fixture`"; 3_1-1 --> 2_2-2; end @@ -173,7 +348,7 @@ graph RL; end ``` -#### Frontend-only MR pipeline +### Frontend-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134661039): @@ -204,7 +379,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -226,7 +400,7 @@ graph RL; 2_1-1 & 2_1-2 & 2_1-3 & 2_1-4 --> 1-6; end - 2_2-2["rspec frontend_fixture/rspec-ee frontend_fixture (7 minutes)"]; + 2_2-2["rspec-all frontend_fixture (7 minutes)"]; class 2_2-2 criticalPath; click 2_2-2 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=7910143&udv=0" 2_2-4["memory-on-boot (3.5 minutes)"]; @@ -266,7 +440,7 @@ graph RL; 3_1-1["jest (14.5 minutes)"]; class 3_1-1 criticalPath; click 3_1-1 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914204&udv=0" - subgraph "Needs `rspec frontend_fixture/rspec-ee frontend_fixture`"; + subgraph "Needs `rspec-all frontend_fixture`"; 3_1-1 --> 2_2-2; end @@ -299,7 +473,7 @@ graph RL; end ``` -#### QA-only MR pipeline +### QA-only MR pipeline [Reference pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/134645109): @@ -330,7 +504,6 @@ graph RL; 1-16["brakeman-sast"]; 1-17["eslint-sast"]; 1-18["kubesec-sast"]; - 1-19["nodejs-scan-sast"]; 1-20["secrets-sast"]; 1-21["static-analysis (14 minutes)"]; click 1-21 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=6914471&udv=0" @@ -358,261 +531,53 @@ graph RL; end ``` -### Fail-fast pipeline in Merge Requests - -To provide faster feedback when a Merge Request breaks existing tests, we are experimenting with a -fail-fast mechanism. - -An `rspec fail-fast` job is added in parallel to all other `rspec` jobs in a Merge -Request pipeline. This job runs the tests that are directly related to the changes -in the Merge Request. - -If any of these tests fail, the `rspec fail-fast` job fails, triggering a -`fail-pipeline-early` job to run. The `fail-pipeline-early` job: - -- Cancels the currently running pipeline and all in-progress jobs. -- Sets pipeline to have status `failed`. - -For example: - -```mermaid -graph LR - subgraph "prepare stage"; - A["detect-tests"] - end - - subgraph "test stage"; - B["jest"]; - C["rspec migration"]; - D["rspec unit"]; - E["rspec integration"]; - F["rspec system"]; - G["rspec fail-fast"]; - end - - subgraph "post-test stage"; - Z["fail-pipeline-early"]; - end - - A --"artifact: list of test files"--> G - G --"on failure"--> Z -``` - -A Merge Request author may choose to opt-out of the fail fast mechanism by doing one of the following: - -- Adding the `pipeline:skip-rspec-fail-fast` label to the merge request -- Starting the `dont-interrupt-me` job found in the `sync` stage of a Merge Request pipeline. - -The `rspec fail-fast` is a no-op if there are more than 10 test files related to the -Merge Request. This prevents `rspec fail-fast` duration from exceeding the average -`rspec` job duration and defeating its purpose. - -This number can be overridden by setting a CI/CD variable named `RSPEC_FAIL_FAST_TEST_FILE_COUNT_THRESHOLD`. - -NOTE: -This experiment is only enabled when the CI/CD variable `RSPEC_FAIL_FAST_ENABLED=true` is set. - -#### Determining related test files in a Merge Request - -The test files related to the Merge Request are determined using the [`test_file_finder`](https://gitlab.com/gitlab-org/ci-cd/test_file_finder) gem. -We are using a custom mapping between source file to test files, maintained in the `tests.yml` file. - -### RSpec minimal jobs - -Before a merge request is approved, the pipeline will run a minimal set of RSpec tests that are related to the merge request changes. -This is to reduce the pipeline cost and shorten the job duration. - -To identify the minimal set of tests needed, we use [Crystalball gem](https://github.com/toptal/crystalball) to create a test mapping. -The test mapping contains a map of each source files to a list of test files which is dependent of the source file. -This mapping is currently generated using a combination of test coverage tracing and a static mapping. -In the `detect-tests` job, we use this mapping to identify the minimal tests needed for the current Merge Request. - -After a merge request has been approved, the pipeline would contain the full RSpec tests. This will ensure that all tests -have been run before a merge request is merged. - -### Jest minimal jobs - -Before a merge request is approved, the pipeline will run a minimal set of Jest tests that are related to the merge request changes. -This is to reduce the pipeline cost and shorten the job duration. - -To identify the minimal set of tests needed, we pass a list of all the changed files into `jest` using the [`--findRelatedTests`](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles) option. -In this mode, `jest` would resolve all the dependencies of related to the changed files, which include test files that have these files in the dependency chain. - -After a merge request has been approved, the pipeline would contain the full Jest tests. This will ensure that all tests -have been run before a merge request is merged. - -In addition, there are a few circumstances where we would always run the full Jest tests: - -- when `package.json`, `yarn.lock`, `jest` config changes -- when vendored JavaScript is changed -- when `.graphql` files are changed - -### PostgreSQL versions testing - -Our test suite runs against PG12 as GitLab.com runs on PG12 and -[Omnibus defaults to PG12 for new installs and upgrades](../administration/package_information/postgresql_versions.md), -Our test suite is currently running against PG11, since GitLab.com still runs on PG11. - -We do run our test suite against PG11 on nightly scheduled pipelines as well as upon specific -database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). - -#### Current versions testing - -| Where? | PostgreSQL version | -| ------ | ------------------ | -| MRs | 12, 11 for DB library changes | -| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | -| 2-hourly scheduled pipelines | 12, 11 for DB library changes | -| `nightly` scheduled pipelines | 12, 11 | - -#### Long-term plan - -We follow the [PostgreSQL versions shipped with Omnibus GitLab](../administration/package_information/postgresql_versions.md): - -| PostgreSQL version | 13.11 (April 2021) | 13.12 (May 2021) | 14.0 (June 2021?) | -| -------------------| ---------------------- | ---------------------- | ---------------------- | -| PG12 | `nightly` | MRs/`2-hour`/`nightly` | MRs/`2-hour`/`nightly` | -| PG11 | MRs/`2-hour`/`nightly` | `nightly` | `nightly` | - -### Test jobs - -Consult [GitLab tests in the Continuous Integration (CI) context](testing_guide/ci.md) -for more information. - -We have dedicated jobs for each [testing level](testing_guide/testing_levels.md) and each job runs depending on the -changes made in your merge request. -If you want to force all the RSpec jobs to run regardless of your changes, you can add the `pipeline:run-all-rspec` label to the merge request. - -> Forcing all jobs on docs only related MRs would not have the prerequisite jobs and would lead to errors - -### Review app jobs - -Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. - -### As-if-FOSS jobs - -The `* as-if-foss` jobs allows the GitLab test suite "as-if-FOSS", meaning as if the jobs would run in the context -of the `gitlab-org/gitlab-foss` project. These jobs are only created in the following cases: - -- `gitlab-org/security/gitlab` merge requests. -- Merge requests with the `pipeline:run-as-if-foss` label -- Merge requests that changes the CI configuration. - -The `* as-if-foss` jobs are run in addition to the regular EE-context jobs. They have the `FOSS_ONLY='1'` variable -set and get their EE-specific folders removed before the tests start running. - -The intent is to ensure that a change doesn't introduce a failure after the `gitlab-org/gitlab` project is synced to -the `gitlab-org/gitlab-foss` project. - -## Performance - -### Interruptible pipelines - -By default, all jobs are [interruptible](../ci/yaml/index.md#interruptible), except the -`dont-interrupt-me` job which runs automatically on `main`, and is `manual` -otherwise. - -If you want a running pipeline to finish even if you push new commits to a merge -request, be sure to start the `dont-interrupt-me` job before pushing. - -### Caching strategy - -1. All jobs must only pull caches by default. -1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. -1. We currently have several different cache definitions defined in - [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), - with fixed keys: - - `.setup-test-env-cache` - - `.rails-cache` - - `.static-analysis-cache` - - `.coverage-cache` - - `.danger-review-cache` - - `.qa-cache` - - `.yarn-cache` - - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). -1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). -1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: - - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). -1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). - -### Artifacts strategy +## CI configuration internals -We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. +### Workflow rules -### Pre-clone step +Pipelines for the GitLab project are created using the [`workflow:rules` keyword](../ci/yaml/index.md#workflow) +feature of the GitLab CI/CD. -The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) -to seed the project with a recent archive of the repository. This is done for -several reasons: +Pipelines are always created for the following scenarios: -- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. -- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. +- `main` branch, including on schedules, pushes, merges, and so on. +- Merge requests. +- Tags. +- Stable, `auto-deploy`, and security branches. -The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). +Pipeline creation is also affected by the following CI/CD variables: -The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: +- If `$FORCE_GITLAB_CI` is set, pipelines are created. +- If `$GITLAB_INTERNAL` is not set, pipelines are not created. -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz +No pipeline is created in any other cases (for example, when pushing a branch with no +MR for it). - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi +The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` +### Default image -The first step of the script downloads `gitlab-master.tar.gz` from -Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) -that is responsible for keeping that archive up-to-date. Every two hours -on a scheduled pipeline, it does the following: +The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). -1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Saves the data as a `.tar.gz`. -1. Uploads it into the Google Cloud Storage bucket. +<!-- vale gitlab.Spelling = NO --> -When a CI job runs with this configuration, the output looks something like this: +It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/group/project... -Fetching changes... -Reinitialized existing Git repository in /builds/group/project/.git/ -``` +<!-- vale gitlab.Spelling = YES --> -Note that the `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. +The images used in our pipelines are configured in the +[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) +project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) +for redundancy. -`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a -GitLab Team Member, find credentials in the -[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). +The current version of the build images can be found in the +["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). +### Default variables -## CI configuration internals +In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), +each pipeline includes default variables defined in +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). ### Stages @@ -644,24 +609,6 @@ that is deployed in stage `review`. [an issue with the deployment](https://gitlab.com/gitlab-org/gitlab/-/issues/233458)). - `notify`: This stage includes jobs that notify various failures to Slack. -### Default image - -The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - -<!-- vale gitlab.Spelling = NO --> - -It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. - -<!-- vale gitlab.Spelling = YES --> - -The images used in our pipelines are configured in the -[`gitlab-org/gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) -project, which is push-mirrored to [`gitlab/gitlab-build-images`](https://dev.gitlab.org/gitlab/gitlab-build-images) -for redundancy. - -The current version of the build images can be found in the -["Used by GitLab section"](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/.gitlab-ci.yml). - ### Dependency Proxy Some of the jobs are using images from Docker Hub, where we also use @@ -681,12 +628,6 @@ Projects in the `gitlab-org` group pull from the Dependency Proxy, while forks that reside on any other personal namespaces or groups fall back to Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. -### Default variables - -In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), -each pipeline includes default variables defined in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). - ### Common job definitions Most of the jobs [extend from a few CI definitions](../ci/yaml/index.md#extends) @@ -756,8 +697,6 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | | `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | -| `if-rspec-fail-fast-disabled` | Limits jobs to pipelines with `$RSPEC_FAIL_FAST_ENABLED` CI/CD variable not set to `"true"`. | | -| `if-rspec-fail-fast-skipped` | Matches if the pipeline is for a merge request and the MR has label ~"pipeline:skip-rspec-fail-fast". | | | `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> @@ -783,6 +722,114 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | | `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | +## Performance + +### Interruptible pipelines + +By default, all jobs are [interruptible](../ci/yaml/index.md#interruptible), except the +`dont-interrupt-me` job which runs automatically on `main`, and is `manual` +otherwise. + +If you want a running pipeline to finish even if you push new commits to a merge +request, be sure to start the `dont-interrupt-me` job before pushing. + +### Caching strategy + +1. All jobs must only pull caches by default. +1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. +1. We currently have several different cache definitions defined in + [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), + with fixed keys: + - `.setup-test-env-cache` + - `.rails-cache` + - `.static-analysis-cache` + - `.coverage-cache` + - `.danger-review-cache` + - `.qa-cache` + - `.yarn-cache` + - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). +1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). +1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: + - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). + - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-storybook-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). +1. These jobs can also be forced to run in merge requests with the `pipeline:update-cache` label (this can be useful to warm the caches in a MR that updates the cache keys). + +### Artifacts strategy + +We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. + +### Pre-clone step + +The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) +to seed the project with a recent archive of the repository. This is done for +several reasons: + +- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. +- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. + +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). + +The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: + +```shell +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) +``` + +The first step of the script downloads `gitlab-master.tar.gz` from +Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) +that is responsible for keeping that archive up-to-date. Every two hours +on a scheduled pipeline, it does the following: + +1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Saves the data as a `.tar.gz`. +1. Uploads it into the Google Cloud Storage bucket. + +When a CI job runs with this configuration, the output looks something like this: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/group/project... +Fetching changes... +Reinitialized existing Git repository in /builds/group/project/.git/ +``` + +Note that the `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a +GitLab Team Member, find credentials in the +[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). + +Note that this bucket should be located in the same continent as the +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). + --- [Return to Development documentation](index.md) diff --git a/doc/development/profiling.md b/doc/development/profiling.md index a58e1d60cc5..656b30402a6 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -98,11 +98,13 @@ profile and log output to S3. ## Speedscope flamegraphs -You can generate a flamegraph for a particular URL by adding the `performance_bar=flamegraph` parameter to the request. +You can generate a flamegraph for a particular URL by selecting a flamegraph sampling mode button in the performance bar or by adding the `performance_bar=flamegraph` parameter to the request.  -More information about the views can be found in the [Speedscope docs](https://github.com/jlfwong/speedscope#views) +Find more information about the views in the [Speedscope docs](https://github.com/jlfwong/speedscope#views). + +Find more information about different sampling modes in the [Stackprof docs](https://github.com/tmm1/stackprof#sampling). This is enabled for all users that can access the performance bar. diff --git a/doc/development/rails_update.md b/doc/development/rails_update.md new file mode 100644 index 00000000000..f25d68a8900 --- /dev/null +++ b/doc/development/rails_update.md @@ -0,0 +1,110 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Rails update guidelines + +We strive to run GitLab using the latest Rails releases to benefit from performance, security updates, and new features. + +## Rails update approach + +1. [Prepare an MR for GitLab](#prepare-an-mr-for-gitlab). +1. [Prepare an MR for Gitaly](#prepare-an-mr-for-gitaly). +1. [Create patch releases and backports for security patches](#create-patch-releases-and-backports-for-security-patches). + +### Prepare an MR for GitLab + +1. Check the [Upgrading Ruby on Rails](https://guides.rubyonrails.org/upgrading_ruby_on_rails.html) guide and prepare the application for the upcoming changes. +1. Update the `rails` gem version in `Gemfile`. +1. Run `bundle update rails`. +1. Run the update task `rake rails:update`. +1. Update the `activesupport` version in `qa/Gemfile`. +1. Run `bundle update --conservative activesupport` in the `qa` folder. +1. Resolve any Bundler conflicts. +1. Ensure that `@rails/ujs` and `@rails/actioncable` npm packages match the new rails version in [`package.json`](https://gitlab.com/gitlab-org/gitlab/blob/master/package.json). +1. Create an MR with the `pipeline:run-all-rspec` label and see if pipeline breaks. +1. To resolve and debug spec failures use `git bisect` against the rails repository. See the [debugging section](#git-bisect-against-rails) below. +1. Include links to the Gem diffs between the two versions in the merge request description. For example, this is the gem diff for [`activesupport` 6.1.3.2 to +6.1.4.1](https://my.diffend.io/gems/activerecord/6.1.3.2/6.1.4.1). + +### Prepare an MR for Gitaly + +1. Update the `activesupport` gem version in [Gitaly Ruby's Gemfile](https://gitlab.com/gitlab-org/gitaly/-/blob/master/ruby/Gemfile). +1. Run `bundle update --conservative activesupport`. +1. Create an MR against the Gitaly project with these changes. +1. Make this MR dependent on the MR created in the GitLab project. +1. Merged this MR only **after** merging the GitLab project's MR. + +### Create patch releases and backports for security patches + +If the Rails update was over a patch release and it contains important security fixes, +make sure to release it in a +GitLab patch release to self-managed customers. Consult with our [release managers](https://about.gitlab.com/community/release-managers/) +for how to proceed. + +### Deprecation Logger + +We also log Ruby and Rails deprecation warnings into a dedicated log file, `log/deprecation_json.log`. It provides +clues when there is code that is not adequately covered by tests and hence would slip past `DeprecationToolkitEnv`. + +For GitLab SaaS, GitLab team members can inspect these log events in Kibana (`https://log.gprd.gitlab.net/goto/f7cebf1ff05038d901ba2c45925c7e01`). + +## Git bisect against Rails + +Usually, if you know which Rails change caused the spec to fail, it adds additional context and +helps to find the fix for the failure. +To efficiently and quickly find which Rails change caused the spec failure you can use the +[`git bisect`](https://git-scm.com/docs/git-bisect) command against the Rails repository: + +1. Clone the `rails` project in a folder of your choice. For example, it might be the GDK root dir: + + ```shell + cd <GDK_FOLDER> + git clone https://github.com/rails/rails.git + ``` + +1. Replace the `gem 'rails'` line in GitLab `Gemfile` with: + + ```ruby + gem 'rails', ENV['RAILS_VERSION'], path: ENV['RAILS_FOLDER'] + ``` + +1. Set the `RAILS_FOLDER` env variable with the folder you cloned Rails into: + + ```shell + export RAILS_FOLDER="<GDK_FOLDER>/rails" + ``` + +1. Change the directory to `RAILS_FOLDER` and set the range for the `git bisect` command: + + ```shell + cd $RAILS_FOLDER + git bisect start <NEW_VERSION_TAG> <OLD_VERSION_TAG> + ``` + + Where `<NEW_VERSION_TAG>` is the tag where the spec is red and `<OLD_VERSION_TAG>` is the one with the green spec. + For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. + Replace `<NEW_VERSION_TAG>` with the tag where the spec is red and `<OLD_VERSION_TAG>` with the one with the green spec. For example, `git bisect start v6.1.4.1 v6.1.3.2` if we're upgrading from version 6.1.3.2 to 6.1.4.1. + In the output, you can see how many steps approximately it takes to find the commit. +1. Start the `git bisect` process and pass spec's file name(s) to `scripts/rails-update-bisect` as an argument or arguments. It can be faster to pick only one example instead of an entire spec file. + + ```shell + git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb + # OR + git bisect run <GDK_FOLDER>/gitlab/scripts/rails-update-bisect spec/models/ability_spec.rb:7 + ``` + +1. When the process is completed, `git bisect` prints the commit hash, which you can use to find the corresponding MR in the [`rails/rails`](https://github.com/rails/rails) repository. +1. Execute `git bisect reset` to exit the `bisect` mode. +1. Revert the changes to `Gemfile`: + + ```shell + git checkout -- Gemfile + ``` + +### Follow-up reading material + +- [Upgrading Ruby on Rails guide](https://guides.rubyonrails.org/upgrading_ruby_on_rails.html) +- [Rails releases page](https://github.com/rails/rails/releases) diff --git a/doc/development/redis.md b/doc/development/redis.md index e631a6ec80c..fa07cebdc61 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -6,12 +6,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redis guidelines +## Redis instances + GitLab uses [Redis](https://redis.io) for the following distinct purposes: - Caching (mostly via `Rails.cache`). - As a job processing queue with [Sidekiq](sidekiq_style_guide.md). - To manage the shared application state. +- To store CI trace chunks. - As a Pub/Sub queue backend for ActionCable. +- Rate limiting state storage. +- Sessions. In most environments (including the GDK), all of these point to the same Redis instance. @@ -29,6 +34,8 @@ more often than it is read. If [Geo](geo.md) is enabled, each Geo node gets its own, independent Redis database. +We have [development documentation on adding a new Redis instance](redis/new_redis_instance.md). + ## Key naming Redis is a flat namespace with no hierarchy, which means we must pay attention diff --git a/doc/development/redis/new_redis_instance.md b/doc/development/redis/new_redis_instance.md new file mode 100644 index 00000000000..37ee51ebb82 --- /dev/null +++ b/doc/development/redis/new_redis_instance.md @@ -0,0 +1,132 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Add a new Redis instance + +GitLab can make use of multiple [Redis instances](../redis.md#redis-instances). +These instances are functionally partitioned so that, for example, we +can store [CI trace chunks](../../administration/job_logs.md#incremental-logging-architecture) +from one Redis instance while storing sessions in another. + +From time to time we might want to add a new Redis instance. Typically this will +be a functional partition split from one of the existing instances such as the +cache or shared state. This document describes an approach +for adding a new Redis instance that handles existing data, based on +prior examples: + +- [Dedicated Redis instance for Trace Chunk storage](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/462). +- [Create dedicated Redis instance for Rate Limiting data](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/526). + +This document does not cover the operational side of preparing and configuring +the new Redis instance in detail, but the example epics do contain information +on previous approaches to this. + +## Step 1: Support configuring the new instance + +Before we can switch any features to using the new instance, we have to support +configuring it and referring to it in the codebase. We must support the +main installation types: + +- Source installs (including development environments) - [example MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62767) +- Omnibus - [example MR](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5316) +- Helm charts - [example MR](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2031) + +### Fallback instance + +In the application code, we need to define a fallback instance in case the new +instance is not configured. For example, if a GitLab instance has already +configured a separate shared state Redis, and we are partitioning data from the +shared state Redis, our new instance's configuration should default to that of +the shared state Redis when it's not present. Otherwise we could break instances +that don't configure the new Redis instance as soon as it's available. + +You can [define a `.config_fallback` method](https://gitlab.com/gitlab-org/gitlab/-/blob/a75471dd744678f1a59eeb99f71fca577b155acd/lib/gitlab/redis/wrapper.rb#L69-87) +in `Gitlab::Redis::Wrapper` (the base class for all Redis instances) +that defines the instance to be used if this one is not configured. If we were +adding a `Foo` instance that should fall back to `SharedState`, we can do that +like this: + +```ruby +module Gitlab + module Redis + class Foo < ::Gitlab::Redis::Wrapper + # The data we store on Foo used to be stored on SharedState. + def self.config_fallback + SharedState + end + end + end +end +``` + +We should also add specs like those in +[`trace_chunks_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/redis/trace_chunks_spec.rb) +to ensure that this fallback works correctly. + +## Step 2: Support writing to and reading from the new instance + +When migrating to the new instance, we must account for cases where data is +either on: + +- The 'old' (original) instance. +- The new one that we have just added support for. + +As a result we may need to support reading from and writing to both +instances, depending on some condition. + +The exact condition to use varies depending on the data to be migrated. For +the trace chunks case above, there was already a database column indicating where the +data was stored (as there are other storage options than Redis). + +This step may not apply if the data has a very short lifetime (a few minutes at most) +and is not critical. In that case, we +may decide that it is OK to incur a small amount of data loss and switch +over through configuration only. + +If there is not a more natural way to mark where the data is stored, using a +[feature flag](../feature_flags/index.md) may be convenient: + +- It does not require an application restart to take effect. +- It applies to all application instances (Sidekiq, API, web, etc.) at + the same time. +- It supports incremental rollout - ideally by actor (project, group, + user, etc.) - so that we can monitor for errors and roll back easily. + +## Step 3: Migrate the data + +We then need to configure the new instance for GitLab.com's production and +staging environments. Hopefully it will be possible to test this change +effectively on staging, to at least make sure that basic usage continues to +work. + +After that is done, we can roll out the change to production. Ideally this would +be in an incremental fashion, following the +[standard incremental rollout](../feature_flags/controls.md#rolling-out-changes) +documentation for feature flags. + +When we have been using the new instance 100% of the time in production for a +while and there are no issues, we can proceed. + +## Step 4: clean up after the migration + +<!-- markdownlint-disable MD044 --> +We may choose to keep the migration paths or remove them, depending on whether +or not we expect self-managed instances to perform this migration. +[gitlab-com/gl-infra/scalability#1131](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1131#note_603354746) +contains a discussion on this topic for the trace chunks feature flag. It may +be - as in that case - that we decide that the maintenance costs of supporting +the migration code are higher than the benefits of allowing self-managed +instances to perform this migration seamlessly, if we expect self-managed +instances to cope without this functional partition. +<!-- markdownlint-enable MD044 --> + +If we decide to keep the migration code: + +- We should document the migration steps. +- If we used a feature flag, we should ensure it's an [ops type feature + flag](../feature_flags/index.md#ops-type), as these are long-lived flags. + +Otherwise, we can remove the flags and conclude the project. diff --git a/doc/development/repository_mirroring.md b/doc/development/repository_mirroring.md index bb4c62d70ee..573ffaccaf9 100644 --- a/doc/development/repository_mirroring.md +++ b/doc/development/repository_mirroring.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w <!-- vale gitlab.Spelling = NO --> In December 2018, Tiago Botelho hosted a Deep Dive (GitLab team members only: `https://gitlab.com/gitlab-org/create-stage/issues/1`) -on the GitLab [Pull Repository Mirroring functionality](../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository) +on the GitLab [Pull Repository Mirroring functionality](../user/project/repository/mirror/pull.md) to share his domain specific knowledge with anyone who may work in this part of the codebase in the future. You can find the <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [recording on YouTube](https://www.youtube.com/watch?v=sSZq0fpdY-Y), and the slides in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/8693404888a941fd851f8a8ecdec9675/Gitlab_Create_-_Pull_Mirroring_Deep_Dive.pdf). diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index ded6b074324..568e8a9d123 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -183,6 +183,8 @@ queries they produce. Everything in `app/presenters`, used for exposing complex data to a Rails view, without having to create many instance variables. +See [the documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/presenters/README.md) for more information. + ### Serializers Everything in `app/serializers`, used for presenting the response to a request, diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md new file mode 100644 index 00000000000..ad6bff8499a --- /dev/null +++ b/doc/development/ruby_upgrade.md @@ -0,0 +1,275 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Ruby upgrade guidelines + +We strive to run GitLab using the latest Ruby MRI releases to benefit from performance and +security updates and new Ruby APIs. When upgrading Ruby across GitLab, we should do +so in a way that: + +- Is least disruptive to contributors. +- Optimizes for GitLab SaaS availability. +- Maintains Ruby version parity across all parts of GitLab. + +Before making changes to Ruby versions, read through this document carefully and entirely to get a high-level +understanding of what changes may be necessary. It is likely that every Ruby upgrade is a little +different than the one before it, so assess the order and necessity of the documented +steps. + +## Scope of a Ruby upgrade + +The first thing to consider when upgrading Ruby is scope. In general, we consider +the following areas in which Ruby updates may have to occur: + +- The main GitLab Rails repository. +- Any ancillary Ruby system repositories. +- Any third-party libraries used by systems in these repositories. +- Any GitLab libraries used by systems in these repositories. + +We may not always have to touch all of these. For instance, a patch-level Ruby update is +unlikely to require updates in third-party gems. + +### Patch, minor, and major upgrades + +When assessing scope, the Ruby version level matters. For instance, it is harder and riskier +to upgrade GitLab from Ruby 2.x to 3.x than it is to upgrade from Ruby 2.7.2 to 2.7.4, as +patch releases are typically restricted to security or bug fixes. +Be aware of this when preparing an upgrade and plan accordingly. + +To help you estimate the scope of future upgrades, see the efforts required for the following upgrades: + +- [Patch upgrade 2.7.2 -> 2.7.4](https://gitlab.com/gitlab-org/gitlab/-/issues/335890) +- [Minor upgrade 2.6.x -> 2.7.x](https://gitlab.com/groups/gitlab-org/-/epics/2380) +- [Major upgrade 2.x.x -> 3.x.x](https://gitlab.com/groups/gitlab-org/-/epics/5149) + +## Affected audiences and targets + +Before any upgrade, consider all audiences and targets, ordered by how immediately they are affected by Ruby upgrades: + +1. **Developers.** We have many contributors to GitLab and related projects both inside and outside the company. Changing files such as `.ruby-version` affects everyone using tooling that interprets these files. +The developers are affected as soon as they pull from the repository containing the merged changes. +1. **GitLab CI/CD.** We heavily lean on CI/CD for code integration and testing. CI/CD jobs do not interpret files such as `.ruby-version`. +Instead, they use the Ruby installed in the Docker container they execute in, which is defined in `.gitlab-ci.yml`. +The container images used in these jobs are maintained in the [`gitlab-build-images`](https://gitlab.com/gitlab-org/gitlab-build-images) repository. +When we merge an update to an image, CI/CD jobs are affected as soon as the [image is built](https://gitlab.com/gitlab-org/gitlab-build-images/#pushing-a-rebuild-image). +1. **GitLab SaaS**. GitLab.com is deployed from customized Helm charts that use Docker images from [Cloud Native GitLab (CNG)](https://gitlab.com/gitlab-org/build/CNG). +Just like CI/CD, `.ruby-version` is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby. +GitLab SaaS is affected with the next deployment. +1. **Self-managed GitLab.** Customers installing GitLab via [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab) use none of the above. +Instead, their Ruby version is defined by the [Ruby software bundle](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/ruby.rb) in Omnibus. +Self-managed customers are affected as soon as they upgrade to the release containing this change. + +## Ruby upgrade approach + +Timing all steps in a Ruby upgrade correctly is critical. As a general guideline, consider the following: + +- For smaller upgrades where production behavior is unlikely to change, aim to keep the version gap between +repositories and production minimal. Coordinate with stakeholders to merge all changes closely together +(within a day or two) to avoid drift. In this scenario the likely order is to upgrade developer tooling and +environments first, production second. +- For larger changes, the risk of going to production with a new Ruby is significant. In this case, try to get into a +position where all known incompatibilities with the new Ruby version are already fixed, then work +with production engineers to deploy the new Ruby to a subset of the GitLab production fleet. In this scenario +the likely order is to update production first, developer tooling and environments second. This makes rollbacks +easier in case of critical regressions in production. + +Either way, we found that from past experience the following approach works well, with some steps likely only +necessary for minor and major upgrades. Note that some of these steps can happen in parallel or may have their +order reversed as described above. + +### Create an epic + +Tracking this work in an epic is useful to get a sense of progress. For larger upgrades, include a +timeline in the epic description so stakeholders know when the final switch is expected to go live. + +Break changes to individual repositories into separate issues under this epic. + +### Communicate the intent to upgrade + +Especially for upgrades that introduce or deprecate features, +communicate early that an upgrade is due, ideally with an associated timeline. Provide links to important or +noteworthy changes, so developers can start to familiarize themselves with +changes ahead of time. + +GitLab team members should announce the intent in relevant Slack channels (`#backend` and `#development` at minimum) +and Engineering Week In Review (EWIR). Include a link to the upgrade epic in your +[communication](https://about.gitlab.com/handbook/engineering/#communication). + +### Add new Ruby to CI/CD and development environments + +To build and run Ruby gems and the GitLab Rails application with a new Ruby, you must first prepare CI/CD +and developer environments to include the new Ruby version. +At this stage, you *must not make it the default Ruby yet*, but make it optional instead. This allows +for a smoother transition by supporting both old and new Ruby versions for a period of time. + +There are two places that require changes: + +1. **[GitLab Build Images](https://gitlab.com/gitlab-org/gitlab-build-images).** These are Docker images +we use for runners and other Docker-based pre-production environments. The kind of change necessary +depends on the scope. + - For [patch level updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/418), it should suffice to increment the patch level of `RUBY_VERSION`. +All projects building against the same minor release automatically download the new patch release. + - For [major and minor updates](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/320), create a new set of Docker images that can be used side-by-side with existing images during the upgrade process. **Important:** Make sure to copy over all Ruby patch files +in the `/patches` directory to a new folder matching the Ruby version you upgrade to, or they aren't applied. +1. **[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit).** +Update GDK to add the new Ruby as an additional option for +developers to choose from. This typically only requires it to be appended to `.tool-versions` so `asdf` +users will benefit from this. Other users will have to install it manually +([example](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/2136).) + +For larger version upgrades, consider working with [Quality Engineering](https://about.gitlab.com/handbook/engineering/quality/) +to identify and set up a test plan. + +### Update third-party gems + +For patch releases this is unlikely to be necessary, but +for minor and major releases, there could be breaking changes or Bundler dependency issues when gems +pin Ruby to a particular version. A good way to find out is to create a merge request in `gitlab-org/gitlab` +and see what breaks. + +### Update GitLab gems and related systems + +This is typically necessary, since gems or Ruby applications that we maintain ourselves contain the build setup such as +`.ruby-version`, `.tool-versions`, or `.gitlab-ci.yml` files. While there isn't always a technical necessity to +update these repositories for the GitLab Rails application to work with a new Ruby, +it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major +upgrades, add new CI/CD jobs to these repositories using the new Ruby. +A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently. + +#### Decide which repositories to update + +When upgrading Ruby, consider updating the following repositories: + +- [Gitaly](https://gitlab.com/gitlab-org/gitaly) ([example](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3771)) +- [GitLab Labkit](https://gitlab.com/gitlab-org/labkit-ruby) ([example](https://gitlab.com/gitlab-org/labkit-ruby/-/merge_requests/79)) +- [GitLab Exporter](https://gitlab.com/gitlab-org/gitlab-exporter) ([example](https://gitlab.com/gitlab-org/gitlab-exporter/-/merge_requests/150)) +- [GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment) ([example](https://gitlab.com/gitlab-org/gitlab-experiment/-/merge_requests/128)) +- [Gollum Lib](https://gitlab.com/gitlab-org/gollum-lib) ([example](https://gitlab.com/gitlab-org/gollum-lib/-/merge_requests/21)) +- [GitLab Helm Chart](https://gitlab.com/gitlab-org/charts/gitlab) ([example](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2162)) +- [GitLab Sidekiq fetcher](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch) ([example](https://gitlab.com/gitlab-org/sidekiq-reliable-fetch/-/merge_requests/33)) +- [Prometheus Ruby Mmap Client](https://gitlab.com/gitlab-org/prometheus-client-mmap) ([example](https://gitlab.com/gitlab-org/prometheus-client-mmap/-/merge_requests/59)) +- [GitLab-mail_room](https://gitlab.com/gitlab-org/gitlab-mail_room) ([example](https://gitlab.com/gitlab-org/gitlab-mail_room/-/merge_requests/16)) + +To assess which of these repositories are critical to be updated alongside the main GitLab application consider: + +- The Ruby version scope. +- The role that the service or library plays in the overall functioning of GitLab. + +Refer to the [list of GitLab projects](https://about.gitlab.com/handbook/engineering/projects/) for a complete +account of which repositories could be affected. +For smaller version upgrades, it can be acceptable to delay updating libraries that are non-essential or where +we are certain that the main application test suite would catch regressions under a new Ruby version. + +NOTE: +Consult with the respective code owners whether it is acceptable to merge these changes ahead +of updating the GitLab application. It might be best to get the necessary approvals +but wait to merge the change until everything is ready. + +### Prepare the GitLab application MR + +With the dependencies updated and the new gem versions released, you can update the main Rails +application with any necessary changes, similar to the gems and related systems. +On top of that, update the documentation to reflect the version change in the installation +and update instructions ([example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68363)). + +NOTE: +Be especially careful with timing this merge request, since as soon as it is merged, all GitLab contributors +will be affected by it and the changes will be deployed. You must ensure that this MR remains +open until everything else is ready, but it can be useful to get approval early to reduce lead time. + +### Give developers time to upgrade (grace period) + +With the new Ruby made available as an option, and all merge requests either ready or merged, +there should be a grace period (1 week at minimum) during which developers can +install the new Ruby on their machines. For GDK and `asdf` users this should happen automatically +via `gdk update`. + +This pause is a good time to assess the risk of this upgrade for GitLab SaaS. +For Ruby upgrades that are high risk, such as major version upgrades, it is recommended to +coordinate the changes with the infrastructure team through a [change management request](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/). +Create this issue early to give everyone enough time to schedule and prepare changes. + +### Make it the default Ruby + +If there are no known version compatibility issues left, and the grace +period has passed, all affected repositories and developer tools should be updated to make the new Ruby +default. + +At this point, update the [GitLab Compose Kit (GCK)](https://gitlab.com/gitlab-org/gitlab-compose-kit). +This is an alternative development environment for users that prefer to run GitLab in `docker-compose`. +This project relies on the same Docker images as our runners, so it should maintain parity with changes +in that repository. This change is only necessary when the minor or major version changes +([example](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/176).) + +As mentioned above, if the impact of the Ruby upgrade on SaaS availability is uncertain, it is +prudent to skip this step until you have verified that it runs smootly in production via a staged +rollout. In this case, go to the next step first, and then, after the verification period has passed, promote +the new Ruby to be the new default. + +### Update CNG and Omnibus, merge the GitLab MR + +The last step is to use the new Ruby in production. This +requires updating Omnibus and production Docker images to use the new version. +Helm charts may also have to be updated if there were changes to related systems that maintain +their own charts (such as `gitlab-exporter`.) + +To use the new Ruby in production, update the following projects: + +- [Cloud-native GitLab Docker Images (CNG)](https://gitlab.com/gitlab-org/build/CNG) ([example](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/739)) +- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab) ([example](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5545)) + +If you submit a change management request, coordinate the rollout with infrastructure +engineers. When dealing with larger upgrades, involve [Release Managers](https://about.gitlab.com/community/release-managers/) +in the rollout plan. + +### Create patch releases and backports for security patches + +If the upgrade was a patch release and contains important security fixes, it should be released as a +GitLab patch release to self-managed customers. Consult our [release managers](https://about.gitlab.com/community/release-managers/) +for how to proceed. + +## Ruby upgrade tooling + +There are several tools that ease the upgrade process. + +### Deprecation Toolkit + +A common problem with Ruby upgrades is that deprecation warnings turn into errors. This means that every single +deprecation warning must be resolved before making the switch. To avoid new warnings from making it into the +main application branch, we use [`DeprecationToolkitEnv`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/deprecation_toolkit_env.rb). +This module observes deprecation warnings emitted from spec runs and turns them into test failures. This prevents +developers from checking in new code that would fail under a new Ruby. + +Sometimes it cannot be avoided to introduce new warnings, for example when a Ruby gem we use emits these warnings +and we have no control over it. In these cases, add silences, like [this merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68865) did. + +### Deprecation Logger + +We also log Ruby and Rails deprecation warnings to a dedicated log file, `log/deprecation_json.log` +(see [GitLab Developers Guide to Logging](logging.md) for where to find GitLab log files), +which can provide clues when there is code that is not adequately covered by tests and hence would slip past `DeprecationToolkitEnv`. + +For GitLab SaaS, GitLab team members can inspect these log events in Kibana +(`https://log.gprd.gitlab.net/goto/f7cebf1ff05038d901ba2c45925c7e01`). + +## Recommendations + +During the upgrade process, consider the following recommendations: + +- **Front-load as many changes as possible.** Especially for minor and major releases, it is likely that application +code will break or change. Any changes that are backward compatible should be merged into the main branch and +released independently ahead of the Ruby version upgrade. This ensures that we move in small increments and +get feedback from production environments early. +- **Create an experimental branch for larger updates.** We generally try to avoid long-running topic branches, +but for purposes of feedback and experimentation, it can be useful to have such a branch to get regular +feedback from CI/CD when running a newer Ruby. This can be helpful when first assessing what problems +we might run into, as [this MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50640) demonstrates. +These experimental branches are not intended to be merged; they can be closed once all required changes have been broken out +and merged back independently. +- **Give yourself enough time to fix problems ahead of a milestone release.** GitLab moves fast. +As a Ruby upgrade requires many MRs to be sent and reviewed, make sure all changes are merged at least a week +before the 22nd. This gives us extra time to act if something breaks. If in doubt, it is better to +postpone the upgrade to the following month, as we [prioritize availability over velocity](https://about.gitlab.com/handbook/engineering/#prioritizing-technical-decisions). diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 824c98b4b03..fdae66b7abc 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -45,7 +45,7 @@ many groups or projects, and the access level (including guest, developer, or maintainer) to groups and projects determines what users can see and what they can access. -Users with admin access can access all projects and even impersonate +Users with the administrator role can access all projects and even impersonate users. #### Sharding and partitioning diff --git a/doc/development/service_ping/dictionary.md b/doc/development/service_ping/dictionary.md index e7e8464ff7a..810c789bc03 100644 --- a/doc/development/service_ping/dictionary.md +++ b/doc/development/service_ping/dictionary.md @@ -1,4 +1,4 @@ --- -redirect_to: 'https://gitlab-org.gitlab.io/growth/product-intelligence/metric-dictionary' +redirect_to: 'https://metrics.gitlab.com/index.html' remove_date: '2021-11-10' --- diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index d86b06a6965..34a845147dc 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -340,6 +340,9 @@ WARNING: HyperLogLog (HLL) is a probabilistic algorithm and its **results always includes some small error**. According to [Redis documentation](https://redis.io/commands/pfcount), data from used HLL implementation is "approximated with a standard error of 0.81%". +NOTE: + A user's consent for usage_stats (`User.single_user&.requires_usage_stats_consent?`) is not checked during the data tracking stage due to performance reasons. Keys corresponding to those counters are present in Redis even if `usage_stats_consent` is still required. However, no metric is collected from Redis and reported back to GitLab as long as `usage_stats_consent` is required. + With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). @@ -939,7 +942,6 @@ Aggregated metrics collected in `7d` and `28d` time frames are added into Servic :packages => 155, :personal_snippets => 2106, :project_snippets => 407, - :promoted_issues => 719, :aggregated_metrics => { :example_metrics_union => 7, :example_metrics_intersection => 2 @@ -1028,9 +1030,9 @@ Example metrics persistence: class UsageData def count_secure_pipelines(time_period) ... - relation = ::Security::Scan.latest_successful_by_build.by_scan_types(scan_type).where(security_scans: time_period) + relation = ::Security::Scan.by_scan_types(scan_type).where(time_period) - pipelines_with_secure_jobs['dependency_scanning_pipeline'] = estimate_batch_distinct_count(relation, :commit_id, batch_size: 1000, start: start_id, finish: finish_id) do |result| + pipelines_with_secure_jobs['dependency_scanning_pipeline'] = estimate_batch_distinct_count(relation, :pipeline_id, batch_size: 1000, start: start_id, finish: finish_id) do |result| ::Gitlab::Usage::Metrics::Aggregates::Sources::PostgresHll .save_aggregated_metrics(metric_name: 'dependency_scanning_pipeline', recorded_at_timestamp: recorded_at, time_period: time_period, data: result) end diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 0a94fa2ff6c..19bf7446da9 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -68,9 +68,9 @@ We use the following terminology to describe the Service Ping components: Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. -The paid feature available in this offering is [Email from GitLab](../../tools/email.md). -Administrators can use this [Premium](https://about.gitlab.com/pricing/premium/) feature to streamline -their workflow by emailing all or some instance users directly from the Admin Area. +##### Features available in 14.1 and later + +1. [Email from GitLab](../../tools/email.md). NOTE: Registration is not yet required for participation, but will be added in a future milestone. @@ -110,7 +110,7 @@ To disable Service Ping in the GitLab UI: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. -1. Clear the **Enable service ping** checkbox. +1. Clear the **Enable Service Ping** checkbox. 1. Select **Save changes**. ### Disable Service Ping using the configuration file @@ -554,5 +554,5 @@ To work around this bug, you have two options: 1. In GitLab, on the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage Statistics**. - 1. Clear the **Enable service ping** checkbox. + 1. Clear the **Enable Service Ping** checkbox. 1. Select **Save Changes**. diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index 8dc2d2255d1..c1478e6290e 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -6,7 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Metrics Dictionary Guide -This guide describes the [Metrics Dictionary](https://gitlab-org.gitlab.io/growth/product-intelligence/metric-dictionary) and how it's implemented. +[Service Ping](index.md) metrics are defined in the +[Metrics Dictionary](https://metrics.gitlab.com/index.html). +This guide describes the dictionary and how it's implemented. ## Metrics Definition and validation diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index c0446aece8b..46040146de2 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -136,7 +136,10 @@ Product Intelligence team as inactive and is assigned to the group owner for rev We are working on automating this process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338466) for details. -Only deprecated metrics can be removed from Service Ping. +Metrics can be removed from Service Ping if they: + +- Were previously [deprecated](#deprecate-a-metric). +- Are not used in any Sisense dashboard. For an example of the metric removal process take a look at this [example issue](https://gitlab.com/gitlab-org/gitlab/-/issues/297029) diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index 048b705636f..eb64d460b5a 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for reviewers - [Service Ping Guide](index.md) -- [Metrics Dictionary](https://gitlab-org.gitlab.io/growth/product-intelligence/metric-dictionary) +- [Metrics Dictionary](https://metrics.gitlab.com/index.html) ## Review process diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 04b7e2f5c45..d45e2073fe7 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -154,12 +154,6 @@ A good example of that would be a cache expiration worker. A job scheduled for an idempotent worker is [deduplicated](#deduplication) when an unstarted job with the same arguments is already in the queue. -WARNING: -For [data consistency jobs](#job-data-consistency-strategies), the deduplication is not compatible with the -`data_consistency` attribute set to `:sticky` or `:delayed`. -The reason for this is that deduplication always takes into account the latest binary replication pointer into account, not the first one. -There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325291) to improve this. - ### Ensuring a worker is idempotent Make sure the worker tests pass using the following shared example: @@ -285,6 +279,55 @@ module AuthorizedProjectUpdate end ``` +### Deduplication with load balancing + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6763) in GitLab 14.4. + +Jobs that declare either `:sticky` or `:delayed` data consistency +are eligible for database load-balancing. +In both cases, jobs are [scheduled in the future](#scheduling-jobs-in-the-future) with a short delay (1 second). +This minimizes the chance of replication lag after a write. + +If you really want to deduplicate jobs eligible for load balancing, +specify `including_scheduled: true` argument when defining deduplication strategy: + +```ruby +class DelayedIdempotentWorker + include ApplicationWorker + data_consistency :delayed + + deduplicate :until_executing, including_scheduled: true + idempotent! + + # ... +end +``` + +#### Preserve the latest WAL location for idempotent jobs + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69372) in GitLab 14.3. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) in GitLab 14.4. + +The deduplication always take into account the latest binary replication pointer, not the first one. +This happens because we drop the same job scheduled for the second time and the Write-Ahead Log (WAL) is lost. +This could lead to comparing the old WAL location and reading from a stale replica. + +To support both deduplication and maintaining data consistency with load balancing, +we are preserving the latest WAL location for idempotent jobs in Redis. +This way we are always comparing the latest binary replication pointer, +making sure that we read from the replica that is fully caught up. + +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, +ask an administrator to [enable the preserve_latest_wal_locations_for_idempotent_jobs flag](../administration/feature_flags.md). +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, +ask an administrator to [enable the `preserve_latest_wal_locations_for_idempotent_jobs` flag](../administration/feature_flags.md). +This feature flag is related to GitLab development and is not intended to be used by GitLab administrators, though. +On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. + ## Limited capacity worker It is possible to limit the number of concurrent running jobs for a worker class @@ -553,11 +596,6 @@ class DelayedWorker end ``` -For [idempotent jobs](#idempotent-jobs), the deduplication is not compatible with the -`data_consistency` attribute set to `:sticky` or `:delayed`. -The reason for this is that deduplication always takes into account the latest binary replication pointer into account, not the first one. -There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325291) to improve this. - ### `feature_flag` property The `feature_flag` property allows you to toggle a job's `data_consistency`, @@ -583,6 +621,12 @@ class DelayedWorker end ``` +### Data consistency with idempotent jobs + +For [idempotent jobs](#idempotent-jobs) that declare either `:sticky` or `:delayed` data consistency, we are +[preserving the latest WAL location](#preserve-the-latest-wal-location-for-idempotent-jobs) while deduplicating, +ensuring that we read from the replica that is fully caught up. + ## Jobs with External Dependencies Most background jobs in the GitLab application communicate with other GitLab diff --git a/doc/development/snowplow/dictionary.md b/doc/development/snowplow/dictionary.md index 589d6f6fb9f..02e9ba5ce20 100644 --- a/doc/development/snowplow/dictionary.md +++ b/doc/development/snowplow/dictionary.md @@ -1,44 +1,4 @@ --- -stage: Growth -group: Product Intelligence -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: 'https://metrics.gitlab.com/snowplow.html' +remove_date: '2021-12-28' --- - -<!--- - This documentation is auto generated by a script. - - Please do not edit this file directly, check generate_event_dictionary task on lib/tasks/gitlab/snowplow.rake. ----> - -<!-- vale gitlab.Spelling = NO --> - -# Event Dictionary - -This file is autogenerated, please do not edit it directly. - -To generate these files from the GitLab repository, run: - -```shell -bundle exec rake gitlab:snowplow:generate_event_dictionary -``` - -The Event Dictionary is based on the following event definition YAML files: - -- [`config/events`](https://gitlab.com/gitlab-org/gitlab/-/tree/f9a404301ca22d038e7b9a9eb08d9c1bbd6c4d84/config/events) -- [`ee/config/events`](https://gitlab.com/gitlab-org/gitlab/-/tree/f9a404301ca22d038e7b9a9eb08d9c1bbd6c4d84/ee/config/events) - -## Event definitions - -### `epics promote` - -| category | action | label | property | value | -|---|---|---|---|---| -| `epics` | `promote` | `` | `The string "issue_id"` | `ID of the issue` | - -Issue promoted to epic - -YAML definition: `/ee/config/events/epics_promote.yml` - -Owner: `group::product planning` - -Tiers: `premium`, `ultimate` diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md new file mode 100644 index 00000000000..0d81b442850 --- /dev/null +++ b/doc/development/snowplow/implementation.md @@ -0,0 +1,543 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Implement Snowplow tracking + +This page describes how to: + +- Implement Snowplow frontend and backend tracking +- Test Snowplow events + +## Snowplow JavaScript frontend tracking + +GitLab provides a `Tracking` interface that wraps the [Snowplow JavaScript tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) +to track custom events. + +For the recommended frontend tracking implementation, see [Usage recommendations](#usage-recommendations). + +Tracking implementations must have an `action` and a `category`. You can provide additional +categories from the [structured event taxonomy](index.md#structured-event-taxonomy) with an `extra` object +that accepts key-value pairs. + +| Field | Type | Default value | Description | +|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | `document.body.dataset.page` | Page or subsection of a page in which events are captured. | +| `action` | string | generic | Action the user is taking. Clicks must be `click` and activations must be `activate`. For example, focusing a form field is `activate_form_input`, and clicking a button is `click_button`. | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, `context` as described in [Structured event taxonomy](index.md#structured-event-taxonomy), and `extra` (key-value pairs object). | + +### Usage recommendations + +- Use [data attributes](#implement-data-attribute-tracking) on HTML elements that emit `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. +- Use the [Vue mixin](#implement-vue-component-tracking) for tracking custom events, or if the supported events for data attributes are not propagating. +- Use the [tracking class](#implement-raw-javascript-tracking) when tracking raw JavaScript files. + +### Implement data attribute tracking + +To implement tracking for HAML or Vue templates, add a [`data-track` attribute](#data-track-attributes) to the element. + +The following example shows `data-track-*` attributes assigned to a button: + +```haml +%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } +``` + +```html +<button class="btn" + data-track-action="click_button" + data-track-label="template_preview" + data-track-property="my-template" + data-track-extra='{ "template_variant": "primary" }' +/> +``` + +#### `data-track` attributes + +| Attribute | Required | Description | +|:----------------------|:---------|:------------| +| `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field is `activate_form_input` and clicking a button is `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | +| `data-track-label` | false | The specific element or object to act on. This can be: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | +| `data-track-property` | false | Any additional property of the element, or object being acted on. | +| `data-track-value` | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | +| `data-track-extra` | false | A key-value pair object passed as a valid JSON string. This attribute is added to the `extra` property in our [`gitlab_standard`](schemas.md#gitlab_standard) schema. | +| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](index.md#structured-event-taxonomy). | + +#### Event listeners + +Event listeners bind at the document level to handle click events in elements with data attributes. +This allows them to be handled when the DOM re-renders or changes. Document-level binding reduces +the likelihood that click events stop propagating up the DOM tree. + +If click events stop propagating, you must implement listeners and [Vue component tracking](#implement-vue-component-tracking) or [raw JavaScript tracking](#implement-raw-javascript-tracking). + +#### Helper methods + +Use the following Ruby helper: + +```ruby +tracking_attrs(label, action, property) # { data: { track_label... } } + +%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } +``` + +If you use the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76), you must wrap `html_options` under the `html_options` keyword argument. If you +use the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to), you don't need to wrap `html_options`. + +```ruby +# Bad += nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", +track_action: "click_button" }) + +# Good += nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: +"explore_groups", track_action: "click_button" } }) + +# Good (other helpers) += link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: +"click_button" } +``` + +### Implement Vue component tracking + +For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` +static method and the `track` method called from components or templates. You can specify tracking +options in `data` or `computed`. These options override any defaults and allow the values to be dynamic +from props or based on state. + +Default options are passed when an event is tracked from the component. If you don't specify an option, +the default `document.body.dataset.page` is used. The default options are: + +- `category` +- `label` +- `property` +- `value` + +To implement Vue component tracking: + +1. Import the `Tracking` library and request a `mixin`: + + ```javascript + import Tracking from '~/tracking'; + const trackingMixin = Tracking.mixin; + ``` + +1. Provide categories to track the event from the component. For example, to track all events in a +component with a label, use the `label` category: + + ```javascript + import Tracking from '~/tracking'; + const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); + ``` + +1. In the component, declare the Vue `mixin`: + + ```javascript + export default { + mixins: [trackingMixin], + // ...[component implementation]... + data() { + return { + expanded: false, + tracking: { + label: 'left_sidebar', + }, + }; + }, + }; + ``` + +1. To receive event data as a tracking object or computed property: + - Declare it in the `data` function. Use a `tracking` object when default event properties are dynamic or provided at runtime: + + ```javascript + export default { + name: 'RightSidebar', + mixins: [Tracking.mixin()], + data() { + return { + tracking: { + label: 'right_sidebar', + // category: '', + // property: '', + // value: '', + // experiment: '', + // extra: {}, + }, + }; + }, + }; + ``` + + - Declare it in the event data in the `track` function. This object merges with any previously provided options: + + ```javascript + this.track('click_button', { + label: 'right_sidebar', + }); + ``` + +1. Optional. Use the `track` method in a template: + + ```html + <template> + <div> + <button data-testid="toggle" @click="toggle">Toggle</button> + + <div v-if="expanded"> + <p>Hello world!</p> + <button @click="track('click_action')">Track another event</button> + </div> + </div> + </template> + ``` + +The following example shows an implementation of Vue component tracking: + +```javascript +export default { + name: 'RightSidebar', + mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { + return { + expanded: false, + }; + }, + methods: { + toggle() { + this.expanded = !this.expanded; + // Additional data will be merged, like `value` below + this.track('click_toggle', { value: Number(this.expanded) }); + } + } +}; +``` + +#### Testing example + +```javascript +import { mockTracking } from 'helpers/tracking_helper'; +// mockTracking(category, documentOverride, spyMethod) + +describe('RightSidebar.vue', () => { + let trackingSpy; + let wrapper; + + beforeEach(() => { + trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); + }); + + const findToggle = () => wrapper.find('[data-testid="toggle"]'); + + it('tracks turning off toggle', () => { + findToggle().trigger('click'); + + expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', { + label: 'right_sidebar', + value: 0, + }); + }); +}); +``` + +### Implement raw JavaScript tracking + +To call custom event tracking and instrumentation directly from the JavaScript file, call the `Tracking.event` static function. + +The following example demonstrates tracking a click on a button by manually calling `Tracking.event`. + +```javascript +import Tracking from '~/tracking'; + +const button = document.getElementById('create_from_template_button'); + +button.addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + extra: { + templateVariant: 'primary', + valid: 1, + }, + }); +}); +``` + +#### Testing example + +```javascript +import Tracking from '~/tracking'; + +describe('MyTracking', () => { + let wrapper; + + beforeEach(() => { + jest.spyOn(Tracking, 'event'); + }); + + const findButton = () => wrapper.find('[data-testid="create_from_template"]'); + + it('tracks event', () => { + findButton().trigger('click'); + + expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { + label: 'create_from_template', + property: 'template_preview', + extra: { + templateVariant: 'primary', + valid: true, + }, + }); + }); +}); +``` + +### Form tracking + +To enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking): + +1. Call `Tracking.enableFormTracking` when the DOM is ready. +1. Provide a `config` object that includes at least one of the following elements: + - `forms` determines the forms to track. Identified by the CSS class name. + - `fields` determines the fields inside the tracked forms to track. Identified by the field `name`. +1. Optional. Provide a list of contexts as the second argument. The [`gitlab_standard`](schemas.md#gitlab_standard) schema is excluded from these events. + +```javascript +Tracking.enableFormTracking({ + forms: { allow: ['sign-in-form', 'password-recovery-form'] }, + fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, +}); +``` + +#### Testing example + +```javascript +import Tracking from '~/tracking'; + +describe('MyFormTracking', () => { + let formTrackingSpy; + + beforeEach(() => { + formTrackingSpy = jest + .spyOn(Tracking, 'enableFormTracking') + .mockImplementation(() => null); + }); + + it('initialized with the correct configuration', () => { + expect(formTrackingSpy).toHaveBeenCalledWith({ + forms: { allow: ['sign-in-form', 'password-recovery-form'] }, + fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, + }); + }); +}); +``` + +## Implement Ruby backend tracking + +`Gitlab::Tracking` is an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. +Backend tracking provides: + +- User behavior tracking +- Instrumentation to monitor and visualize performance over time in a section or aspect of code. + +To add custom event tracking and instrumentation, call the `GitLab::Tracking.event` class method. +For example: + +```ruby +class Projects::CreateService < BaseService + def execute + project = Project.create(params) + + Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, + property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) + end +end +``` + +Use the following arguments: + +| Argument | Type | Default value | Description | +|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| +| `category` | String | | Area or aspect of the application. For example, `HealthCheckController` or `Lfs::FileTransformer`. | +| `action` | String | | The action being taken. For example, a controller action such as `create`, or an Active Record callback. | +| `label` | String | nil | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | +| `property` | String | nil | Any additional property of the element, or object being acted on. | +| `value` | Numeric | nil | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | +| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | +| `project` | Project | nil | The project associated with the event. | +| `user` | User | nil | The user associated with the event. | +| `namespace` | Namespace | nil | The namespace associated with the event. | +| `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | + +### Unit testing + +To test backend Snowplow events, use the `expect_snowplow_event` helper. For more information, see +[testing best practices](../testing_guide/best_practices.md#test-snowplow-events). + +### Performance + +We use the [AsyncEmitter](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/emitters/#the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. + +## Develop and test Snowplow + +To develop and test a Snowplow event, there are several tools to test frontend and backend events: + +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | +|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| +| Snowplow Analytics Debugger Chrome Extension | Yes | No | Yes | Yes | Yes | +| Snowplow Inspector Chrome Extension | Yes | No | Yes | Yes | Yes | +| Snowplow Micro | Yes | Yes | Yes | No | No | + +### Test frontend events + +Before you test frontend events in development, you must: + +1. [Enable Snowplow tracking in the Admin Area](index.md#enable-snowplow-tracking). +1. Turn off ad blockers that could prevent Snowplow JavaScript from loading in your environment. +1. Turn off "Do Not Track" (DNT) in your browser. + +All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#Setting_a_custom_page_URL_and_referrer_URL) personally identifiable +information (PII). PII includes usernames, group, and project names. + +#### Snowplow Analytics Debugger Chrome Extension + +[Snowplow Analytics Debugger](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html) is a browser extension for testing frontend events. It works in production, staging, and local development environments. + +1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. + +#### Snowplow Inspector Chrome Extension + +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works in production, staging, and local development environments. + +1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). +1. To open the extension, select the Snowplow Inspector icon beside the address bar. +1. Click around on a webpage with Snowplow to see JavaScript events firing in the inspector window. + +### Test backend events + +#### Snowplow Micro + +[Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) is a +Docker-based solution for testing backend and frontend in a local development environment. Snowplow Micro +records the same events as the full Snowplow pipeline. To query events, use the Snowplow Micro API. + +To install and run Snowplow Micro, complete these steps to modify the +[GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): + +1. Ensure Docker is installed and running. + +1. To install Snowplow Micro, clone the settings in +[this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). + +1. Navigate to the directory with the cloned project, + and start the appropriate Docker container: + + ```shell + ./snowplow-micro.sh + ``` + +1. Use GDK to start the PostgreSQL terminal and connect + to the `gitlabhq_development` database: + + ```shell + gdk psql -d gitlabhq_development + ``` + +1. Update your instance's settings to enable Snowplow events and + point to the Snowplow Micro collector: + + ```shell + update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; + ``` + +1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking/constants.js` to remove `forceSecureTracker: true`: + + ```diff + diff --git a/app/assets/javascripts/tracking/constants.js b/app/assets/javascripts/tracking/constants.js + index 598111e4086..eff38074d4c 100644 + --- a/app/assets/javascripts/tracking/constants.js + +++ b/app/assets/javascripts/tracking/constants.js + @@ -7,7 +7,6 @@ export const DEFAULT_SNOWPLOW_OPTIONS = { + appId: '', + userFingerprint: false, + respectDoNotTrack: true, + - forceSecureTracker: true, + eventMethod: 'post', + contexts: { webPage: true, performanceTiming: true }, + formTracking: false, + ``` + +1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: + + ```diff + diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb + index 618e359211b..e9084623c43 100644 + --- a/lib/gitlab/tracking.rb + +++ b/lib/gitlab/tracking.rb + @@ -41,7 +41,9 @@ def options(group) + cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, + app_id: Gitlab::CurrentSettings.snowplow_app_id, + form_tracking: additional_features, + - link_click_tracking: additional_features + + link_click_tracking: additional_features, + + protocol: 'http', + + port: 9090 + }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } + end + ``` + +1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: + + ```diff + diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb + index 4fa844de325..5dd9d0eacfb 100644 + --- a/lib/gitlab/tracking/destinations/snowplow.rb + +++ b/lib/gitlab/tracking/destinations/snowplow.rb + @@ -40,7 +40,7 @@ def tracker + def emitter + SnowplowTracker::AsyncEmitter.new( + Gitlab::CurrentSettings.snowplow_collector_hostname, + - protocol: 'https' + + protocol: 'http' + ) + end + end + + ``` + +1. Restart GDK: + + ```shell + gdk restart + ``` + +1. Send a test Snowplow event from the Rails console: + + ```ruby + Gitlab::Tracking.event('category', 'action') + ``` + +1. Navigate to `localhost:9090/micro/good` to see the event. + +#### Useful links + +- [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) +- [Installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) + +### Troubleshoot + +To control content security policy warnings when using an external host, modify `config/gitlab.yml` +to allow or disallow them. To allow them, add the relevant host for `connect_src`. For example, for +`https://snowplow.trx.gitlab.net`: + +```yaml +development: + <<: *base + gitlab: + content_security_policy: + enabled: true + directives: + connect_src: "'self' http://localhost:* http://127.0.0.1:* ws://localhost:* wss://localhost:* ws://127.0.0.1:* https://snowplow.trx.gitlab.net/" +``` diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index e8b7d871b77..b8b35857adf 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -4,49 +4,37 @@ group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Snowplow Guide +# Snowplow -This guide provides an overview of how Snowplow works, and implementation details. - -For more information about Product Intelligence, see: - -- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Service Ping Guide](../service_ping/index.md) - -More useful links: - -- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) +This page provides an overview of how Snowplow works and how to enable it. ## What is Snowplow -Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. +Snowplow is an enterprise-grade marketing and Product Intelligence platform that tracks how users engage with our website and application. -[Snowplow](https://snowplowanalytics.com) consists of the following loosely-coupled sub-systems: +[Snowplow](https://snowplowanalytics.com) consists of several loosely-coupled sub-systems: -- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. -- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. -- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. -- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. -- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. -- **Analytics** are performed on the Snowplow events or on the aggregate tables. +- **Trackers** fire Snowplow events. Snowplow has twelve trackers that cover web, mobile, desktop, server, and IoT. +- **Collectors** receive Snowplow events from trackers. We use different event collectors that synchronize events to Amazon S3, Apache Kafka, or Amazon Kinesis. +- **Enrich** cleans raw Snowplow events, enriches them, and puts them into storage. There is a Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. +- **Storage** stores Snowplow events. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. +- **Data modeling** joins event-level data with other data sets, aggregates them into smaller data sets, and applies business logic. This produces a clean set of tables for data analysis. We use data models for Redshift and Looker. +- **Analytics** are performed on Snowplow events or on aggregate tables.  ### Useful links -- [Understanding the structure of Snowplow data](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) +- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) - [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) -- [List of events used in our codebase (Event Dictionary)](dictionary.md) +- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) ## Enable Snowplow tracking Tracking can be enabled at: - The instance level, which enables tracking on both the frontend and backend layers. -- The user level, though user tracking can be disabled on a per-user basis. +- The user level. User tracking can be disabled on a per user basis. GitLab respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. Snowplow tracking is enabled on GitLab.com, and we use it for most of our tracking strategy. @@ -101,21 +89,22 @@ sequenceDiagram ## Structured event taxonomy -When adding new click events, we should add them in a way that's internally consistent. If we don't, it is difficult to perform analysis across features because each feature captures events differently. +Click events must be consistent. If each feature captures events differently, it can be difficult +to perform analysis. -The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: +Each click event provides attributes that describe the event. -| attribute | type | required | description | +| Attribute | Type | Required | Description | | --------- | ------- | -------- | ----------- | -| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | -| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | -| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element (for example, a tab labeled 'Create from template' for `create_from_template`), a unique identifier if no text is available (for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar), or the name or title attribute of a record being created. | +| category | text | true | The page or backend section of the application. Unless infeasible, use the Rails page attribute by default in the frontend, and namespace + class name on the backend. | +| action | text | true | The action the user takes, or aspect that's being instrumented. The first word must describe the action or aspect. For example, clicks must be `click`, activations must be `activate`, creations must be `create`. Use underscores to describe what was acted on. For example, activating a form field is `activate_form_input`, an interface action like clicking on a dropdown is `click_dropdown`, a behavior like creating a project record from the backend is `create_project`. | +| label | text | false | The specific element or object to act on. This can be one of the following: the label of the element, for example, a tab labeled 'Create from template' for `create_from_template`; a unique identifier if no text is available, for example, `groups_dropdown_close` for closing the Groups dropdown in the top bar; or the name or title attribute of a record being created. | | property | text | false | Any additional property of the element, or object being acted on. | | value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input. For example, `10` when clicking `internal` visibility. | ### Examples -| category* | label | action | property** | value | +| Category* | Label | Action | Property** | Value | |-------------|------------------|-----------------------|----------|:-----:| | `[root:index]` | `main_navigation` | `click_navigation_link` | `[link_label]` | - | | `[groups:boards:show]` | `toggle_swimlanes` | `click_toggle_button` | - | `[is_active]` | @@ -125,8 +114,8 @@ The current method provides several attributes that are sent on each click event | `[projects:clusters:new]` | `chart_options` | `generate_link` | `[chart_link]` | - | | `[projects:clusters:new]` | `chart_options` | `click_add_label_button` | `[label_id]` | - | -_* It's ok to omit the category, and use the default._<br> -_** Property is usually the best place for variable strings._ +_* If you choose to omit the category you can use the default._<br> +_** Use property for variable strings._ ### Reference SQL @@ -152,651 +141,33 @@ ORDER BY collector_tstamp DESC LIMIT 20 ``` -### Web-specific parameters - -Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. - -## Implement Snowplow JS (Frontend) tracking - -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. You can provide additional [Structured event taxonomy](#structured-event-taxonomy) properties along with an `extra` object that accepts key-value pairs. - -| field | type | default value | description | -|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | -| `action` | string | generic | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, `context` (as described in our [Structured event taxonomy](#structured-event-taxonomy)), and `extra` (key-value pairs object). | - -### Usage recommendations - -- Use [data attributes](#tracking-with-data-attributes) on HTML elements that emits either the `click`, `show.bs.dropdown`, or `hide.bs.dropdown` events. -- Use the [Vue mixin](#tracking-within-vue-components) when tracking custom events, or if the supported events for data attributes are not propagating. -- Use the [Tracking class directly](#tracking-in-raw-javascript) when tracking on raw JS files. - -### Tracking with data attributes - -When working in HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. You can provide extra data as a valid JSON string using `data-track-extra`. - -Below is an example of `data-track-*` attributes assigned to a button: - -```haml -%button.btn{ data: { track: { action: "click_button", label: "template_preview", property: "my-template" } } } -``` - -```html -<button class="btn" - data-track-action="click_button" - data-track-label="template_preview" - data-track-property="my-template" - data-track-extra='{ "template_variant": "primary" }' -/> -``` - -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If click events are being stopped from propagating, you must implement your own listeners and follow the instructions in [Tracking within Vue components](#tracking-within-vue-components) or [Tracking in raw JavaScript](#tracking-in-raw-javascript). - -Below is a list of supported `data-track-*` attributes: - -| attribute | required | description | -|:----------------------|:---------|:------------| -| `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | -| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | -| `data-track-extra` | false | A key-value pairs object passed as a valid JSON string. This is added to the `extra` property in our [`gitlab_standard`](#gitlab_standard) schema. | -| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | - -#### Available helpers - -```ruby -tracking_attrs(label, action, property) # { data: { track_label... } } - -%button{ **tracking_attrs('main_navigation', 'click_button', 'navigation') } -``` - -#### Caveats - -When using the GitLab helper method [`nav_link`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/tab_helper.rb#L76) be sure to wrap `html_options` under the `html_options` keyword argument. -Be careful, as this behavior can be confused with the `ActionView` helper method [`link_to`](https://api.rubyonrails.org/v5.2.3/classes/ActionView/Helpers/UrlHelper.html#method-i-link_to) that does not require additional wrapping of `html_options` - -```ruby -# Bad -= nav_link(controller: ['dashboard/groups', 'explore/groups'], data: { track_label: "explore_groups", track_action: "click_button" }) - -# Good -= nav_link(controller: ['dashboard/groups', 'explore/groups'], html_options: { data: { track_label: "explore_groups", track_action: "click_button" } }) - -# Good (other helpers) -= link_to explore_groups_path, title: _("Explore"), data: { track_label: "explore_groups", track_action: "click_button" } -``` - -### Tracking within Vue components - -There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. - -```javascript -import Tracking from '~/tracking'; -const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); -``` - -You can provide default options that are passed along whenever an event is tracked from within your component. For example, if all events in a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. - -You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. - -```javascript -export default { - mixins: [trackingMixin], - // ...[component implementation]... - data() { - return { - expanded: false, - tracking: { - label: 'left_sidebar', - }, - }; - }, -}; -``` - -The mixin provides a `track` method that can be called from within the template, -or from component methods. An example of the whole implementation might look like this: - -```javascript -export default { - name: 'RightSidebar', - mixins: [Tracking.mixin({ label: 'right_sidebar' })], - data() { - return { - expanded: false, - }; - }, - methods: { - toggle() { - this.expanded = !this.expanded; - // Additional data will be merged, like `value` below - this.track('click_toggle', { value: Number(this.expanded) }); - } - } -}; -``` - -The event data can be provided with a `tracking` object, declared in the `data` function, -or as a `computed property`. A `tracking` object is convenient when the default -event properties are dynamic or provided at runtime. - -```javascript -export default { - name: 'RightSidebar', - mixins: [Tracking.mixin()], - data() { - return { - tracking: { - label: 'right_sidebar', - // category: '', - // property: '', - // value: '', - // experiment: '', - // extra: {}, - }, - }; - }, -}; -``` - -The event data can be provided directly in the `track` function as well. -This object merges with any previously provided options. - -```javascript -this.track('click_button', { - label: 'right_sidebar', -}); -``` - -Lastly, if needed within the template, you can use the `track` method directly as well. - -```html -<template> - <div> - <button data-testid="toggle" @click="toggle">Toggle</button> - - <div v-if="expanded"> - <p>Hello world!</p> - <button @click="track('click_action')">Track another event</button> - </div> - </div> -</template> -``` - -#### Testing example - -```javascript -import { mockTracking } from 'helpers/tracking_helper'; -// mockTracking(category, documentOverride, spyMethod) - -describe('RightSidebar.vue', () => { - let trackingSpy; - let wrapper; - - beforeEach(() => { - trackingSpy = mockTracking(undefined, wrapper.element, jest.spyOn); - }); - - const findToggle = () => wrapper.find('[data-testid="toggle"]'); - - it('tracks turning off toggle', () => { - findToggle().trigger('click'); - - expect(trackingSpy).toHaveBeenCalledWith(undefined, 'click_toggle', { - label: 'right_sidebar', - value: 0, - }); - }); -}); -``` - -### Tracking in raw JavaScript - -Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. - -```javascript -import Tracking from '~/tracking'; - -const button = document.getElementById('create_from_template_button'); - -button.addEventListener('click', () => { - Tracking.event('dashboard:projects:index', 'click_button', { - label: 'create_from_template', - property: 'template_preview', - extra: { - templateVariant: 'primary', - valid: 1, - }, - }); -}); -``` - -#### Testing example - -```javascript -import Tracking from '~/tracking'; - -describe('MyTracking', () => { - let wrapper; - - beforeEach(() => { - jest.spyOn(Tracking, 'event'); - }); +#### Last 100 page view events - const findButton = () => wrapper.find('[data-testid="create_from_template"]'); - - it('tracks event', () => { - findButton().trigger('click'); - - expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { - label: 'create_from_template', - property: 'template_preview', - extra: { - templateVariant: 'primary', - valid: true, - }, - }); - }); -}); -``` - -### Form tracking - -Enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking) by calling `Tracking.enableFormTracking` (after the DOM is ready) and providing a `config` object that includes at least one of the following elements: - -- `forms`: determines which forms are tracked, and are identified by the CSS class name. -- `fields`: determines which fields inside the tracked forms are tracked, and are identified by the field `name`. - -An optional list of contexts can be provided as the second argument. -Note that our [`gitlab_standard`](#gitlab_standard) schema is excluded from these events. - -```javascript -Tracking.enableFormTracking({ - forms: { allow: ['sign-in-form', 'password-recovery-form'] }, - fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, -}); -``` - -#### Testing example - -```javascript -import Tracking from '~/tracking'; - -describe('MyFormTracking', () => { - let formTrackingSpy; - - beforeEach(() => { - formTrackingSpy = jest - .spyOn(Tracking, 'enableFormTracking') - .mockImplementation(() => null); - }); - - it('initialized with the correct configuration', () => { - expect(formTrackingSpy).toHaveBeenCalledWith({ - forms: { allow: ['sign-in-form', 'password-recovery-form'] }, - fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, - }); - }); -}); -``` - -## Implement Snowplow Ruby (Backend) tracking - -GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/) for tracking custom events. - -Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: - -| argument | type | default value | description | -|------------|---------------------------|---------------|-----------------------------------------------------------------------------------------------------------------------------------| -| `category` | String | | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | -| `action` | String | | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `label` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `property` | String | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `value` | Numeric | nil | As described in [Structured event taxonomy](#structured-event-taxonomy). | -| `context` | Array\[SelfDescribingJSON\] | nil | An array of custom contexts to send with this event. Most events should not have any custom contexts. | -| `project` | Project | nil | The project associated with the event. | -| `user` | User | nil | The user associated with the event. | -| `namespace` | Namespace | nil | The namespace associated with the event. | -| `extra` | Hash | `{}` | Additional keyword arguments are collected into a hash and sent with the event. | - -Tracking can be viewed as either tracking user behavior, or can be used for instrumentation to monitor and visualize performance over time in an area or aspect of code. - -For example: - -```ruby -class Projects::CreateService < BaseService - def execute - project = Project.create(params) - - Gitlab::Tracking.event('Projects::CreateService', 'create_project', label: project.errors.full_messages.to_sentence, - property: project.valid?.to_s, project: project, user: current_user, namespace: namespace) - end -end +```sql +SELECT + -- page_url, + -- page_title, + -- referer_url, + -- marketing_medium, + -- marketing_source, + -- marketing_campaign, + -- browser_window_width, + -- device_is_mobile + * +FROM legacy.snowplow_page_views_30 +ORDER BY page_view_start DESC +LIMIT 100 ``` -### Unit testing - -Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( -https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. - -### Performance - -We use the [AsyncEmitter](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker/emitters/#the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. - -## Develop and test Snowplow - -There are several tools for developing and testing a Snowplow event. - -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | - -**Legend** - -**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned - -### Test frontend events - -To test frontend events in development: - -- [Enable Snowplow tracking in the Admin Area](#enable-snowplow-tracking). -- Turn off any ad blockers that would prevent Snowplow JS from loading in your environment. -- Turn off "Do Not Track" (DNT) in your browser. - -#### Snowplow Analytics Debugger Chrome Extension - -Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging, and local development environments. - -1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. -1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). - -#### Snowplow Inspector Chrome Extension - -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). -1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. -1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. - -### Snowplow Micro - -Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. - -Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You must modify GDK using the instructions below to set this up. - -- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) -- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- Watch our <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) - -1. Ensure Docker is installed and running. - -1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro) by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): -1. Navigate to the directory with the cloned project, and start the appropriate Docker - container with the following script: - - ```shell - ./snowplow-micro.sh - ``` - -1. Use GDK to start the PostgreSQL terminal and connect to the `gitlabhq_development` database: - - ```shell - gdk psql -d gitlabhq_development - ``` - -1. Update your instance's settings to enable Snowplow events and point to the Snowplow Micro collector: - - ```shell - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking/constants.js` to remove `forceSecureTracker: true`: - - ```diff - diff --git a/app/assets/javascripts/tracking/constants.js b/app/assets/javascripts/tracking/constants.js - index 598111e4086..eff38074d4c 100644 - --- a/app/assets/javascripts/tracking/constants.js - +++ b/app/assets/javascripts/tracking/constants.js - @@ -7,7 +7,6 @@ export const DEFAULT_SNOWPLOW_OPTIONS = { - appId: '', - userFingerprint: false, - respectDoNotTrack: true, - - forceSecureTracker: true, - eventMethod: 'post', - contexts: { webPage: true, performanceTiming: true }, - formTracking: false, - ``` - -1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: - - ```diff - diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb - index 618e359211b..e9084623c43 100644 - --- a/lib/gitlab/tracking.rb - +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def options(group) - cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, - app_id: Gitlab::CurrentSettings.snowplow_app_id, - form_tracking: additional_features, - - link_click_tracking: additional_features - + link_click_tracking: additional_features, - + protocol: 'http', - + port: 9090 - }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } - end - ``` - -1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: - - ```diff - diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb - index 4fa844de325..5dd9d0eacfb 100644 - --- a/lib/gitlab/tracking/destinations/snowplow.rb - +++ b/lib/gitlab/tracking/destinations/snowplow.rb - @@ -40,7 +40,7 @@ def tracker - def emitter - SnowplowTracker::AsyncEmitter.new( - Gitlab::CurrentSettings.snowplow_collector_hostname, - - protocol: 'https' - + protocol: 'http' - ) - end - end - - ``` - -1. Restart GDK: - - ```shell - gdk restart - ``` - -1. Send a test Snowplow event from the Rails console: +### Web-specific parameters - ```ruby - Gitlab::Tracking.event('category', 'action') - ``` +Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/snowplow-tracker-protocol/#Web-specific_parameters) to all web events by default. -1. Navigate to `localhost:9090/micro/good` to see the event. - -### Snowplow Mini - -[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. - -Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. - -For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. - -### Troubleshooting - -To control content security policy warnings when using an external host, you can allow or disallow them by modifying `config/gitlab.yml`. To allow them, add the relevant host for `connect_src`. For example, for `https://snowplow.trx.gitlab.net`: - -```yaml -development: - <<: *base - gitlab: - content_security_policy: - enabled: true - directives: - connect_src: "'self' http://localhost:* http://127.0.0.1:* ws://localhost:* wss://localhost:* ws://127.0.0.1:* https://snowplow.trx.gitlab.net/" -``` +## Related topics -## Snowplow Schemas - -### `gitlab_standard` - -We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. - -The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. - -| Field Name | Required | Type | Description | -|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| -| `project_id` | **{dotted-circle}** | integer | | -| `namespace_id` | **{dotted-circle}** | integer | | -| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | -| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | -| `plan` | **{dotted-circle}** | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | -| `extra` | **{dotted-circle}** | JSON | Any additional data associated with the event, in the form of key-value pairs | - -### Default Schema - -| Field Name | Required | Type | Description | -|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| -| `app_id` | **{check-circle}** | string | Unique identifier for website / application | -| `base_currency` | **{dotted-circle}** | string | Reporting currency | -| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | -| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | -| `br_family` | **{dotted-circle}** | string | Browser family | -| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | -| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | -| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | -| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | -| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | -| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | -| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | -| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | -| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | -| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | -| `br_name` | **{dotted-circle}** | string | Browser name | -| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | -| `br_type` | **{dotted-circle}** | string | Browser type | -| `br_version` | **{dotted-circle}** | string | Browser version | -| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | -| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | -| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | -| `contexts` | **{dotted-circle}** | | | -| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | -| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | -| `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | -| `doc_height` | **{dotted-circle}** | string | Web page height | -| `doc_width` | **{dotted-circle}** | string | Web page width | -| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | -| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | -| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | -| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | -| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | -| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | -| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | -| `dvce_type` | **{dotted-circle}** | string | Type of device | -| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | -| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | -| `event` | **{dotted-circle}** | string | Event type | -| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | -| `event_format` | **{dotted-circle}** | string | Format for event | -| `event_id` | **{dotted-circle}** | string | Event UUID | -| `event_name` | **{dotted-circle}** | string | Event name | -| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | -| `event_version` | **{dotted-circle}** | string | Version of event schema | -| `geo_city` | **{dotted-circle}** | string | City of IP origin | -| `geo_country` | **{dotted-circle}** | string | Country of IP origin | -| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | -| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | -| `geo_region` | **{dotted-circle}** | string | Region of IP origin | -| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | -| `geo_timezone` | **{dotted-circle}** | string | Timezone of IP origin | -| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | -| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor's IP address | -| `ip_isp` | **{dotted-circle}** | string | Visitor's ISP | -| `ip_netspeed` | **{dotted-circle}** | string | Visitor's connection type | -| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor's IP address – defaults to ISP name if none is found | -| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | -| `mkt_clickid` | **{dotted-circle}** | string | The click ID | -| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | -| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | -| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | -| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | -| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | -| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | -| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn't be set by a tracker) | -| `os_family` | **{dotted-circle}** | string | Operating system family | -| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | -| `os_name` | **{dotted-circle}** | string | Name of operating system | -| `os_timezone` | **{dotted-circle}** | string | Client operating system timezone | -| `page_referrer` | **{dotted-circle}** | string | Referrer URL | -| `page_title` | **{dotted-circle}** | string | Page title | -| `page_url` | **{dotted-circle}** | string | Page URL | -| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | -| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | -| `page_urlpath` | **{dotted-circle}** | string | Path to page | -| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | -| `page_urlquery` | **{dotted-circle}** | string | Query string | -| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | -| `platform` | **{dotted-circle}** | string | The platform the app runs on | -| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | -| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | -| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | -| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | -| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | -| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | -| `refr_medium` | **{dotted-circle}** | string | Type of referer | -| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | -| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | -| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | -| `refr_urlhost` | **{dotted-circle}** | string | Referer host | -| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | -| `refr_urlport` | **{dotted-circle}** | integer | Referer port | -| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | -| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | -| `se_action` | **{dotted-circle}** | string | The action / event itself | -| `se_category` | **{dotted-circle}** | string | The category of event | -| `se_label` | **{dotted-circle}** | string | A label often used to refer to the 'object' the action is performed on | -| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | -| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | -| `ti_category` | **{dotted-circle}** | string | Item category | -| `ti_currency` | **{dotted-circle}** | string | Currency | -| `ti_name` | **{dotted-circle}** | string | Item name | -| `ti_orderid` | **{dotted-circle}** | string | Order ID | -| `ti_price` | **{dotted-circle}** | decimal | Item price | -| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | -| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | -| `ti_sku` | **{dotted-circle}** | string | Item SKU | -| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | -| `tr_city` | **{dotted-circle}** | string | Delivery address: city | -| `tr_country` | **{dotted-circle}** | string | Delivery address: country | -| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | -| `tr_orderid` | **{dotted-circle}** | string | Order ID | -| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | -| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | -| `tr_state` | **{dotted-circle}** | string | Delivery address: state | -| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | -| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | -| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | -| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | -| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | -| `txn_id` | **{dotted-circle}** | string | Transaction ID | -| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | -| `uploaded_at` | **{dotted-circle}** | | | -| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | -| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | -| `user_ipaddress` | **{dotted-circle}** | string | IP address | -| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | -| `v_collector` | **{dotted-circle}** | string | Collector version | -| `v_etl` | **{dotted-circle}** | string | ETL version | -| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | +- [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Service Ping Guide](../service_ping/index.md) +- [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-technology/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-technology/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-technology/data-team/platform/infrastructure/) diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 8edcbf06a0e..69fad1794e2 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -14,7 +14,7 @@ general best practices for code reviews, refer to our [code review guide](../cod ## Resources for reviewers - [Snowplow Guide](index.md) -- [Event Dictionary](dictionary.md) +- [Event Dictionary](https://metrics.gitlab.com/snowplow.html) ## Review process @@ -26,18 +26,18 @@ events or touches Snowplow related files. #### The merge request **author** should - For frontend events, when relevant, add a screenshot of the event in - the [testing tool](../snowplow/index.md#develop-and-test-snowplow) used. + the [testing tool](implementation.md#develop-and-test-snowplow) used. - For backend events, when relevant, add the output of the - [Snowplow Micro](index.md#snowplow-mini) good events + [Snowplow Micro](implementation.md#snowplow-micro) good events `GET http://localhost:9090/micro/good` (it might be a good idea to reset with `GET http://localhost:9090/micro/reset` first). - Update the [Event Dictionary](event_dictionary_guide.md). #### The Product Intelligence **reviewer** should -- Check that the [event taxonomy](../snowplow/index.md#structured-event-taxonomy) is correct. -- Check the [usage recommendations](../snowplow/index.md#usage-recommendations). +- Check that the [event taxonomy](index.md#structured-event-taxonomy) is correct. +- Check the [usage recommendations](implementation.md#usage-recommendations). - Check that the [Event Dictionary](event_dictionary_guide.md) is up-to-date. - If needed, check that the events are firing locally using one of the -[testing tools](../snowplow/index.md#develop-and-test-snowplow) available. +[testing tools](implementation.md#develop-and-test-snowplow) available. - Approve the MR, and relabel the MR with `~"product intelligence::approved"`. diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md new file mode 100644 index 00000000000..5b9e4f5256e --- /dev/null +++ b/doc/development/snowplow/schemas.md @@ -0,0 +1,166 @@ +--- +stage: Growth +group: Product Intelligence +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Snowplow schemas + +This page provides Snowplow schema reference for GitLab events. + +## `gitlab_standard` + +We are including the [`gitlab_standard` schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_standard/jsonschema/) with every event. See [Standardize Snowplow Schema](https://gitlab.com/groups/gitlab-org/-/epics/5218) for details. + +The [`StandardContext`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/tracking/standard_context.rb) class represents this schema in the application. + +| Field Name | Required | Type | Description | +|----------------|---------------------|-----------------------|---------------------------------------------------------------------------------------------| +| `project_id` | **{dotted-circle}** | integer | | +| `namespace_id` | **{dotted-circle}** | integer | | +| `environment` | **{check-circle}** | string (max 32 chars) | Name of the source environment, such as `production` or `staging` | +| `source` | **{check-circle}** | string (max 32 chars) | Name of the source application, such as `gitlab-rails` or `gitlab-javascript` | +| `plan` | **{dotted-circle}** | string (max 32 chars) | Name of the plan for the namespace, such as `free`, `premium`, or `ultimate`. Automatically picked from the `namespace`. | +| `google_analytics_id` | **{dotted-circle}** | string (max 32 chars) | Google Analytics ID, present when set from our marketing sites. | +| `extra` | **{dotted-circle}** | JSON | Any additional data associated with the event, in the form of key-value pairs | + +## Default Schema + +Frontend events include a [web-specific schema](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/#Web-specific_fields) provided by Snowplow. +All URLs are pseudonymized. The entity identifier [replaces](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracker-setup/other-parameters-2/#Setting_a_custom_page_URL_and_referrer_URL) personally identifiable +information (PII). PII includes usernames, group, and project names. + +| Field Name | Required | Type | Description | +|--------------------------|---------------------|-----------|----------------------------------------------------------------------------------------------------------------------------------| +| `app_id` | **{check-circle}** | string | Unique identifier for website / application | +| `base_currency` | **{dotted-circle}** | string | Reporting currency | +| `br_colordepth` | **{dotted-circle}** | integer | Browser color depth | +| `br_cookies` | **{dotted-circle}** | boolean | Does the browser permit cookies? | +| `br_family` | **{dotted-circle}** | string | Browser family | +| `br_features_director` | **{dotted-circle}** | boolean | Director plugin installed? | +| `br_features_flash` | **{dotted-circle}** | boolean | Flash plugin installed? | +| `br_features_gears` | **{dotted-circle}** | boolean | Google gears installed? | +| `br_features_java` | **{dotted-circle}** | boolean | Java plugin installed? | +| `br_features_pdf` | **{dotted-circle}** | boolean | Adobe PDF plugin installed? | +| `br_features_quicktime` | **{dotted-circle}** | boolean | Quicktime plugin installed? | +| `br_features_realplayer` | **{dotted-circle}** | boolean | RealPlayer plugin installed? | +| `br_features_silverlight` | **{dotted-circle}** | boolean | Silverlight plugin installed? | +| `br_features_windowsmedia` | **{dotted-circle}** | boolean | Windows media plugin installed? | +| `br_lang` | **{dotted-circle}** | string | Language the browser is set to | +| `br_name` | **{dotted-circle}** | string | Browser name | +| `br_renderengine` | **{dotted-circle}** | string | Browser rendering engine | +| `br_type` | **{dotted-circle}** | string | Browser type | +| `br_version` | **{dotted-circle}** | string | Browser version | +| `br_viewheight` | **{dotted-circle}** | string | Browser viewport height | +| `br_viewwidth` | **{dotted-circle}** | string | Browser viewport width | +| `collector_tstamp` | **{dotted-circle}** | timestamp | Time stamp for the event recorded by the collector | +| `contexts` | **{dotted-circle}** | | | +| `derived_contexts` | **{dotted-circle}** | | Contexts derived in the Enrich process | +| `derived_tstamp` | **{dotted-circle}** | timestamp | Timestamp making allowance for inaccurate device clock | +| `doc_charset` | **{dotted-circle}** | string | Web page's character encoding | +| `doc_height` | **{dotted-circle}** | string | Web page height | +| `doc_width` | **{dotted-circle}** | string | Web page width | +| `domain_sessionid` | **{dotted-circle}** | string | Unique identifier (UUID) for this visit of this user_id to this domain | +| `domain_sessionidx` | **{dotted-circle}** | integer | Index of number of visits that this user_id has made to this domain (The first visit is `1`) | +| `domain_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a first party cookie (so domain specific) | +| `dvce_created_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event occurred, as recorded by client device | +| `dvce_ismobile` | **{dotted-circle}** | boolean | Indicates whether device is mobile | +| `dvce_screenheight` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_screenwidth` | **{dotted-circle}** | string | Screen / monitor resolution | +| `dvce_sent_tstamp` | **{dotted-circle}** | timestamp | Timestamp when event was sent by client device to collector | +| `dvce_type` | **{dotted-circle}** | string | Type of device | +| `etl_tags` | **{dotted-circle}** | string | JSON of tags for this ETL run | +| `etl_tstamp` | **{dotted-circle}** | timestamp | Timestamp event began ETL | +| `event` | **{dotted-circle}** | string | Event type | +| `event_fingerprint` | **{dotted-circle}** | string | Hash client-set event fields | +| `event_format` | **{dotted-circle}** | string | Format for event | +| `event_id` | **{dotted-circle}** | string | Event UUID | +| `event_name` | **{dotted-circle}** | string | Event name | +| `event_vendor` | **{dotted-circle}** | string | The company who developed the event model | +| `event_version` | **{dotted-circle}** | string | Version of event schema | +| `geo_city` | **{dotted-circle}** | string | City of IP origin | +| `geo_country` | **{dotted-circle}** | string | Country of IP origin | +| `geo_latitude` | **{dotted-circle}** | string | An approximate latitude | +| `geo_longitude` | **{dotted-circle}** | string | An approximate longitude | +| `geo_region` | **{dotted-circle}** | string | Region of IP origin | +| `geo_region_name` | **{dotted-circle}** | string | Region of IP origin | +| `geo_timezone` | **{dotted-circle}** | string | Time zone of IP origin | +| `geo_zipcode` | **{dotted-circle}** | string | Zip (postal) code of IP origin | +| `ip_domain` | **{dotted-circle}** | string | Second level domain name associated with the visitor's IP address | +| `ip_isp` | **{dotted-circle}** | string | Visitor's ISP | +| `ip_netspeed` | **{dotted-circle}** | string | Visitor's connection type | +| `ip_organization` | **{dotted-circle}** | string | Organization associated with the visitor's IP address – defaults to ISP name if none is found | +| `mkt_campaign` | **{dotted-circle}** | string | The campaign ID | +| `mkt_clickid` | **{dotted-circle}** | string | The click ID | +| `mkt_content` | **{dotted-circle}** | string | The content or ID of the ad. | +| `mkt_medium` | **{dotted-circle}** | string | Type of traffic source | +| `mkt_network` | **{dotted-circle}** | string | The ad network to which the click ID belongs | +| `mkt_source` | **{dotted-circle}** | string | The company / website where the traffic came from | +| `mkt_term` | **{dotted-circle}** | string | Keywords associated with the referrer | +| `name_tracker` | **{dotted-circle}** | string | The tracker namespace | +| `network_userid` | **{dotted-circle}** | string | Unique identifier for a user, based on a cookie from the collector (so set at a network level and shouldn't be set by a tracker) | +| `os_family` | **{dotted-circle}** | string | Operating system family | +| `os_manufacturer` | **{dotted-circle}** | string | Manufacturers of operating system | +| `os_name` | **{dotted-circle}** | string | Name of operating system | +| `os_timezone` | **{dotted-circle}** | string | Client operating system time zone | +| `page_referrer` | **{dotted-circle}** | string | Referrer URL | +| `page_title` | **{dotted-circle}** | string | Page title | +| `page_url` | **{dotted-circle}** | string | Page URL | +| `page_urlfragment` | **{dotted-circle}** | string | Fragment aka anchor | +| `page_urlhost` | **{dotted-circle}** | string | Host aka domain | +| `page_urlpath` | **{dotted-circle}** | string | Path to page | +| `page_urlport` | **{dotted-circle}** | integer | Port if specified, 80 if not | +| `page_urlquery` | **{dotted-circle}** | string | Query string | +| `page_urlscheme` | **{dotted-circle}** | string | Scheme (protocol name) | +| `platform` | **{dotted-circle}** | string | The platform the app runs on | +| `pp_xoffset_max` | **{dotted-circle}** | integer | Maximum page x offset seen in the last ping period | +| `pp_xoffset_min` | **{dotted-circle}** | integer | Minimum page x offset seen in the last ping period | +| `pp_yoffset_max` | **{dotted-circle}** | integer | Maximum page y offset seen in the last ping period | +| `pp_yoffset_min` | **{dotted-circle}** | integer | Minimum page y offset seen in the last ping period | +| `refr_domain_userid` | **{dotted-circle}** | string | The Snowplow `domain_userid` of the referring website | +| `refr_dvce_tstamp` | **{dotted-circle}** | timestamp | The time of attaching the `domain_userid` to the inbound link | +| `refr_medium` | **{dotted-circle}** | string | Type of referer | +| `refr_source` | **{dotted-circle}** | string | Name of referer if recognised | +| `refr_term` | **{dotted-circle}** | string | Keywords if source is a search engine | +| `refr_urlfragment` | **{dotted-circle}** | string | Referer URL fragment | +| `refr_urlhost` | **{dotted-circle}** | string | Referer host | +| `refr_urlpath` | **{dotted-circle}** | string | Referer page path | +| `refr_urlport` | **{dotted-circle}** | integer | Referer port | +| `refr_urlquery` | **{dotted-circle}** | string | Referer URL query string | +| `refr_urlscheme` | **{dotted-circle}** | string | Referer scheme | +| `se_action` | **{dotted-circle}** | string | The action / event itself | +| `se_category` | **{dotted-circle}** | string | The category of event | +| `se_label` | **{dotted-circle}** | string | A label often used to refer to the 'object' the action is performed on | +| `se_property` | **{dotted-circle}** | string | A property associated with either the action or the object | +| `se_value` | **{dotted-circle}** | decimal | A value associated with the user action | +| `ti_category` | **{dotted-circle}** | string | Item category | +| `ti_currency` | **{dotted-circle}** | string | Currency | +| `ti_name` | **{dotted-circle}** | string | Item name | +| `ti_orderid` | **{dotted-circle}** | string | Order ID | +| `ti_price` | **{dotted-circle}** | decimal | Item price | +| `ti_price_base` | **{dotted-circle}** | decimal | Item price in base currency | +| `ti_quantity` | **{dotted-circle}** | integer | Item quantity | +| `ti_sku` | **{dotted-circle}** | string | Item SKU | +| `tr_affiliation` | **{dotted-circle}** | string | Transaction affiliation (such as channel) | +| `tr_city` | **{dotted-circle}** | string | Delivery address: city | +| `tr_country` | **{dotted-circle}** | string | Delivery address: country | +| `tr_currency` | **{dotted-circle}** | string | Transaction Currency | +| `tr_orderid` | **{dotted-circle}** | string | Order ID | +| `tr_shipping` | **{dotted-circle}** | decimal | Delivery cost charged | +| `tr_shipping_base` | **{dotted-circle}** | decimal | Shipping cost in base currency | +| `tr_state` | **{dotted-circle}** | string | Delivery address: state | +| `tr_tax` | **{dotted-circle}** | decimal | Transaction tax value (such as amount of VAT included) | +| `tr_tax_base` | **{dotted-circle}** | decimal | Tax applied in base currency | +| `tr_total` | **{dotted-circle}** | decimal | Transaction total value | +| `tr_total_base` | **{dotted-circle}** | decimal | Total amount of transaction in base currency | +| `true_tstamp` | **{dotted-circle}** | timestamp | User-set exact timestamp | +| `txn_id` | **{dotted-circle}** | string | Transaction ID | +| `unstruct_event` | **{dotted-circle}** | JSON | The properties of the event | +| `uploaded_at` | **{dotted-circle}** | | | +| `user_fingerprint` | **{dotted-circle}** | integer | User identifier based on (hopefully unique) browser features | +| `user_id` | **{dotted-circle}** | string | Unique identifier for user, set by the business using setUserId | +| `user_ipaddress` | **{dotted-circle}** | string | IP address | +| `useragent` | **{dotted-circle}** | string | User agent (expressed as a browser string) | +| `v_collector` | **{dotted-circle}** | string | Collector version | +| `v_etl` | **{dotted-circle}** | string | ETL version | +| `v_tracker` | **{dotted-circle}** | string | Identifier for Snowplow tracker | diff --git a/doc/development/sql.md b/doc/development/sql.md index 3483305c113..129280598fe 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -241,7 +241,7 @@ MergeRequest.where(source_project_id: Project.all.select(:id)) ``` The _only_ time you should use `pluck` is when you actually need to operate on -the values in Ruby itself (e.g. write them to a file). In almost all other cases +the values in Ruby itself (for example, writing them to a file). In almost all other cases you should ask yourself "Can I not just use a sub-query?". In line with our `CodeReuse/ActiveRecord` cop, you should only use forms like diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 7c518e9b6ca..a887558e473 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -1,6 +1,6 @@ --- -stage: Enablement -group: Infrastructure +stage: Platforms +group: Scalability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -58,6 +58,12 @@ component can have 2 indicators: [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) services, that threshold is **5 seconds**. + We're working on making this target configurable per endpoint in [this + project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525). Learn + how to [customize the request + apdex](application_slis/rails_request_apdex.md), this new apdex + measurement is not yet part of the error budget. + For Sidekiq job execution, the threshold depends on the [job urgency](sidekiq_style_guide.md#job-urgency). It is [currently](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L25-38) @@ -120,7 +126,7 @@ Inside a stage group dashboard, there are some notable components. Let's take th  -- By default, all the times are in UTC timezone. [We use UTC when communicating in Engineering](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). +- By default, all the times are in UTC time zone. [We use UTC when communicating in Engineering](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). - All metrics recorded in the GitLab production system have [1-year retention](https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/-/blob/31526b03fef823e2f9b3cda7c75dcd28a12418a3/attributes/prometheus.rb#L40). - Alternatively, you can zoom in or filter the time range directly on a graph. See the [Grafana Time Range Controls](https://grafana.com/docs/grafana/latest/dashboards/time-range-controls/) documentation for more information. diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 79664490368..52e89a10556 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -68,7 +68,7 @@ SILENCE_DEPRECATIONS=1 bin/rspec spec/models/project_spec.rb ### Test speed -GitLab has a massive test suite that, without [parallelization](ci.md#test-suite-parallelization-on-the-ci), can take hours +GitLab has a massive test suite that, without [parallelization](../pipelines.md#test-suite-parallelization), can take hours to run. It's important that we make an effort to write tests that are accurate and effective _as well as_ fast. diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index e3fccdcee34..de024084c9c 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -1,45 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../pipelines.md' +remove_date: '2022-01-12' --- -# GitLab tests in the Continuous Integration (CI) context +This file was moved to [another location](../pipelines.md). -## Test suite parallelization on the CI - -Our current CI parallelization setup is as follows: - -1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a - `knapsack/report-master.json` file: - - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` - (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. -1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with - `knapsack rspec` and should have an evenly distributed share of tests: - - It works because the jobs have access to the `knapsack/report-master.json` - since the "artifacts from all previous stages are passed by default". - - the jobs set their own report path to - `"knapsack/${TEST_TOOL}_${TEST_LEVEL}_${DATABASE}_${CI_NODE_INDEX}_${CI_NODE_TOTAL}_report.json"`. - - if knapsack is doing its job, test files that are run should be listed under - `Report specs`, not under `Leftover specs`. -1. The `update-tests-metadata` job (which only runs on scheduled pipelines for - [the canonical project](https://gitlab.com/gitlab-org/gitlab) takes all the - `knapsack/rspec*_pg_*.json` files and merge them all together into a single - `knapsack/report-master.json` file that is saved as artifact. - -After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. - -## Monitoring - -The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch -that includes `rspec-profile` in their name. - -## CI setup - -- Rails logging to `log/test.log` is disabled by default in CI [for - performance reasons](https://jtway.co/speed-up-your-rails-test-suite-by-6-in-1-line-13fedb869ec4). To override this setting, provide the - `RAILS_ENABLE_TEST_LOG` environment variable. - ---- - -[Return to Testing documentation](index.md) +<!-- This redirect file can be deleted after <2022-01-12>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index a3caa8bf2b3..9491c89c2a0 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -245,7 +245,11 @@ end ## Prefer `aggregate_failures` when there are back-to-back expectations -In cases where there must be multiple (back-to-back) expectations within a test case, it is preferable to use `aggregate_failures`. +See [Prefer aggregate failures when there are multiple expectations](#prefer-aggregate_failures-when-there-are-multiple-expectations) + +## Prefer `aggregate_failures` when there are multiple expectations + +In cases where there must be multiple expectations within a test case, it is preferable to use `aggregate_failures`. This allows you to group a set of expectations and see all the failures altogether, rather than having the test being aborted on the first failure. @@ -270,6 +274,32 @@ Page::Search::Results.perform do |search| end ``` +Attach the `:aggregate_failures` metadata to the example if multiple expectations are separated by statements. + +```ruby +#=> Good +it 'searches', :aggregate_failures do + Page::Search::Results.perform do |search| + expect(search).to have_file_in_project(template[:file_name], project.name) + + search.switch_to_code + + expect(search).to have_file_with_content(template[:file_name], content[0..33]) + end +end + +#=> Bad +it 'searches' do + Page::Search::Results.perform do |search| + expect(search).to have_file_in_project(template[:file_name], project.name) + + search.switch_to_code + + expect(search).to have_file_with_content(template[:file_name], content[0..33]) + end +end +``` + ## Prefer to split tests across multiple files Our framework includes a couple of parallelization mechanisms that work by executing spec files in parallel. @@ -333,11 +363,11 @@ after(:all) do end ``` -## Tag tests that require Administrator access +## Tag tests that require the Administrator role -We don't run tests that require Administrator access against our Production environments. +We don't run tests that require the Administrator role against our Production environments. -When you add a new test that requires Administrator access, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests. +When you add a new test that requires the Administrator role, apply the RSpec metadata `:requires_admin` so that the test will not be included in the test suites executed against Production and other environments on which we don't want to run those tests. When running tests locally or configuring a pipeline, the environment variable `QA_CAN_TEST_ADMIN_FEATURES` can be set to `false` to skip tests that have the `:requires_admin` tag. diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md index 6c504e6fa28..8770a5d33cd 100644 --- a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -52,7 +52,7 @@ Simply put, a required element is a visible HTML element that appears on a UI co "Visible" can be defined as -- Not having any CSS preventing its display. E.g.: `display: none` or `width: 0px; height: 0px;` +- Not having any CSS preventing its display, for example, `display: none` or `width: 0px; height: 0px;` - Being able to be interacted with by the user "UI component" can be defined as diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index c9acb2e9371..994ee3f253c 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -15,7 +15,7 @@ token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN and `GITLAB_ADMIN_PASSWORD`. Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments -where admin access is not available. +where administrator access is not available. WARNING: You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/index.md#feature-actors), diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 36c0f5adf00..b097d6b0729 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -184,7 +184,7 @@ in the pipeline: - Fetches these source files from all test jobs. - Generates and uploads the report to the `GCS` bucket `gitlab-qa-allure-report` under the project `gitlab-qa-resources`. -A common CI template for report uploading is stored in +A common CI template for report uploading is stored in [`allure-report.yml`](https://gitlab.com/gitlab-org/quality/pipeline-common/-/blob/master/ci/allure-report.yml). #### Merge requests diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md index 9ffa7ea4f77..85ab4d479f9 100644 --- a/doc/development/testing_guide/end_to_end/page_objects.md +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -105,7 +105,7 @@ code but **this is deprecated** in favor of the above method for two reasons: - Consistency: there is only one way to define an element - Separation of concerns: QA uses dedicated `data-qa-*` attributes instead of reusing code - or classes used by other components (e.g. `js-*` classes etc.) + or classes used by other components (for example, `js-*` classes etc.) ```ruby view 'app/views/my/view.html.haml' do diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index b6e92367f89..3749511fef5 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -26,6 +26,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:ldap_no_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS not enabled. | | `:ldap_tls` | The test requires a GitLab instance to be configured to use an external LDAP server with TLS enabled. | | `:mattermost` | The test requires a GitLab Mattermost service on the GitLab instance. | +| `:mixed_env` | The test should only be executed in environments that have a paired canary version available through traffic routing based on the existence of the `gitlab_canary=true` cookie. Tests in this category are switching the cookie mid-test to validate mixed deployment environments. | | `:object_storage` | The test requires a GitLab instance to be configured to use multiple [object storage types](../../../administration/object_storage.md). Uses MinIO as the object storage server. | | `:only` | The test is only to be run in specific execution contexts. See [test execution context selection](execution_context_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify the GitLab configuration (for example, Staging). | @@ -34,7 +35,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:relative_url` | The test requires a GitLab instance to be installed under a [relative URL](../../../install/relative_url.md). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | | `:repository_storage` | The test requires a GitLab instance to be configured to use multiple [repository storage paths](../../../administration/repository_storage_paths.md). Paired with the `:orchestrated` tag. | -| `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. | +| `:requires_admin` | The test requires an administrator account. Tests with the tag are excluded when run against Canary and Production environments. | | `:requires_git_protocol_v2` | The test requires that Git protocol version 2 is enabled on the server. It's assumed to be enabled by default but if not the test can be skipped by setting `QA_CAN_TEST_GIT_PROTOCOL_V2` to `false`. | | `:requires_praefect` | The test requires that the GitLab instance uses [Gitaly Cluster](../../../administration/gitaly/praefect.md) (a.k.a. Praefect) as the repository storage . It's assumed to be used by default but if not the test can be skipped by setting `QA_CAN_TEST_PRAEFECT` to `false`. | | `:runner` | The test depends on and sets up a GitLab Runner instance, typically to run a pipeline. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 46a3053c267..eadd0ef49a0 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -45,7 +45,7 @@ docker run \ Jenkins is available on `http://localhost:8080`. -Admin username is `admin` and password is `password`. +Administrator username is `admin` and password is `password`. It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5) only to prevent it from running in the pipelines for live environments such as Staging. @@ -167,9 +167,9 @@ The following includes more information on the command: -`QA_DEBUG` - Set to `true` to verbosely log page object actions. -`WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. --`GITLAB_ADMIN_USERNAME` - Admin username to use when adding a license. --`GITLAB_ADMIN_PASSWORD` - Admin password to use when adding a license. --`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with admin access. Used for API access as an admin during tests. +-`GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. +-`GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. +-`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. -`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. -`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK -`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs @@ -410,9 +410,9 @@ Tests that are tagged with `:ldap_tls` and `:ldap_no_tls` meta are orchestrated These tests spin up a Docker container [(`osixia/openldap`)](https://hub.docker.com/r/osixia/openldap) running an instance of [OpenLDAP](https://www.openldap.org/). The container uses fixtures [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap) to create -base data such as users and groups including the admin group. The password for [all users](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/2_add_users.ldif) including [the `tanuki` user](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/tanuki.ldif) is `password`. +base data such as users and groups including the administrator group. The password for [all users](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/2_add_users.ldif) including [the `tanuki` user](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/9ffb9ad3be847a9054967d792d6772a74220fb42/fixtures/ldap/tanuki.ldif) is `password`. -A GitLab instance is also created in a Docker container based on our [General LDAP setup](../../../administration/auth/ldap/index.md#general-ldap-setup) documentation. +A GitLab instance is also created in a Docker container based on our [LDAP setup](../../../administration/auth/ldap/index.md) documentation. Tests that are tagged `:ldap_tls` enable TLS on GitLab using the certificate [checked into the GitLab-QA repository](https://gitlab.com/gitlab-org/gitlab-qa/-/tree/9ffb9ad3be847a9054967d792d6772a74220fb42/tls_certificates/gitlab). diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index bfcd68dbaf3..9489020de5d 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -37,9 +37,9 @@ bin/rspec --tag quarantine Once a test is in quarantine, there are 3 choices: -- Should the test be fixed (i.e. get rid of its flakiness)? +- Should the test be fixed (that is, get rid of its flakiness)? - Should the test be moved to a lower level of testing? -- Should the test be removed entirely (e.g. because there's already a +- Should the test be removed entirely (for example, because there's already a lower-level test, or it's duplicating another same-level test, or it's testing too much etc.)? diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 76687db3a3f..0e721ba2760 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -146,7 +146,7 @@ it('does not display a dropdown if no metricTypes exist', () => { }); ``` -Keep an eye out for these kinds of tests, as they just make updating logic more fragile and tedious than it needs to be. This is also true for other libraries. A rule of thumb here is: if you are checking a `wrapper.vm` property, you should probably stop and rethink the test to check the rendered template instead. +Keep an eye out for these kinds of tests, as they just make updating logic more fragile and tedious than it needs to be. This is also true for other libraries. A suggestion here is: if you are checking a `wrapper.vm` property, you should probably stop and rethink the test to check the rendered template instead. Some more examples can be found in the [Frontend unit tests section](testing_levels.md#frontend-unit-tests) @@ -783,20 +783,25 @@ often using fixtures to validate correct integration with the backend code. ### Use fixtures -Jest uses `spec/frontend/__helpers__/fixtures.js` to import fixtures in tests. - -The following are examples of tests that work for Jest: +To import a JSON fixture, `import` it using the `test_fixtures` alias. ```javascript +import responseBody from 'test_fixtures/some/fixture.json' // loads spec/frontend/fixtures/some/fixture.json + it('makes a request', () => { - const responseBody = getJSONFixture('some/fixture.json'); // loads spec/frontend/fixtures/some/fixture.json axiosMock.onGet(endpoint).reply(200, responseBody); myButton.click(); // ... }); +``` + +For other fixtures, Jest uses `spec/frontend/__helpers__/fixtures.js` to import them in tests. +The following are examples of tests that work for Jest: + +```javascript it('uses some HTML element', () => { loadFixtures('some/page.html'); // loads spec/frontend/fixtures/some/page.html and adds it to the DOM @@ -843,10 +848,6 @@ describe GraphQL::Query, type: :request do all_releases_query_path = 'releases/graphql/queries/all_releases.query.graphql' - before(:all) do - clean_frontend_fixtures('graphql/releases/') - end - it "graphql/#{all_releases_query_path}.json" do query = get_graphql_query_as_string(all_releases_query_path) @@ -860,7 +861,7 @@ end This will create a new fixture located at `tmp/tests/frontend/fixtures-ee/graphql/releases/graphql/queries/all_releases.query.graphql.json`. -You can import the JSON fixture in a Jest test using the `getJSONFixture` method +You can import the JSON fixture in a Jest test using the `test_fixtures` alias [as described below](#use-fixtures). ## Data-driven tests @@ -998,7 +999,7 @@ it like so: import Subject from '~/feature/the_subject.vue'; // Force Jest to transpile and cache -// eslint-disable-next-line import/order, no-unused-vars +// eslint-disable-next-line no-unused-vars import _Thing from '~/feature/path/to/thing.vue'; ``` diff --git a/doc/development/testing_guide/img/review-app-parent-pipeline.png b/doc/development/testing_guide/img/review-app-parent-pipeline.png Binary files differnew file mode 100644 index 00000000000..5686d5f6ebe --- /dev/null +++ b/doc/development/testing_guide/img/review-app-parent-pipeline.png diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 015d8a92a4d..2e00a00c454 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -48,7 +48,7 @@ testing promises, stubbing etc. What are flaky tests, the different kind of flaky tests we encountered, and what we do about them. -## [GitLab tests in the Continuous Integration (CI) context](ci.md) +## [GitLab pipelines](../pipelines.md) How GitLab test suite is run in the CI context: setup, caches, artifacts, parallelization, monitoring. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 72d63fd8194..4091f213a8f 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -6,8 +6,98 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Review Apps -Review Apps are automatically deployed by [the -pipeline](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6665). +Review Apps are deployed using the `start-review-app-pipeline` job. This job triggers a child pipeline containing a series of jobs to perform the various tasks needed to deploy a Review App. + + + +For any of the following scenarios, the `start-review-app-pipeline` job would be automatically started: + +- for merge requests with CI config changes +- for merge requests with frontend changes +- for merge requests with QA changes +- for scheduled pipelines + +## QA runs on Review Apps + +On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the +`review` stage), the `review-qa-smoke` job is automatically started and it runs +the QA smoke suite. + +You can also manually start the `review-qa-all`: it runs the full QA suite. + +After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by +the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. + +## Performance Metrics + +On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the +`review-performance` job is automatically started: this job does basic +browser performance testing using a +[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). + +## How to + +### Get access to the GCP Review Apps cluster + +You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) +for the `gcp-review-apps-dev` GCP group and role. + +This grants you the following permissions for: + +- [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). +- [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). + +### Log into my Review App + +For GitLab Team Members only. If you want to sign in to the review app, review +the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). + +- The default username is `root`. +- The password can be found in the 1Password login item named `GitLab EE Review App`. + +### Enable a feature flag for my Review App + +1. Open your Review App and log in as documented above. +1. Create a personal access token. +1. Enable the feature flag using the [Feature flag API](../../api/features.md). + +### Find my Review App slug + +1. Open the `review-deploy` job. +1. Look for `** Deploying review-*`. +1. For instance for `** Deploying review-1234-abc-defg... **`, + your Review App slug would be `review-1234-abc-defg` in this case. + +### Run a Rails console + +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), + e.g. `review-qa-raise-e-12chm0`. +1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. +1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. +1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. +1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the + default command or + - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and + - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` + with your Pod's name. + +### Dig into a Pod's logs + +1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), + e.g. `review-qa-raise-e-12chm0`. +1. Find and open the `migrations` Deployment, e.g. + `review-qa-raise-e-12chm0-migrations.1`. +1. Click on the Pod in the "Managed pods" section, e.g. + `review-qa-raise-e-12chm0-migrations.1-nqwtx`. +1. Click on the `Container logs` link. + +Alternatively, you could use the [Logs Explorer](https://console.cloud.google.com/logs/query;query=?project=gitlab-review-apps) which provides more utility to search logs. An example query for a pod name is as follows: + +```shell +resource.labels.pod_name:"review-qa-raise-e-12chm0-migrations" +``` ## How does it work? @@ -89,7 +179,7 @@ subgraph "CNG-mirror pipeline" **Additional notes:** -- If the `review-deploy` job keep failing (note that we already retry it twice), +- If the `review-deploy` job keeps failing (and a manual retry didn't help), please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~bug` issue with a link to your merge request. Note that the deployment failure can reveal an actual problem introduced in your merge request (i.e. this isn't @@ -105,7 +195,7 @@ subgraph "CNG-mirror pipeline" stop a Review App manually, and is also started by GitLab once a merge request's branch is deleted after being merged. - The Kubernetes cluster is connected to the `gitlab` projects using the - [GitLab Kubernetes integration](../../user/project/clusters/index.md). This basically + [GitLab Kubernetes integration](../../user/infrastructure/clusters/index.md). This basically allows to have a link to the Review App directly from the merge request widget. ### Auto-stopping of Review Apps @@ -126,24 +216,6 @@ The `review-gcp-cleanup` job that automatically runs in scheduled pipelines (and is manual in merge request) removes any dangling GCP network resources that were not removed along with the Kubernetes resources. -## QA runs - -On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage (which comes after the -`review` stage), the `review-qa-smoke` job is automatically started and it runs -the QA smoke suite. - -You can also manually start the `review-qa-all`: it runs the full QA suite. - -After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by -the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. - -## Performance Metrics - -On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the -`review-performance` job is automatically started: this job does basic -browser performance testing using a -[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). - ## Cluster configuration The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project. @@ -157,64 +229,6 @@ The Helm version used is defined in the [`registry.gitlab.com/gitlab-org/gitlab-build-images:gitlab-helm3-kubectl1.14` image](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/Dockerfile.gitlab-helm3-kubectl1.14#L7) used by the `review-deploy` and `review-stop` jobs. -## How to - -### Get access to the GCP Review Apps cluster - -You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) -for the `gcp-review-apps-dev` GCP group and role. - -This grants you the following permissions for: - -- [Retrieving pod logs](#dig-into-a-pods-logs). Granted by [Viewer (`roles/viewer`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). -- [Running a Rails console](#run-a-rails-console). Granted by [Kubernetes Engine Developer (`roles/container.pods.exec`)](https://cloud.google.com/iam/docs/understanding-roles#kubernetes-engine-roles). - -### Log into my Review App - -For GitLab Team Members only. If you want to sign in to the review app, review -the GitLab handbook information for the [shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). - -- The default username is `root`. -- The password can be found in the 1Password login item named `GitLab EE Review App`. - -### Enable a feature flag for my Review App - -1. Open your Review App and log in as documented above. -1. Create a personal access token. -1. Enable the feature flag using the [Feature flag API](../../api/features.md). - -### Find my Review App slug - -1. Open the `review-deploy` job. -1. Look for `** Deploying review-*`. -1. For instance for `** Deploying review-1234-abc-defg... **`, - your Review App slug would be `review-1234-abc-defg` in this case. - -### Run a Rails console - -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.exec` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. -1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. -1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. -1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the - default command or - - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and - - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` - with your Pod's name. - -### Dig into a Pod's logs - -1. Make sure you [have access to the cluster](#get-access-to-the-gcp-review-apps-cluster) and the `container.pods.getLogs` permission first. -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), - e.g. `review-qa-raise-e-12chm0`. -1. Find and open the `migrations` Deployment, e.g. - `review-qa-raise-e-12chm0-migrations.1`. -1. Click on the Pod in the "Managed pods" section, e.g. - `review-qa-raise-e-12chm0-migrations.1-nqwtx`. -1. Click on the `Container logs` link. - ## Diagnosing unhealthy Review App releases If [Review App Stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399) diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md index e7e8464ff7a..810c789bc03 100644 --- a/doc/development/usage_ping/dictionary.md +++ b/doc/development/usage_ping/dictionary.md @@ -1,4 +1,4 @@ --- -redirect_to: 'https://gitlab-org.gitlab.io/growth/product-intelligence/metric-dictionary' +redirect_to: 'https://metrics.gitlab.com/index.html' remove_date: '2021-11-10' --- diff --git a/doc/development/usage_ping/index.md b/doc/development/usage_ping/index.md deleted file mode 100644 index aa06cb36f0c..00000000000 --- a/doc/development/usage_ping/index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../service_ping/index.md' -remove_date: '2021-10-09' ---- - -This file was moved to [another location](../service_ping/index.md). - -<!-- This redirect file can be deleted after <2021-10-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/usage_ping/metrics_dictionary.md b/doc/development/usage_ping/metrics_dictionary.md deleted file mode 100644 index 3743c2e0414..00000000000 --- a/doc/development/usage_ping/metrics_dictionary.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../service_ping/metrics_dictionary.md' -remove_date: '2021-10-09' ---- - -This file was moved to [another location](../service_ping/metrics_dictionary.md). - -<!-- This redirect file can be deleted after <2021-10-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/usage_ping/metrics_instrumentation.md b/doc/development/usage_ping/metrics_instrumentation.md deleted file mode 100644 index f2d731803b8..00000000000 --- a/doc/development/usage_ping/metrics_instrumentation.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../service_ping/metrics_instrumentation.md' -remove_date: '2021-10-09' ---- - -This file was moved to [another location](../service_ping/metrics_instrumentation.md). - -<!-- This redirect file can be deleted after <2021-10-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/usage_ping/product_intelligence_review.md b/doc/development/usage_ping/product_intelligence_review.md deleted file mode 100644 index dc51e3e300a..00000000000 --- a/doc/development/usage_ping/product_intelligence_review.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../service_ping/review_guidelines.md' -remove_date: '2021-10-09' ---- - -This file was moved to [another location](../service_ping/review_guidelines.md). - -<!-- This redirect file can be deleted after <2021-10-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/usage_ping/review_guidelines.md b/doc/development/usage_ping/review_guidelines.md deleted file mode 100644 index dc51e3e300a..00000000000 --- a/doc/development/usage_ping/review_guidelines.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: '../service_ping/review_guidelines.md' -remove_date: '2021-10-09' ---- - -This file was moved to [another location](../service_ping/review_guidelines.md). - -<!-- This redirect file can be deleted after <2021-10-09>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/windows.md b/doc/development/windows.md index 07f8a80e95f..fb095b68939 100644 --- a/doc/development/windows.md +++ b/doc/development/windows.md @@ -33,7 +33,7 @@ tags: - windows-1809 ``` -A list of software preinstalled on the Windows images is available at: [Preinstalled software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). +A list of software preinstalled on the Windows images is available at: [Preinstalled software](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). ## GCP Windows image for development @@ -57,7 +57,7 @@ Build a Google Cloud image with the above shared runners repository by doing the 1. Clone the repository <https://github.com/rgl/packer-provisioner-windows-update> and `cd` into the cloned directory. 1. Run the command `go build -o packer-provisioner-windows-update` (requires `go` to be installed). 1. Verify `packer-provisioner-windows-update` is in the `PATH` environment variable. -1. Add all [required environment variables](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/-/blob/master/packer.json#L2-10) +1. Add all [required environment variables](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/-/blob/main/packer.json#L2-10) in the `packer.json` file to your environment (perhaps use [`direnv`](https://direnv.net/)). 1. Build the image by running the command: `packer build packer.json`. @@ -136,7 +136,7 @@ PowerShell has aliases for all of the following commands so you don't have to le - `/` ---> `\` (path separator) - `cat` ---> `type` - `mv` ---> `move` -- Redirection works the same (i.e. `>` and `2>&1`) +- Redirection works the same (for example, `>` and `2>&1`) - `.\some.exe` to call a local executable - curl is available - `..` and `.` are available diff --git a/doc/index.md b/doc/index.md index cb8b0e67f7d..a638647e3e3 100644 --- a/doc/index.md +++ b/doc/index.md @@ -40,7 +40,7 @@ Have a look at some of our most popular topics: |:-------------------------------------------------------------------------------------------|:------------| | [Two-factor authentication](user/profile/account/two_factor_authentication.md) | Improve the security of your GitLab account. | | [GitLab groups](user/group/index.md) | Manage projects together. | -| [GitLab CI/CD pipeline configuration reference](ci/yaml/index.md) | Available configuration options for `.gitlab-ci.yml` files. | +| [Keyword reference for the `.gitlab-ci.yml` file](ci/yaml/index.md) | Available configuration options for `.gitlab-ci.yml` files. | | [Activate GitLab EE with a license](user/admin_area/license.md) | Activate GitLab Enterprise Edition functionality with a license. | | [Back up and restore GitLab](raketasks/backup_restore.md) | Rake tasks for backing up and restoring GitLab self-managed instances. | | [GitLab release and maintenance policy](policy/maintenance.md) | Policies for version naming and cadence, and also upgrade recommendations. | @@ -108,7 +108,7 @@ There are many ways to integrate with GitLab, including: | Topic | Description | |:-------------------------------------------|:------------| -| [GitLab REST API](api/README.md) | Integrate with GitLab using our REST API. | +| [GitLab REST API](api/index.md) | Integrate with GitLab using our REST API. | | [GitLab GraphQL API](api/graphql/index.md) | Integrate with GitLab using our GraphQL API. | | [Integrations](integration/index.md) | Integrations with third-party products. | diff --git a/doc/install/aws/eks_clusters_aws.md b/doc/install/aws/eks_clusters_aws.md index 95f9f81f601..3c19a83f128 100644 --- a/doc/install/aws/eks_clusters_aws.md +++ b/doc/install/aws/eks_clusters_aws.md @@ -17,15 +17,14 @@ This documentation is not for clusters for deployment of GitLab itself, but inst Information on deploying GitLab onto EKS can be found in [Provisioning GitLab Cloud Native Hybrid on AWS EKS](gitlab_hybrid_on_aws.md). -## Use AWS EKS quick start or `eksctl` +## Use `eksctl` -Using the EKS Quick Start or `eksctl` enables the following when building an EKS Cluster: +Using `eksctl` enables the following when building an EKS Cluster: -- It can be part of CloudFormation IaC or [CLI (`eksctl`)](https://eksctl.io/) automation - You have various cluster configuration options: - Selection of operating system: Amazon Linux 2, Windows, Bottlerocket - Selection of Hardware Architecture: x86, ARM, GPU - - Selection of Fargate backend + - Selection of Kubernetes version (the GitLab-managed clusters for your project's applications have [specific Kubernetes version requirements](../../user/infrastructure/clusters/connect/index.md#supported-cluster-versions)) - It can deploy high value-add items to the cluster, including: - A bastion host to keep the cluster endpoint private and possible perform performance testing. - Prometheus and Grafana for monitoring. diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index 9f53f333463..4d22a29ad0a 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -15,13 +15,21 @@ Amazon provides a managed Kubernetes service offering known as [Amazon Elastic K ## Tested AWS Bill of Materials by reference architecture size -| GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Perf Test Results Omnibus on Instances | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs | +| GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Perf Test Results Omnibus on Instances | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs* | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| [2K Omnibus](../../administration/reference_architectures/2k_users.md) | [2K Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k) | [2K Cloud Native Hybrid on EKS](#2k-cloud-native-hybrid-on-eks) | GPT Test Results | 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs<br />(2 AZ Cost Estimate is in BOM Below) | -| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs<br />(2 AZ Cost Estimate is in BOM Below) | -| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs | +| [2K Omnibus](../../administration/reference_architectures/2k_users.md) | [2K Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k) | [2K Cloud Native Hybrid on EKS](#2k-cloud-native-hybrid-on-eks) | GPT Test Results | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=544bcf1162beae6b8130ad257d081cdf9d4504e3)<br />(2 AZ Cost Estimate is in BOM Below) | +| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | +| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | | [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K AutoScale GPT Test Results](hhttps://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | -| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | 10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs | +| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | + +\*Cost calculations for actual implementations are a rough guideline with the following considerations: + +- Actual choices about instance types should be based on GPT testing of your configuration. +- The first year of actual usage will reveal potential savings due to lower than expected usage, especially for ramping migrations where the full loading takes months, so be careful not to commit to savings plans too early or for too long. +- The cost estimates assume full scale of the Kubernetes cluster nodes 24 x 7 x 365. Savings due to 'idling scale-in' are not considered because they are highly dependent on the usage patterns of the specific implementation. +- Costs such as GitLab Runners, data egress and storage costs are not included as they are very dependent on the configuration of a specific implementation and on development behaviors (for example, frequency of committing or frequency of builds). +- These estimates will change over time as GitLab tests and optimizes compute choices. ## Available Infrastructure as Code for GitLab Cloud Native Hybrid @@ -45,9 +53,16 @@ It is helpful to review the [GitLab Environment Toolkit (GET) Issues](https://gi | Results in a Ready-to-Use instance | Yes | Manual Actions or <br />Supplemental IaC Required | | **<u>Configuration Features</u>** | | | | Can deploy Omnibus GitLab (non-Kubernetes | No | Yes | +| Results in a self-healing Gitaly Cluster configuration | Yes | No | | Complete Internal Encryption | 85%, Targeting 100% | Manual | | AWS GovCloud Support | Yes | TBD | +### Two and Three Zone High Availability + +While GitLab Reference Architectures generally encourage three zone redundancy, AWS Quick Starts and AWS Well Architected consider two zone redundancy as AWS Well Architected. Individual implementations should weigh the costs of two and three zone configurations against their own high availability requirements for a final configuration. + +Gitaly Cluster uses a consistency voting system to implement strong consistency between synchronized nodes. Regardless of the number of availability zones implemented, there will always need to be a minimum of three Gitaly and three Praefect nodes in the cluster to avoid voting stalemates cause by an even number of nodes. + ### Streamlined Performance Testing of AWS Quick Start Prepared GitLab Instances A set of performance testing instructions have been abbreviated for testing a GitLab instance prepared using the AWS Quick Start for GitLab Cloud Native Hybrid on EKS. They assume zero familiarity with GitLab Performance Tool. They can be accessed here: [Performance Testing an Instance Prepared using AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/Easy-DIY-Perf-Testing.md). diff --git a/doc/install/aws/gitlab_sre_for_aws.md b/doc/install/aws/gitlab_sre_for_aws.md index a2d3a2d0295..06e3bf784bd 100644 --- a/doc/install/aws/gitlab_sre_for_aws.md +++ b/doc/install/aws/gitlab_sre_for_aws.md @@ -9,7 +9,7 @@ type: index # GitLab Site Reliability Engineering for AWS **(FREE SELF)** -## Known issues list +## AWS known issues list Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Please consult individual issues to understand GitLabs stance and plans on any given known issue. @@ -17,11 +17,11 @@ See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/a ## Gitaly SRE considerations -Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries. Here is indepth technical reading on the topic: +Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is indepth technical reading on the topic: ### Why Gitaly was built -Below are some links to better understand why Gitaly was built: +If you would like to understand the underlying rationale on why GitLab had to invest in creating Gitaly, read the following minimal list of topics: - [Git characteristics that make horizontal scaling difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult) - [Git architectural characteristics and assumptions](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-architectural-characteristics-and-assumptions) @@ -36,19 +36,63 @@ As part of Gitaly cluster consistency, Praefect nodes will occasionally need to Complete performance metrics should be collected for Gitaly instances for identification of bottlenecks, as they could have to do with disk IO, network IO or memory. -Gitaly must be implemented on instance compute. +### Gitaly performance guidelines -### Gitaly EBS volume sizing guidelines +Gitaly functions as the primary Git Repository Storage in GitLab. However, it's not simply a streaming file server. It also does a lot of demanding computing work, such as preparing and caching Git pack files which informs some of the performance recommendations below. -Gitaly storage is expected to be local (not NFS of any type including EFS). -Gitaly servers also need disk space for building and caching Git pack files. +NOTE: +All recommendations are for production configurations, including performance testing. For test configurations, like training or functional testing, you can use less expensive options. However, you should adjust or rebuild if performance is an issue. -Background: +#### Overall recommendations -- When not using provisioned EBS IO, EBS volume size determines the IO level, so provisioning volumes that are much larger than needed can be the least expensive way to improve EBS IO. -- Only use nitro instance types due to higher IO and EBS optimization. -- Use Amazon Linux 2 to ensure the best disk and memory optimizations (for example, ENA network adapters and drivers). -- If GitLab backup scripts are used, they need a temporary space location large enough to hold 2 times the current size of the Git File system. If that will be done on Gitaly servers, separate volumes should be used. +- Production-grade Gitaly must be implemented on instance compute due to all of the above and below characteristics. +- Never use [burstable instance types](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html) (such as `t2`, `t3`, `t4g`) for Gitaly. +- Always use at least the [AWS Nitro generation of instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances) to ensure many of the below concerns are automatically handled. +- Use Amazon Linux 2 to ensure that all [AWS oriented hardware and OS optimizations](https://aws.amazon.com/amazon-linux-2/faqs/) are maximized without additional configuration or SRE management. + +#### CPU and memory recommendations + +- The general GitLab Gitaly node recommendations for CPU and Memory assume relatively even loading across repositories. GPT testing of any non-characteristic repositories and/or SRE monitoring of Gitaly metrics may inform when to choose memory and/or CPU higher than general recommendations. + +**To accommodate:** + +- Git Pack file operations are memory and CPU intensive. +- If repository commit traffic is dense, large, or very frequent, then more CPU and Memory are required to handle the load. Patterns such as storing binaries and/or busy or large monorepos are examples that can cause high loading. + +#### Disk I/O recommendations + +- Use only SSD storage and the [class of EBS storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) that suites your durability and speed requirements. +- When not using provisioned EBS IO, EBS volume size determines the I/O level, so provisioning volumes that are much larger than needed can be the least expensive way to improve EBS IO. +- If Gitaly performance monitoring shows signs of disk stress then one of the provisioned IOPs levels can be chosen. Note that EBS IOPs levels also have enhanced durability which may be appealing for some implementations aside from performance considerations. + +**To accommodate:** + +- Gitaly storage is expected to be local (not NFS of any type including EFS). +- Gitaly servers also need disk space for building and caching Git pack files. This is above and beyond the permanent storage of your Git Repositories. +- Git Pack files are cached in Gitaly. Creation of pack files in temporary disk benefits from fast disk, and disk caching of pack files benefits from ample disk space. + +#### Network I/O recommendations + +- Use only instance types [from the list of ones that support ENA advanced networking]( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecking. +- Choose instances with sizes with more than 10 Gbps - but only if needed and only when having proven a node level network bottleneck with monitoring and/or stress testing. + +**To accommodate:** + +- Gitaly nodes do the main work of streaming repositories for push and pull operations (to add development endpoints, and to CI/CD). +- Gitaly servers need reasonable low latency between cluster nodes and with Praefect services in order for the cluster to maintain operational and data integrity. +- Gitaly nodes should be selected with network bottlenecking avoidance as a primary consideration. +- Gitaly nodes should be monitored for network saturation. +- Not all networking issues can be solved through optimizing the node level networking: + - Gitaly cluster node replication depends on all networking between nodes. + - Gitaly networking performance to pull and push endpoints depends on all networking in between. + +### AWS Gitaly backup + +Due to the nature of how Praefect tracks the replication metadata of Gitaly disk information, the best backup method is [the official backup and restore Rake tasks](../../raketasks/backup_restore.md). + +### AWS Gitaly recovery + +Gitaly Cluster does not support snapshot backups as these can cause issues where the Praefect database becomes out of syn with the disk storage. Due to the nature of how Praefect rebuilds the replication metadata of Gitaly disk information during a restore, the best recovery method is [the official backup and restore Rake tasks](../../raketasks/backup_restore.md). ### Gitaly HA in EKS quick start diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 342b6962628..80193d882dd 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -87,7 +87,7 @@ Implementation patterns may also provide specialized implementations beyond the For example: -- Small, self-contained GitLab instances for per-person admin training, perhaps on Kubernetes so that a deployment cluster is self-contained as well. +- Small, self-contained GitLab instances for per-person administration training, perhaps on Kubernetes so that a deployment cluster is self-contained as well. - GitLab Runner implementation patterns, including using platform-specific PaaS. ## Intended audiences and contributors diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index a490cf0eb73..2ad54a17715 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -7,15 +7,29 @@ type: howto {::options parse_block_html="true" /} -# Installing GitLab on Amazon Web Services (AWS) (DEPRECATED) **(FREE SELF)** +# Installing a GitLab POC on Amazon Web Services (AWS) **(FREE SELF)** This page offers a walkthrough of a common configuration for GitLab on AWS using the official GitLab Linux package. You should customize it to accommodate your needs. NOTE: -For organizations with 1,000 users or less, the recommended AWS installation method is to launch an EC2 single box [Omnibus Installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. See the [1,000 user reference architecture](../../administration/reference_architectures/1k_users.md) for more. +For organizations with 1,000 users or less, the recommended AWS installation method is to launch an EC2 single box [Omnibus Installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. See the [1,000 user reference architecture](../../administration/reference_architectures/1k_users.md) for more information. + +## Getting started for production-grade GitLab NOTE: -The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is GitLabs internal effort to create a multi-cloud, multi-GitLab toolkit to provision GitLab. It can be used to deploy Omnibus GitLab on AWS. GET is developed by GitLab developers and is open to community contributions. +This document is an installation guide for a proof of concept instance. It is not a reference architecture and it does not result in a highly available configuration. + +Following this guide exactly results in a proof of concept instance that roughly equates to a **scaled down** version of a **two availability zone implementation** of the **Non-HA** [Omnibus 2000 User Reference Architecture](../../administration/reference_architectures/2k_users.md). The 2K reference architecture is not HA because it is primarily intended to provide some scaling while keeping costs and complexity low. The [3000 User Reference Architecture](../../administration/reference_architectures/3k_users.md) is the smallest size that is GitLab HA. It has additional service roles to achieve HA, most notably it uses Gitaly Cluster to achieve HA for Git repository storage and specifies triple redundancy. + +GitLab maintains and tests two main types of Reference Architectures. The **Omnibus architectures** are implemented on instance compute while **Cloud Native Hybrid architectures** maximize the use of a Kubernetes cluster. Cloud Native Hybrid reference architecture specifications are addendum sections to the Reference Architecture size pages that start by describing the Omnibus architecture. For example, the 3000 User Cloud Native Reference Architecture is in the subsection titled [Cloud Native Hybrid reference architecture with Helm Charts (alternative)](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) in the 3000 User Reference Architecture page. + +### Getting started for production-grade Omnibus GitLab + +The Infrastructure as Code tooling [GitLab Environment Tool (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is the best place to start for building Omnibus GitLab on AWS and most especially if you are targeting an HA setup. While it does not automate everything, it does complete complex setups like Gitaly Cluster for you. GET is open source so anyone can build on top of it and contribute improvements to it. + +### Getting started for production-grade Cloud Native Hybrid GitLab + +For the Cloud Native Hybrid architectures there are two Infrastructure as Code options which are compared in GitLab Cloud Native Hybrid on AWS EKS implementation pattern in the section [Available Infrastructure as Code for GitLab Cloud Native Hybrid](gitlab_hybrid_on_aws.md#available-infrastructure-as-code-for-gitlab-cloud-native-hybrid). It compares the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) to the AWS Quick Start for GitLab Cloud Native Hybrid on EKS which was codeveloped by GitLab and AWS. GET and the AWS Quick Start are both open source so anyone can build on top of them and contribute improvements to them. ## Introduction @@ -517,7 +531,7 @@ gitlab=# \q ```ruby # Disable the built-in Postgres postgresql['enable'] = false - + # Fill in the connection details gitlab_rails['db_adapter'] = "postgresql" gitlab_rails['db_encoding'] = "unicode" @@ -533,7 +547,7 @@ gitlab=# \q ```ruby # Disable the built-in Redis redis['enable'] = false - + # Fill in the connection details gitlab_rails['redis_host'] = "<redis-endpoint>" gitlab_rails['redis_port'] = 6379 @@ -734,7 +748,7 @@ Read more on configuring an ## Backup and restore -GitLab provides [a tool to back up](../../raketasks/backup_restore.md#back-up-gitlab) +GitLab provides [a tool to back up](../../raketasks/backup_restore.md) and restore its Git data, database, attachments, LFS objects, and so on. Some important things to know: diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 50cbb9fb3b6..06518ff58de 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -229,7 +229,7 @@ You can now visit GitLab with your browser at the new external URL. Use the domain name you set up earlier to visit your new GitLab instance in your browser. In this example, it's `https://gitlab-prod.eastus.cloudapp.azure.com`. -The first thing that appears is the sign-in page. GitLab creates an admin user by default. +The first thing that appears is the sign-in page. GitLab creates an administrator user by default. The credentials are: - Username: `root` diff --git a/doc/install/docker.md b/doc/install/docker.md index b611f87938e..b3e7e758ec3 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -103,7 +103,7 @@ sudo docker run --detach \ ``` This will ensure that the Docker process has enough permissions to create the -config files in the mounted volumes. +configuration files in the mounted volumes. If you're using the [Kerberos integration](../integration/kerberos.md) **(PREMIUM ONLY)**, you must also publish your Kerberos port (for example, `--publish 8443:8443`). @@ -573,7 +573,7 @@ sudo docker restart gitlab This error occurs when using Docker Toolbox with VirtualBox on Windows or Mac, and making use of Docker volumes. The `/c/Users` volume is mounted as a -VirtualBox Shared Folder, and does not support the all POSIX filesystem features. +VirtualBox Shared Folder, and does not support the all POSIX file system features. The directory ownership and permissions cannot be changed without remounting, and GitLab fails. diff --git a/doc/install/installation.md b/doc/install/installation.md index b524177abc4..852ddea41bd 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -51,7 +51,7 @@ If the highest number stable branch is unclear, check the [GitLab blog](https:// | -------- | --------------- | ----- | | [Ruby](#2-ruby) | `2.7` | From GitLab 13.6, Ruby 2.7 is required. Ruby 3.0 is not supported yet (see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/5149) for the current status). You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | | [Go](#3-go) | `1.15` | | -| [Git](#git) | `2.31.x` | From GitLab 13.11, Git 2.31.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | +| [Git](#git) | `2.33.x` | From GitLab 14.4, Git 2.33.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | | [Node.js](#4-node) | `12.22.1` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 14.x is recommended, as it's faster. You can check which version you're running with `node -v`. You need to update it to a newer version if needed. | ## GitLab directory structure @@ -557,9 +557,10 @@ sudo -u git cp config/database.yml.postgresql config/database.yml # Once modified, the `production` settings will be as follows: # # production: -# adapter: postgresql -# encoding: unicode -# database: gitlabhq_production +# main: +# adapter: postgresql +# encoding: unicode +# database: gitlabhq_production # sudo -u git -H editor config/database.yml diff --git a/doc/install/next_steps.md b/doc/install/next_steps.md index e25241f0378..b5cfbfc9bbb 100644 --- a/doc/install/next_steps.md +++ b/doc/install/next_steps.md @@ -52,7 +52,7 @@ installation. Activate all GitLab Enterprise Edition functionality with a license. - [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers. -## Cross-repo Code Search +## Cross-repository Code Search - [Advanced Search](../integration/elasticsearch.md): Leverage Elasticsearch for faster, more advanced code search across your entire GitLab instance. diff --git a/doc/install/openshift_and_gitlab/img/add-gitlab-to-project.png b/doc/install/openshift_and_gitlab/img/add-gitlab-to-project.png Binary files differdeleted file mode 100644 index 5b6059dd022..00000000000 --- a/doc/install/openshift_and_gitlab/img/add-gitlab-to-project.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/add-to-project.png b/doc/install/openshift_and_gitlab/img/add-to-project.png Binary files differdeleted file mode 100644 index f9b00431d00..00000000000 --- a/doc/install/openshift_and_gitlab/img/add-to-project.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/create-project-ui.png b/doc/install/openshift_and_gitlab/img/create-project-ui.png Binary files differdeleted file mode 100644 index 43b151264c5..00000000000 --- a/doc/install/openshift_and_gitlab/img/create-project-ui.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/gitlab-logs.png b/doc/install/openshift_and_gitlab/img/gitlab-logs.png Binary files differdeleted file mode 100644 index 8b90b2f74ac..00000000000 --- a/doc/install/openshift_and_gitlab/img/gitlab-logs.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/gitlab-overview.png b/doc/install/openshift_and_gitlab/img/gitlab-overview.png Binary files differdeleted file mode 100644 index 3a7bec7c2bc..00000000000 --- a/doc/install/openshift_and_gitlab/img/gitlab-overview.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/gitlab-running.png b/doc/install/openshift_and_gitlab/img/gitlab-running.png Binary files differdeleted file mode 100644 index 0fcd9f00d08..00000000000 --- a/doc/install/openshift_and_gitlab/img/gitlab-running.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/gitlab-scale.png b/doc/install/openshift_and_gitlab/img/gitlab-scale.png Binary files differdeleted file mode 100644 index ebae8b588b1..00000000000 --- a/doc/install/openshift_and_gitlab/img/gitlab-scale.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/gitlab-settings.png b/doc/install/openshift_and_gitlab/img/gitlab-settings.png Binary files differdeleted file mode 100644 index 0dd1e1f5b8e..00000000000 --- a/doc/install/openshift_and_gitlab/img/gitlab-settings.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/no-resources.png b/doc/install/openshift_and_gitlab/img/no-resources.png Binary files differdeleted file mode 100644 index 1ef0a0b31e5..00000000000 --- a/doc/install/openshift_and_gitlab/img/no-resources.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/openshift-infra-project.png b/doc/install/openshift_and_gitlab/img/openshift-infra-project.png Binary files differdeleted file mode 100644 index e31dda1461c..00000000000 --- a/doc/install/openshift_and_gitlab/img/openshift-infra-project.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/rc-name.png b/doc/install/openshift_and_gitlab/img/rc-name.png Binary files differdeleted file mode 100644 index 16d967b8460..00000000000 --- a/doc/install/openshift_and_gitlab/img/rc-name.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/running-pods.png b/doc/install/openshift_and_gitlab/img/running-pods.png Binary files differdeleted file mode 100644 index e08487c881c..00000000000 --- a/doc/install/openshift_and_gitlab/img/running-pods.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/web-console.png b/doc/install/openshift_and_gitlab/img/web-console.png Binary files differdeleted file mode 100644 index 012d7703c73..00000000000 --- a/doc/install/openshift_and_gitlab/img/web-console.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index b13293eccfc..3b7ea5c1975 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -1,506 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'https://docs.gitlab.com/charts/installation/operator.html' +remove_date: '2022-09-22' --- -# How to install GitLab on OpenShift Origin 3 **(FREE SELF)** +This file was moved to [another location](https://docs.gitlab.com/charts/installation/operator.html). -WARNING: -This article is deprecated. Use the official Kubernetes Helm charts for -installing GitLab to OpenShift. Check out the -[official installation docs](https://docs.gitlab.com/charts/installation/cloud/openshift.html) -for details. - -## Introduction - -[OpenShift Origin](https://www.okd.io/) (**Note:** renamed to OKD in August 2018) is an open source container application -platform created by [RedHat](https://www.redhat.com/en), based on [Kubernetes](https://kubernetes.io/) and [Docker](https://www.docker.com). That means -you can host your own PaaS for free and almost with no hassle. - -In this tutorial, we will see how to deploy GitLab in OpenShift using the GitLab -official Docker image while getting familiar with the web interface and CLI -tools that help us achieve our goal. - -## Prerequisites - -WARNING: -This information is no longer up to date, as the current versions -have changed and products have been renamed. - -OpenShift 3 is not yet deployed on RedHat's offered [Online platform](https://www.openshift.com/), -so in order to test it, we use an [all-in-one VirtualBox image](https://www.okd.io/minishift/) that is -offered by the OpenShift developers and managed by Vagrant. If you haven't done -already, go ahead and install the following components as they are essential to -test OpenShift easily: - -- [VirtualBox](https://www.virtualbox.org/wiki/Downloads) -- [Vagrant](https://www.vagrantup.com/downloads) -- [OpenShift Client](https://docs.okd.io/3.11/cli_reference/get_started_cli.html) (`oc` for short) - -It is also important to mention that for the purposes of this tutorial, the -latest Origin release is used: - -- **`oc`** `v1.3.0` (must be [installed](https://github.com/openshift/origin/releases/tag/v1.3.0) locally on your computer) -- **OpenShift** `v1.3.0` (is pre-installed in the [VM image](https://app.vagrantup.com/openshift/boxes/origin-all-in-one)) -- **Kubernetes** `v1.3.0` (is pre-installed in the [VM image](https://app.vagrantup.com/openshift/boxes/origin-all-in-one)) - -NOTE: -If you intend to deploy GitLab on a production OpenShift cluster, there are some -limitations to bare in mind. Read on the [limitations](#current-limitations) -section for more information and follow the linked links for the relevant -discussions. - -Now that you have all batteries, let's see how easy it is to test OpenShift -on your computer. - -## Getting familiar with OpenShift Origin - -The environment we are about to use is based on CentOS 7, which comes with all -the tools needed pre-installed, including Docker, Kubernetes, and OpenShift. - -### Test OpenShift using Vagrant - -As of this writing, the all-in-one VM is at version 1.3, and that's -what we use in this tutorial. - -In short: - -1. Open a terminal and in a new directory run: - - ```shell - vagrant init openshift/origin-all-in-one - ``` - -1. This generates a Vagrantfile based on the all-in-one VM image -1. In the same directory where you generated the Vagrantfile - enter: - - ```shell - vagrant up - ``` - -This downloads the VirtualBox image and fire up the VM with some preconfigured -values as you can see in the Vagrantfile. As you may have noticed, you need -plenty of RAM (5GB in our example), so make sure you have enough. - -Now that OpenShift is set up, let's see how the web console looks like. - -### Explore the OpenShift web console - -Once Vagrant finishes its thing with the VM, you are presented with a -message which has some important information. One of them is the IP address -of the deployed OpenShift platform and in particular `https://10.2.2.2:8443/console/`. -Open this link with your browser and accept the self-signed certificate in -order to proceed. - -Let's login as admin with username/password `admin/admin`. This is what the -landing page looks like: - - - -You can see that a number of [projects](https://docs.okd.io/3.11/dev_guide/projects.html) are already created for testing purposes. - -If you head over the `openshift-infra` project, a number of services with their -respective pods are there to explore. - - - -We are not exploring the whole interface, but if you want to learn about -the key concepts of OpenShift, read the [core concepts reference](https://docs.okd.io/3.11/architecture/core_concepts/index.html) -in the official documentation. - -### Explore the OpenShift CLI - -OpenShift Client (`oc`), is a powerful CLI tool that talks to the OpenShift API -and performs pretty much everything you can do from the web UI and much more. - -Assuming you have [installed](https://docs.okd.io/3.11/cli_reference/get_started_cli.html) it, let's explore some of its main -functionalities. - -Let's first see the version of `oc`: - -```shell -$ oc version - -oc v1.3.0 -kubernetes v1.3.0+52492b4 -``` - -With `oc help` you can see the top level arguments you can run with `oc` and -interact with your cluster, Kubernetes, run applications, create projects and -much more. - -Let's login to the all-in-one VM and see how to achieve the same results like -when we visited the web console earlier. The username/password for the -administrator user is `admin/admin`. There is also a test user with username/ -password `user/user`, with limited access. Let's login as admin for the moment: - -```shell -$ oc login https://10.2.2.2:8443 - -Authentication required for https://10.2.2.2:8443 (openshift) -Username: admin -Password: -Login successful. - -You have access to the following projects and can switch between them with 'oc project <projectname>': - -- cockpit -- default (current) -- delete -- openshift -- openshift-infra -- sample - -Using project "default". -``` - -Switch to the `openshift-infra` project with: - -```shell -oc project openshift-infra -``` - -And finally, see its status: - -```shell -oc status -``` - -The last command should spit a bunch of information about the statuses of the -pods and the services, which if you look closely is what we encountered in the -second image when we explored the web console. - -You can always read more about `oc` in the [OpenShift CLI documentation](https://docs.okd.io/3.11/cli_reference/get_started_cli.html). - -### Troubleshooting the all-in-one VM - -Using the all-in-one VM gives you the ability to test OpenShift whenever you -want. That means you get to play with it, shutdown the VM, and pick up where -you left off. - -Occasionally, you may encounter issues, like OpenShift not running when booting -up the VM. The web UI may not respond, or you may see issues when trying to sign -in with `oc`, like: - -```plaintext -The connection to the server 10.2.2.2:8443 was refused - did you specify the right host or port? -``` - -In that case, the OpenShift service might not be running, so in order to fix it: - -1. SSH into the VM by selecting the directory where the Vagrantfile is and then - run: - - ```shell - vagrant ssh - ``` - -1. Run `systemctl` and verify by the output that the `openshift` service is not - running (it is in red color). If that's the case start the service with: - - ```shell - sudo systemctl start openshift - ``` - -1. Verify the service is up with: - - ```shell - systemctl status openshift -l - ``` - -You can now sign in by using `oc` (like we did before) and visit the web console. - -## Deploy GitLab - -Now that you got a taste of what OpenShift looks like, let's deploy GitLab! - -### Create a new project - -First, create a new project to host our application. You can do this -either by running the CLI client: - -```shell -oc new-project gitlab -``` - -or by using the web interface: - - - -If you used the command line, `oc` automatically uses the new project and you -can see its status with: - -```shell -$ oc status - -In project gitlab on server https://10.2.2.2:8443 - -You have no services, deployment configs, or build configs. -Run 'oc new-app' to create an application. -``` - -If you visit the web console, you can now see `gitlab` listed in the projects list. - -The next step is to import the OpenShift template for GitLab. - -### Import the template - -The [template](https://docs.okd.io/3.11/architecture/core_concepts/templates.html) is basically a JSON file which describes a set of -related object definitions to be created together, as well as a set of -parameters for those objects. - -The template for GitLab resides in the Omnibus GitLab repository under the -Docker directory. Let's download it locally with `wget`: - -```shell -wget https://gitlab.com/gitlab-org/omnibus-gitlab/raw/master/docker/openshift-template.json -``` - -And then let's import it in OpenShift: - -```shell -oc create -f openshift-template.json -n openshift -``` - -NOTE: -The `-n openshift` namespace flag is a trick to make the template available to all -projects. If you recall from when we created the `gitlab` project, `oc` switched -to it automatically, and that can be verified by the `oc status` command. If -you omit the namespace flag, the application will be available only to the -current project, in our case `gitlab`. The `openshift` namespace is a global -one that the administrators should use if they want the application to be -available to all users. - -We are now ready to finally deploy GitLab! - -### Create a new application - -The next step is to use the template we previously imported. Head over to the -`gitlab` project and hit the **Add to Project** button. - - - -This will bring you to the catalog where you can find all the pre-defined -applications ready to deploy with the click of a button. Search for `gitlab` -and you will see the previously imported template: - - - -Select it, and in the following screen you will be presented with the predefined -values used with the GitLab template: - - - -Notice at the top that there are three resources to be created with this -template: - -- `gitlab-ce` -- `gitlab-ce-redis` -- `gitlab-ce-postgresql` - -While PostgreSQL and Redis are bundled in Omnibus GitLab, the template is using -separate images as you can see from [this line](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/658c065c8d022ce858dd63eaeeadb0b2ddc8deea/docker/openshift-template.json#L239) in the template. - -The predefined values have been calculated for the purposes of testing out -GitLab in the all-in-one VM. You don't need to change anything here, hit -**Create** to start the deployment. - -If you are deploying to production you will want to change the **GitLab instance -hostname** and use greater values for the volume sizes. If you don't provide a -password for PostgreSQL, it will be created automatically. - -NOTE: -The `gitlab.apps.10.2.2.2.nip.io` hostname that is used by default will -resolve to the host with IP `10.2.2.2` which is the IP our VM uses. It is a -trick to have distinct FQDNs pointing to services that are on our local network. -Read more on how this works at [nip.io](https://nip.io). - -Now that we configured this, let's see how to manage and scale GitLab. - -## Manage and scale GitLab - -Setting up GitLab for the first time might take a while depending on your -internet connection and the resources you have attached to the all-in-one VM. -The GitLab Docker image is quite big (approximately 500 MB), so you'll have to -wait until it's downloaded and configured before you use it. - -### Watch while GitLab gets deployed - -Navigate to the `gitlab` project at **Overview**. You can notice that the -deployment is in progress by the orange color. The Docker images are being -downloaded and soon they will be up and running. - - - -Switch to the **Browse > Pods** and you will eventually see all 3 pods in a -running status. Remember the 3 resources that were to be created when we first -created the GitLab app? This is where you can see them in action. - - - -You can see GitLab being reconfigured by taking look at the logs in real time. -Click on `gitlab-ce-2-j7ioe` (your ID will be different) and go to the **Logs** -tab. - - - -At a point you should see a `gitlab Reconfigured!` message in the logs. -Navigate back to the **Overview** and hopefully all pods will be up and running. - - - -Congratulations! You can now navigate to your new shinny GitLab instance by -visiting `http://gitlab.apps.10.2.2.2.nip.io` where you will be asked to -change the root user password. Login using `root` as username and providing the -password you just set, and start using GitLab! - -### Scale GitLab with the push of a button - -If you reach to a point where your GitLab instance could benefit from a boost -of resources, you'd be happy to know that you can scale up with the push of a -button. - -In the **Overview** page just click the up arrow button in the pod where -GitLab is. The change is instant and you can see the number of [replicas](https://docs.okd.io/3.11/architecture/core_concepts/deployments.html#replication-controllers) now -running scaled to 2. - - - -Upping the GitLab pods is actually like adding new application servers to your -cluster. You can see how that would work if you didn't use GitLab with -OpenShift by following the [HA documentation](../../administration/reference_architectures/index.md) for the application servers. - -Bare in mind that you may need more resources (CPU, RAM, disk space) when you -scale up. If a pod is in pending state for too long, you can navigate to -**Browse > Events** and see the reason and message of the state. - - - -### Scale GitLab using the `oc` CLI - -Using `oc` is super easy to scale up the replicas of a pod. You may want to -skim through the [basic CLI operations](https://docs.okd.io/3.11/cli_reference/basic_cli_operations.html) to get a taste how the CLI -commands are used. Pay extra attention to the object types as we will use some -of them and their abbreviated versions below. - -In order to scale up, we need to find out the name of the replication controller. -Let's see how to do that using the following steps. - -1. Make sure you are in the `gitlab` project: - - ```shell - oc project gitlab - ``` - -1. See what services are used for this project: - - ```shell - oc get svc - ``` - - The output will be similar to: - - ```plaintext - NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE - gitlab-ce 172.30.243.177 <none> 22/TCP,80/TCP 5d - gitlab-ce-postgresql 172.30.116.75 <none> 5432/TCP 5d - gitlab-ce-redis 172.30.105.88 <none> 6379/TCP 5d - ``` - -1. We need to see the replication controllers of the `gitlab-ce` service. - Get a detailed view of the current ones: - - ```shell - oc describe rc gitlab-ce - ``` - - This will return a large detailed list of the current replication controllers. - Search for the name of the GitLab controller, usually `gitlab-ce-1` or if - that failed at some point and you spawned another one, it will be named - `gitlab-ce-2`. - -1. Scale GitLab using the previous information: - - ```shell - oc scale --replicas=2 replicationcontrollers gitlab-ce-2 - ``` - -1. Get the new replicas number to make sure scaling worked: - - ```shell - oc get rc gitlab-ce-2 - ``` - - which will return something like: - - ```plaintext - NAME DESIRED CURRENT AGE - gitlab-ce-2 2 2 5d - ``` - -And that's it! We successfully scaled the replicas to 2 using the CLI. - -As always, you can find the name of the controller using the web console. Just -click on the service you are interested in and you will see the details in the -right sidebar. - - - -### Autoscaling GitLab - -In case you were wondering whether there is an option to autoscale a pod based -on the resources of your server, the answer is yes, of course there is. - -We will not expand on this matter, but feel free to read the documentation on -OpenShift's website about [autoscaling](https://docs.okd.io/3.11/dev_guide/pod_autoscaling.html). - -## Current limitations - -As stated in the [all-in-one VM](https://www.okd.io/minishift/) page: - -> By default, OpenShift will not allow a container to run as root or even a -non-random container assigned user ID. Most Docker images in Docker Hub do not -follow this best practice and instead run as root. - -The all-in-one VM we are using has this security turned off so it will not -bother us. In any case, it is something to keep in mind when deploying GitLab -on a production cluster. - -In order to deploy GitLab on a production cluster, you will need to assign the -GitLab service account to the `anyuid` [Security Context Constraints](https://docs.okd.io/3.11/admin_guide/manage_scc.html). - -For OpenShift v3.0, you will need to do this manually: - -1. Edit the Security Context: - - ```shell - oc edit scc anyuid - ``` - -1. Add `system:serviceaccount:<project>:gitlab-ce-user` to the `users` section. - If you changed the Application Name from the default the user will - will be `<app-name>-user` instead of `gitlab-ce-user` - -1. Save and exit the editor - -For OpenShift v3.1 and above, you can do: - -```shell -oc adm policy add-scc-to-user anyuid system:serviceaccount:gitlab:gitlab-ce-user -``` - -## Conclusion - -You should now have an understanding of the basic OpenShift Origin concepts, and -a sense of how things work using the web console or the CLI. - -Upload a template, create a project, add an application, and you're done. You're -ready to sign in to your new GitLab instance. - -Remember that this tutorial doesn't address all that Origin is capable of. As -always, refer to the detailed [documentation](https://docs.okd.io) to learn more -about deploying your own OpenShift PaaS and managing your applications with -containers. +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 641b092e1e3..c136fb21a90 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -114,7 +114,7 @@ MySQL/MariaDB are advised to [migrate to PostgreSQL](../update/mysql_to_postgres The server running PostgreSQL should have _at least_ 5-10 GB of storage available, though the exact requirements [depend on the number of users](../administration/reference_architectures/index.md). -We highly recommend using the minimum PostgreSQL versions (as specified in +We highly recommend using at least the minimum PostgreSQL versions (as specified in the following table) as these were used for development and testing: | GitLab version | Minimum PostgreSQL version | @@ -169,7 +169,7 @@ of GitLab Support or other GitLab engineers. operations to manage partitioned tables. - You should not modify the GitLab schema (for example, adding triggers or modifying tables). - Database migrations are tested against the schema definition in the GitLab code base. GitLab + Database migrations are tested against the schema definition in the GitLab codebase. GitLab version upgrades may fail if the schema is modified. ## Puma settings diff --git a/doc/integration/akismet.md b/doc/integration/akismet.md index efc293569fe..d216b827676 100644 --- a/doc/integration/akismet.md +++ b/doc/integration/akismet.md @@ -16,8 +16,7 @@ Admin page. Privacy note: GitLab submits the user's IP and user agent to Akismet. NOTE: -In GitLab 8.11 and later, all issues are submitted to Akismet. -In earlier GitLab versions, this only applied to API and non-project members. +GitLab submits all issues to Akismet. Akismet configuration is available to users on self-managed GitLab. Akismet is already enabled on GitLab SaaS (GitLab.com), where it's configuration and management are handled by GitLab Inc. diff --git a/doc/integration/azure.md b/doc/integration/azure.md index 47d80ab9a66..3eb0344edda 100644 --- a/doc/integration/azure.md +++ b/doc/integration/azure.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Microsoft Azure OAuth 2.0 OmniAuth Provider **(FREE)** +# Microsoft Azure OAuth 2.0 OmniAuth Provider **(FREE SELF)** NOTE: Per Microsoft, this provider uses the [older Azure Active Directory v1.0 endpoint](https://docs.microsoft.com/en-us/azure/active-directory/azuread-dev/v1-protocols-oauth-code). diff --git a/doc/integration/bitbucket.md b/doc/integration/bitbucket.md index 7a3f4a7710d..fa3b62ba7af 100644 --- a/doc/integration/bitbucket.md +++ b/doc/integration/bitbucket.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Integrate your GitLab server with Bitbucket Cloud **(FREE)** +# Integrate your GitLab server with Bitbucket Cloud **(FREE SELF)** NOTE: Starting from GitLab 11.4, OmniAuth is enabled by default. If you're using an diff --git a/doc/integration/cas.md b/doc/integration/cas.md index 60ce728fa55..471b1e4e262 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CAS OmniAuth Provider **(FREE)** +# CAS OmniAuth Provider **(FREE SELF)** To enable the CAS OmniAuth provider you must register your application with your CAS instance. This requires the service URL GitLab supplies to CAS. It should be diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 9514c298885..b5b09fcd813 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -91,7 +91,7 @@ The former Ruby-based indexer was removed in [GitLab 12.3](https://gitlab.com/gi First, we need to install some dependencies, then we build and install the indexer itself. -This project relies on [International Components for Unicode](http://site.icu-project.org/) (ICU) for text encoding, +This project relies on [International Components for Unicode](https://icu.unicode.org/) (ICU) for text encoding, therefore we must ensure the development packages for your platform are installed before running `make`. @@ -155,7 +155,7 @@ may need to set the `production -> elasticsearch -> indexer_path` setting in you For GitLab instances with more than 50GB repository data you can follow the instructions for [Indexing large instances](#indexing-large-instances) below. -To enable Advanced Search, you must have admin access to GitLab: +To enable Advanced Search, you must have administrator access to GitLab: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Advanced Search**. @@ -461,6 +461,8 @@ The following are some available Rake tasks: | Task | Description | |:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. | +| [`sudo gitlab-rake gitlab:elastic:pause_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Pauses Elasticsearch indexing. Changes are still tracked. Useful for cluster/index migrations. | +| [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Resumes Elasticsearch indexing. | | [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. | | [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. | | [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command will result in a complete wipe of the index, and it should be used with caution. | @@ -842,7 +844,7 @@ You can run `sudo gitlab-rake gitlab:elastic:projects_not_indexed` to display pr ### No new data is added to the Elasticsearch index when I push code NOTE: -This was [fixed in GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35936) and the Rake task is not available for versions greater than that. +This was [fixed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35936) in GitLab 13.2 and the Rake task is not available for versions greater than that. When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, run: diff --git a/doc/integration/github.md b/doc/integration/github.md index 0a96d67ac0c..16241c9293b 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Integrate your GitLab instance with GitHub **(FREE)** +# Integrate your GitLab instance with GitHub **(FREE SELF)** You can integrate your GitLab instance with GitHub.com and GitHub Enterprise. This integration enables users to import projects from GitHub, or sign in to your GitLab instance diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 1f35faacc2e..5e28765bb86 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Integrate your server with GitLab.com **(FREE)** +# Integrate your server with GitLab.com **(FREE SELF)** Import projects from GitLab.com and login to your GitLab instance with your GitLab.com account. @@ -48,7 +48,7 @@ GitLab.com generates an application ID and secret key for you to use. 1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. 1. Add the provider configuration: - For Omnibus installations: + For Omnibus installations authenticating against **GitLab.com**: ```ruby gitlab_rails['omniauth_providers'] = [ @@ -61,7 +61,20 @@ GitLab.com generates an application ID and secret key for you to use. ] ``` - For installations from source: + Or, for Omnibus installations authenticating against a different GitLab instance: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "gitlab", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET", + "args" => { "scope" => "api", "client_options" => { "site" => "https://gitlab.example.com/api/v4" } } + } + ] + ``` + + For installations from source authenticating against **GitLab.com**: ```yaml - { name: 'gitlab', @@ -70,6 +83,15 @@ GitLab.com generates an application ID and secret key for you to use. args: { scope: 'api' } } ``` + Or, for installations from source to authenticate against a different GitLab instance: + + ```yaml + - { name: 'gitlab', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + args: { scope: 'api', "client_options": { "site": 'https://gitlab.example.com/api/v4' } } + ``` + 1. Change `'YOUR_APP_ID'` to the Application ID from the GitLab.com application page. 1. Change `'YOUR_APP_SECRET'` to the secret from the GitLab.com application page. 1. Save the configuration file. diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index bafba2cdf02..26520df18fa 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -28,7 +28,7 @@ To use the GitLab Gitpod integration, it must be enabled for your GitLab instanc 1. It's [enabled and configured by a GitLab administrator](#configure-a-self-managed-instance). 1. It's [enabled in their user settings](#enable-gitpod-in-your-user-settings). -To learn more about Gitpod, see their [features](https://www.gitpod.io/features/) and +To learn more about Gitpod, see their [features](https://www.gitpod.io/features) and [documentation](https://www.gitpod.io/docs/). ## Enable Gitpod in your user settings diff --git a/doc/integration/google.md b/doc/integration/google.md index 4a2c61577ac..172015f7528 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Google OAuth 2.0 OmniAuth Provider **(FREE)** +# Google OAuth 2.0 OmniAuth Provider **(FREE SELF)** To enable the Google OAuth 2.0 OmniAuth provider you must register your application with Google. Google generates a client ID and secret key for you to use. @@ -41,7 +41,7 @@ In Google's side: 1. You should now be able to see a Client ID and Client secret. Note them down or keep this page open as you need them later. -1. To enable projects to access [Google Kubernetes Engine](../user/project/clusters/index.md), you must also +1. To enable projects to access [Google Kubernetes Engine](../user/infrastructure/clusters/index.md), you must also enable these APIs: - Google Kubernetes Engine API - Cloud Resource Manager API diff --git a/doc/integration/index.md b/doc/integration/index.md index 00b65263d32..0b2bf6fde94 100644 --- a/doc/integration/index.md +++ b/doc/integration/index.md @@ -89,7 +89,7 @@ at Super User also has relevant information. **Omnibus Trusted Chain** -[Install the self signed certificate or custom certificate authorities](https://docs.gitlab.com/omnibus/common_installation_problems/README.html#using-self-signed-certificate-or-custom-certificate-authorities) +[Install the self signed certificate or custom certificate authorities](https://docs.gitlab.com/omnibus/common_installation_problems/index.html#using-self-signed-certificate-or-custom-certificate-authorities) in to Omnibus GitLab. It is enough to concatenate the certificate to the main trusted certificate diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md index 18ae71a7059..2bbda74533f 100644 --- a/doc/integration/jenkins.md +++ b/doc/integration/jenkins.md @@ -64,8 +64,8 @@ Grant a GitLab user access to the select GitLab projects. 1. Grant the user permission to the GitLab projects. - If you're integrating Jenkins with many GitLab projects, consider granting the user global - Administrator permission. Otherwise, add the user to each project, and grant the Developer role. + If you're integrating Jenkins with many GitLab projects, consider granting the user the global + Administrator role. Otherwise, add the user to each project, and grant the Developer role. ## Configure GitLab API access diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md index e349bb4e88f..8da3118cf2c 100644 --- a/doc/integration/jenkins_deprecated.md +++ b/doc/integration/jenkins_deprecated.md @@ -14,7 +14,7 @@ was deprecated in favor of the Please use documentation for the new [Jenkins CI service](jenkins.md). NOTE: -This service was [removed in v13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/1600) +This service was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1600) in GitLab 13.0 Integration includes: diff --git a/doc/integration/jira/configure.md b/doc/integration/jira/configure.md index d4e5a1bfca1..979d8a52fb8 100644 --- a/doc/integration/jira/configure.md +++ b/doc/integration/jira/configure.md @@ -41,7 +41,7 @@ To configure your project: For **Jira Server**, use `username`. For **Jira on Atlassian cloud**, use `email`. - **Password/API token**: Use `password` for **Jira Server** or `API token` for **Jira on Atlassian cloud**. -1. To enable users to view Jira issues inside the GitLab project **(PREMIUM)**, select **Enable Jira issues** and +1. To enable users to [view Jira issues](issues.md#view-jira-issues) inside the GitLab project, select **Enable Jira issues** and enter a Jira project key. You can display issues only from a single Jira project in a given GitLab project. @@ -50,7 +50,7 @@ To configure your project: If you enable Jira issues with this setting, all users with access to this GitLab project can view all issues from the specified Jira project. -1. To enable issue creation for vulnerabilities **(ULTIMATE)**, select **Enable Jira issues creation from vulnerabilities**. +1. To enable [issue creation for vulnerabilities](../../user/application_security/vulnerabilities/index.md#create-a-jira-issue-for-a-vulnerability), select **Enable Jira issues creation from vulnerabilities**. 1. Select the **Jira issue type**. If the dropdown is empty, select refresh (**{retry}**) and try again. 1. To verify the Jira connection is working, select **Test settings**. 1. Select **Save changes**. diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index 7d32c080fff..d274710b3cd 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -81,7 +81,7 @@ self-managed GitLab instances with Jira Cloud, you can either: - [Install the application manually](#install-the-application-manually). - [Create a Marketplace listing](#create-a-marketplace-listing). -### Install the application manually **(FREE SELF)** +### Install the application manually You can configure your Atlassian Cloud instance to allow you to install applications from outside the Marketplace, which allows you to install the application: @@ -114,7 +114,7 @@ NOTE: If a GitLab update makes changes to the application descriptor, you must uninstall, then reinstall, the application. -### Create a Marketplace listing **(FREE SELF)** +### Create a Marketplace listing If you prefer to not use development mode on your Jira instance, you can create your own Marketplace listing for your instance. This enables your application diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md index 5a2ce8a0a75..6fa084ee872 100644 --- a/doc/integration/jira/development_panel.md +++ b/doc/integration/jira/development_panel.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Jira Development panel integration **(FREE)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) to GitLab Free in 13.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) from GitLab Premium to GitLab Free in 13.4. With the Jira Development panel integration, you can reference Jira issues in GitLab. When configured, activity (such as pipeline, deployment, and feature flags) displays in the Jira issue's @@ -69,13 +69,13 @@ To simplify administration, we recommend that a GitLab group maintainer or group | Jira usage | GitLab.com customers need | GitLab self-managed customers need | |------------|---------------------------|------------------------------------| -| [Atlassian cloud](https://www.atlassian.com/cloud) | The [GitLab.com for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview) installed from the [Atlassian Marketplace](https://marketplace.atlassian.com). This offers real-time sync between GitLab.com and Jira. For more information, see the documentation for the [GitLab.com for Jira Cloud app](connect-app.md). | The [GitLab.com for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview), using a workaround process. See the documentation for [installing the GitLab.com for Jira Cloud app for self-managed instances](connect-app.md#install-the-gitlabcom-for-jira-cloud-app-for-self-managed-instances) for more information. | +| [Atlassian cloud](https://www.atlassian.com/migration/assess/why-cloud) | The [GitLab.com for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview) installed from the [Atlassian Marketplace](https://marketplace.atlassian.com). This offers real-time sync between GitLab.com and Jira. For more information, see the documentation for the [GitLab.com for Jira Cloud app](connect-app.md). | The [GitLab.com for Jira Cloud app](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud?hosting=cloud&tab=overview), using a workaround process. See the documentation for [installing the GitLab.com for Jira Cloud app for self-managed instances](connect-app.md#install-the-gitlabcom-for-jira-cloud-app-for-self-managed-instances) for more information. | | Your own server | The [Jira DVCS (distributed version control system) connector](dvcs.md). This syncs data hourly. | The [Jira DVCS Connector](dvcs.md). | Each GitLab project can be configured to connect to an entire Jira instance. That means after configuration, one GitLab project can interact with all Jira projects in that instance. For: -- The [view Jira issues](issues.md#view-jira-issues) feature **(PREMIUM)**, you must associate a GitLab project with a +- The [view Jira issues](issues.md#view-jira-issues) feature, you must associate a GitLab project with a specific Jira project. - Other features, you do not have to explicitly associate a GitLab project with any single Jira project. diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index 85f99b49c41..f5f7e8c33fc 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Jira integrations **(FREE)** If your organization uses [Jira](https://www.atlassian.com/software/jira) issues, -you can [migrate your issues from Jira](../../user/project/import/jira.md) **(PREMIUM)** and work +you can [migrate your issues from Jira](../../user/project/import/jira.md) and work exclusively in GitLab. However, if you'd like to continue to use Jira, you can integrate it with GitLab. GitLab offers two types of Jira integrations, and you can use one or both depending on the capabilities you need. It is recommended that you enable both. @@ -20,7 +20,7 @@ in your GitLab project with any of your projects in Jira. ### Jira integration This integration connects one or more GitLab projects to a Jira instance. The Jira instance -can be hosted by you or in [Atlassian cloud](https://www.atlassian.com/cloud). +can be hosted by you or in [Atlassian cloud](https://www.atlassian.com/migration/assess/why-cloud). The supported Jira versions are `v6.x`, `v7.x`, and `v8.x`. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -49,8 +49,8 @@ or the Jira DVCS (distributed version control system) connector, | Mention a Jira issue ID in a GitLab branch name and the Jira issue shows the branch name. | No. | Yes, in the issue's [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). | | Add Jira time tracking to an issue. | No. | Yes. Time can be specified using Jira Smart Commits. | | Use a Git commit or merge request to transition or close a Jira issue. | Yes. Only a single transition type, typically configured to close the issue by setting it to Done. | Yes. Transition to any state using Jira Smart Commits. | -| Display a list of Jira issues. | Yes. **(PREMIUM)** | No. | -| Create a Jira issue from a vulnerability or finding. | Yes. **(ULTIMATE)** | No. | +| Display a list of [Jira issues](issues.md#view-jira-issues). | Yes. | No. | +| Create a Jira issue from a [vulnerability or finding](../../user/application_security/vulnerabilities/index.md#create-a-jira-issue-for-a-vulnerability). | Yes. | No. | | Create a GitLab branch from a Jira issue. | No. | Yes, in the issue's [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). | ## Authentication in Jira diff --git a/doc/integration/jira/issues.md b/doc/integration/jira/issues.md index 0373506062c..70e938a24d4 100644 --- a/doc/integration/jira/issues.md +++ b/doc/integration/jira/issues.md @@ -96,7 +96,7 @@ Consider this example: ## View Jira issues **(PREMIUM)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3622) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3622) in GitLab 13.2. You can browse, search, and view issues from a selected Jira project directly in GitLab, if your GitLab administrator [has configured it](configure.md). diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 48339144292..2d4abb75875 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -2,7 +2,6 @@ stage: Manage group: Access info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, how-to --- # Kerberos integration **(PREMIUM SELF)** @@ -173,7 +172,7 @@ enabled, your users are linked to their LDAP accounts on their first sign-in. For this to work, some prerequisites must be met: The Kerberos username must match the LDAP user's UID. You can choose which LDAP -attribute is used as the UID in the GitLab [LDAP configuration](../administration/auth/ldap/index.md#configuration) +attribute is used as the UID in the GitLab [LDAP configuration](../administration/auth/ldap/index.md#configure-ldap) but for Active Directory, this should be `sAMAccountName`. The Kerberos realm must match the domain part of the LDAP user's Distinguished @@ -296,7 +295,7 @@ Kerberos ticket-based authentication. ## Upgrading from password-based to ticket-based Kerberos sign-ins -Prior to GitLab 8.10 Enterprise Edition, users had to submit their +In previous versions of GitLab users had to submit their Kerberos username and password to GitLab when signing in. We plan to remove support for password-based Kerberos sign-ins in a future release, so we recommend that you upgrade to ticket-based sign-ins. diff --git a/doc/integration/mattermost/img/gitlab-mattermost.png b/doc/integration/mattermost/img/gitlab-mattermost.png Binary files differindex 1eedcb391b0..c3b019988d0 100644 --- a/doc/integration/mattermost/img/gitlab-mattermost.png +++ b/doc/integration/mattermost/img/gitlab-mattermost.png diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 4830f8ba84e..39005940dfc 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -175,7 +175,7 @@ sudo gitlab-psql -d mattermost_production ### Back up GitLab Mattermost -GitLab Mattermost is not included in the regular [Omnibus GitLab backup](../../raketasks/backup_restore.md#back-up-gitlab) Rake task. +GitLab Mattermost is not included in the regular [Omnibus GitLab backup](../../raketasks/backup_restore.md) Rake task. The general Mattermost [backup and disaster recovery](https://docs.mattermost.com/deploy/backup-disaster-recovery.html) documentation can be used as a guide on what needs to be backed up. @@ -229,7 +229,7 @@ sudo gitlab-ctl start mattermost ### Mattermost Command Line Tools (CLI) NOTE: -This CLI will be replaced in a future release with the new [`mmctl` Command Line Tool](https://docs.mattermost.com/administration/mmctl-cli-tool.html). +This CLI will be replaced in a future release with the new [`mmctl` Command Line Tool](https://docs.mattermost.com/manage/mmctl-command-line-tool.html). To use the [Mattermost Command Line Tools (CLI)](https://docs.mattermost.com/administration/command-line-tools.html), ensure that you are in the `/opt/gitlab/embedded/service/mattermost` directory when you run the CLI commands and that you specify the location of the configuration file. The executable is `/opt/gitlab/embedded/bin/mattermost`. @@ -338,6 +338,7 @@ Below is a list of Mattermost versions for GitLab 11.10 and later: | 14.1 | 5.36 | | 14.2 | 5.37 | | 14.3 | 5.38 | +| 14.4 | 5.39 | NOTE: When upgrading the Mattermost version, it is essential to check the diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index 867108d4597..bdf6a0e687d 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Sign into GitLab with (almost) any OAuth2 provider **(FREE)** +# Sign into GitLab with (almost) any OAuth2 provider **(FREE SELF)** The `omniauth-oauth2-generic` gem allows Single Sign-On between GitLab and your own OAuth2 provider (or any OAuth2 provider compatible with this gem) diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index 5df6c4f28b7..af715e47ab9 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -4,18 +4,16 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab as an OAuth 2.0 authentication service provider +# Configure GitLab as an OAuth 2.0 authentication identity provider -This document describes how you can use GitLab as an OAuth 2.0 -authentication service provider. +This document describes how you can use GitLab as an OAuth 2.0 authentication identity provider. -If you want to use: - -- The [OAuth 2.0](https://oauth.net/2/) protocol to access GitLab resources on - a user's behalf, see [OAuth 2.0 provider](../api/oauth2.md). -- Other OAuth 2.0 authentication service providers to sign in to - GitLab, see the [OAuth 2.0 client documentation](omniauth.md). -- The related API, see [Applications API](../api/applications.md). +- OAuth 2 applications can be created and managed using the GitLab UI (described below) + or managed using the [Applications API](../api/applications.md). +- After an application is created, external services can manage access tokens using the + [OAuth 2 API](../api/oauth2.md). +- To allow users to sign in to GitLab using third-party OAuth 2 providers, see + [OmniAuth documentation](omniauth.md). ## Introduction to OAuth @@ -88,6 +86,25 @@ To create an application for your GitLab instance: When creating application in the **Admin Area** , you can mark it as _trusted_. The user authorization step is automatically skipped for this application. +## Expiring Access Tokens + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21745) in GitLab 14.3. + +By default, all new applications expire access tokens after 2 hours. In GitLab 14.2 and +earlier, OAuth access tokens had no expiration. + +All integrations should update to support access token refresh. + +When creating new applications, you can opt-out of expiry for backward compatibility by clearing +**Expire access tokens** when creating them. The ability to opt-out +[is deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/340848). + +Existing: + +- Applications can have expiring access tokens. Edit the application and select + **Expire access tokens** to enable them. +- Tokens must be [revoked](../api/oauth2.md#revoke-a-token) or they don't expire. + ## Authorized applications Every application you authorize with your GitLab credentials is shown diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 7dc775c2557..6c154ee7f5b 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# OmniAuth **(FREE)** +# OmniAuth **(FREE SELF)** GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and other popular services. [OmniAuth](https://rubygems.org/gems/omniauth/) is a @@ -47,7 +47,7 @@ contains some settings that are common for all providers. | [OpenID Connect](../administration/auth/oidc.md) | `openid_connect` | | [Salesforce](salesforce.md) | `salesforce` | | [SAML](saml.md) | `saml` | -| [Shibboleth](shibboleth.md) | `shibboleth` | +| [Shibboleth](saml.md) | `shibboleth` | | [Twitter](twitter.md) | `twitter` | ## Initial OmniAuth Configuration diff --git a/doc/integration/recaptcha.md b/doc/integration/recaptcha.md index fd5170d615f..60e2d70ce32 100644 --- a/doc/integration/recaptcha.md +++ b/doc/integration/recaptcha.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# reCAPTCHA **(FREE)** +# reCAPTCHA **(FREE SELF)** GitLab leverages [Google's reCAPTCHA](https://www.google.com/recaptcha/about/) to protect against spam and abuse. GitLab displays the CAPTCHA form on the sign-up page diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 56d9feb14e0..9555c762761 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Salesforce OmniAuth Provider **(FREE)** +# Salesforce OmniAuth Provider **(FREE SELF)** You can integrate your GitLab instance with [Salesforce](https://www.salesforce.com/) to enable users to log in to your GitLab instance with their Salesforce account. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index b89772ba2ca..876eb7ba80b 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -53,6 +53,7 @@ in your SAML IdP: sudo -u git -H editor config/gitlab.yml ``` +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. 1. To allow your users to use SAML to sign up without having to manually create an account first, add the following values to your configuration: @@ -196,15 +197,13 @@ For example configurations, see the [notes on specific providers](#providers). | Field | Supported keys | |-----------------|----------------| | Email (required)| `email`, `mail` | -| Username | `username`, `nickname` | | Full Name | `name` | | First Name | `first_name`, `firstname`, `firstName` | | Last Name | `last_name`, `lastname`, `lastName` | -If a username is not specified, the email address is used to generate the GitLab username. - -See [`attribute_statements`](#attribute_statements) for examples on how the -assertions are configured. +See [`attribute_statements`](#attribute_statements) for examples on how custom +assertions are configured. This section also describes how to configure custom +username attributes. Please refer to [the OmniAuth SAML gem](https://github.com/omniauth/omniauth-saml/blob/master/lib/omniauth/strategies/saml.rb) for a full list of supported assertions. @@ -247,7 +246,7 @@ The name of the attribute can be anything you like, but it must contain the grou to which a user belongs. To tell GitLab where to find these groups, you need to add a `groups_attribute:` element to your SAML settings. -### Required groups **(FREE SELF)** +### Required groups Your IdP passes Group information to the SP (GitLab) in the SAML Response. To use this response, configure GitLab to identify: @@ -274,7 +273,7 @@ Example: } } ``` -### External groups **(FREE SELF)** +### External groups SAML login supports the automatic identification of a user as an [external user](../user/permissions.md#external-users). This is based on the user's group @@ -294,7 +293,7 @@ membership in the SAML identity provider. } } ``` -### Administrator groups **(FREE SELF)** +### Administrator groups The requirements are the same as the previous settings: @@ -443,7 +442,7 @@ SAML users has an administrator role. You may also bypass the auto sign-in feature by browsing to `https://gitlab.example.com/users/sign_in?auto_sign_in=false`. -### `attribute_statements` +### `attribute_statements` **(FREE SELF)** NOTE: This setting should be used only to map attributes that are part of the OmniAuth @@ -475,12 +474,10 @@ args: { #### Set a username -By default, the email in the SAML response is used to automatically generate the -user's GitLab username. If you'd like to set another attribute as the username, -assign it to the `nickname` OmniAuth `info` hash attribute. +By default, the local part of the email address in the SAML response is used to +generate the user's GitLab username. -For example, if you want to set the `username` attribute in your SAML Response to the username -in GitLab, use the following setting: +Configure `nickname` in `attribute_statements` to specify one or more attributes that contain a user's desired username: ```yaml args: { @@ -493,6 +490,8 @@ args: { } ``` +This also sets the `username` attribute in your SAML Response to the username in GitLab. + ### `allowed_clock_drift` The clock of the Identity Provider may drift slightly ahead of your system clocks. @@ -719,8 +718,8 @@ documentation on how to use SAML to sign in to GitLab. Examples: - [ADFS (Active Directory Federation Services)](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) -- [Auth0](https://auth0.com/docs/protocols/saml-protocol/configure-auth0-as-saml-identity-provider) -- [PingOne by Ping Identity](https://docs.pingidentity.com/bundle/pingone/page/xsh1564020480660-1.html) +- [Auth0](https://auth0.com/docs/configure/saml-configuration/configure-auth0-saml-identity-provider) +- [PingOne by Ping Identity](http://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html) GitLab provides the following setup notes for guidance only. If you have any questions on configuring the SAML app, please contact your provider's support. @@ -729,7 +728,7 @@ If you have any questions on configuring the SAML app, please contact your provi The following guidance is based on this Okta article, on adding a [SAML Application with an Okta Developer account](https://support.okta.com/help/s/article/Why-can-t-I-add-a-SAML-Application-with-an-Okta-Developer-account?language=en_US): -1. In the Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. +1. In the Okta administrator section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. 1. When the app screen comes up you see another button to **Create an App** and choose SAML 2.0 on the next screen. 1. Optionally, you can add a logo @@ -802,11 +801,12 @@ If you only require a SAML provider for testing, a [quick start guide to start a ### 500 error after login If you see a "500 error" in GitLab when you are redirected back from the SAML -sign-in page, this likely indicates that GitLab couldn't get the email address -for the SAML user. +sign-in page, this could indicate that: -Ensure the IdP provides a claim containing the user's email address, using the -claim name `email` or `mail`. +- GitLab couldn't get the email address for the SAML user. Ensure the IdP provides a claim containing the user's + email address using the claim name `email` or `mail`. +- The certificate set your `gitlab.rb` file for `idp_cert_fingerprint` or `idp_cert` file is incorrect. +- Your `gitlab.rb` file is set to enable `idp_cert_fingerprint`, and `idp_cert` is being provided, or the reverse. ### 422 error after login diff --git a/doc/integration/security_partners/index.md b/doc/integration/security_partners/index.md index 2b851b5f614..b8c7a0163f5 100644 --- a/doc/integration/security_partners/index.md +++ b/doc/integration/security_partners/index.md @@ -1,11 +1,11 @@ --- -stage: Protect -group: Container Security +stage: Secure +group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: index --- -# Security partner integrations +# Security partner integrations **(FREE)** You can integrate GitLab with its security partners. This page has information on how do this with each security partner: diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md deleted file mode 100644 index 4a3aa6b3dc5..00000000000 --- a/doc/integration/shibboleth.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'saml.md' -remove_date: '2021-09-29' ---- - -This file was moved to [another location](saml.md). - -<!-- This redirect file can be deleted after <2021-09-29>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index 4059aef9de3..f5916b72d9d 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Slash commands in Mattermost and Slack **(FREE)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) to GitLab Free in 11.9. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) from GitLab Ultimate to GitLab Free in 11.9. If you want to control and view GitLab content while you're working in Slack and Mattermost, you can use slash commands. diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index 6bc467ed7b2..f8dd552ec6a 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -4,7 +4,7 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Twitter OAuth 2.0 OmniAuth Provider **(FREE)** +# Twitter OAuth 2.0 OmniAuth Provider **(FREE SELF)** To enable the Twitter OmniAuth provider you must register your application with Twitter. Twitter generates a client ID and secret key for you to use. diff --git a/doc/integration/vault.md b/doc/integration/vault.md index ebfa91c7516..3bca3767785 100644 --- a/doc/integration/vault.md +++ b/doc/integration/vault.md @@ -2,12 +2,9 @@ stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- -# Vault Authentication with GitLab OpenID Connect - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/22323) in GitLab 9.0 +# Vault Authentication with GitLab OpenID Connect **(FREE)** [Vault](https://www.vaultproject.io/) is a secrets management application offered by HashiCorp. It allows you to store and manage sensitive information such as secret environment variables, encryption keys, and authentication tokens. diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index e694105d794..88785196f6e 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -10,6 +10,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w Error Tracking allows developers to easily discover and view the errors that their application may be generating. By surfacing error information where the code is being developed, efficiency and awareness can be increased. +## How error tracking works + +For error tracking to work, you need two pieces: + +- **Your application with Sentry SDK:** when the error happens, Sentry SDK captures information + about it and sends it over the network to the backend. The backend stores information about all + errors. + +- **Error tracking backend:** the backend can be either GitLab itself or Sentry. When it's GitLab, + we name it _integrated error tracking_ because you don't need to set up a separate backend. It's + already part of the product. + + - To use the GitLab backend, see [integrated error tracking](#integrated-error-tracking). + - To use Sentry as the backend, see [Sentry error tracking](#sentry-error-tracking). + + No matter what backend you choose, the [error tracking UI](#error-tracking-list) + is the same. + ## Sentry error tracking [Sentry](https://sentry.io/) is an open source error tracking system. GitLab allows administrators to connect Sentry to GitLab, to allow users to view a list of Sentry errors in GitLab. @@ -111,7 +129,7 @@ If another event occurs, the error reverts to unresolved. ## Integrated error tracking -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) in GitLab 14.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/329596) in GitLab 14.4. Integrated error tracking is a lightweight alternative to Sentry backend. You still use Sentry SDK with your application. But you don't need to deploy Sentry @@ -125,8 +143,7 @@ Sentry integration. ### Project settings -The feature should be enabled on the project level. However, there is no UI to enable this feature yet. -You must use the GitLab API to enable it. +You can find the feature configuration at **Settings > Monitor > Error Tracking**. #### How to enable @@ -134,26 +151,21 @@ You must use the GitLab API to enable it.  -1. Create a client key (DSN) to use with Sentry SDK in your application. Make sure to save the - response, as it contains a DSN: +1. Select **Save changes**. After page reload you should see a text field with the DSN string. Copy it. - ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/projects/PROJECT_ID/error_tracking/client_keys" - ``` +  1. Take the DSN from the previous step and configure your Sentry SDK with it. Errors are now reported to the GitLab collector and are visible in the [GitLab UI](#error-tracking-list). -#### How to disable +#### Managing DSN -To disable the feature, run this command. This is the same command as the one that enables the -feature, but with a `false` value instead: +When you enable the feature you receive a DSN. It includes a hash used for authentication. This hash +is a client key. GitLab uses client keys to authenticate error tracking requests from your +application to the GitLab backend. -```shell -curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" \ - "https://gitlab.example.com/api/v4/projects/PROJECT_ID/error_tracking/settings?active=false&integrated=false" -``` +In some situations, you may want to create a new client key and remove an existing one. +You can do so by managing client keys with the [error tracking API](../api/error_tracking.md). #### Limitations diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index a8686e2f4b7..2af4ee47292 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Feature Flags **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/7433) in GitLab 11.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to GitLab Free in 13.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) from GitLab Premium to GitLab Free in 13.5. With Feature Flags, you can deploy your application's new features to production in smaller batches. You can toggle a feature on and off to subsets of users, helping you achieve Continuous Delivery. @@ -166,6 +166,21 @@ WARNING: The Unleash client **must** be given a user ID for the feature to be enabled for target users. See the [Ruby example](#ruby-application-example) below. +## Search for Code References **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300299) in GitLab 14.4. + +Search your project and find any references of a feature flag in your +code so that you and clean it up when it's time to remove the feature flag. + +To search for code references of a feature flag: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. Edit the feature flag you want to remove. +1. Select **More actions** (**{ellipsis_v}**). +1. Select **Search code references**. + ### User List > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35930) in GitLab 13.1. @@ -312,7 +327,7 @@ Unleash currently [offers many SDKs for various languages and frameworks](https: For API content, see: - [Feature Flags API](../api/feature_flags.md) -- [Feature Flag Specs API](../api/feature_flag_specs.md) (Deprecated and [scheduled for removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369).) +- [Feature Flag Specs API](../api/feature_flag_specs.md) (Deprecated and [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/213369) in GitLab 14.0.) - [Feature Flag User Lists API](../api/feature_flag_user_lists.md) - [Legacy Feature Flags API](../api/feature_flags_legacy.md) diff --git a/doc/operations/img/error_tracking_setting_dsn_v14_4.png b/doc/operations/img/error_tracking_setting_dsn_v14_4.png Binary files differnew file mode 100644 index 00000000000..b7ecaa047b2 --- /dev/null +++ b/doc/operations/img/error_tracking_setting_dsn_v14_4.png diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index c6ff70103d9..cdf7ca5c8bc 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -10,11 +10,10 @@ Alerts are a critical entity in your incident management workflow. They represen ## Alert List -Users with at least Developer [permissions](../../user/permissions.md) can +Users with at least the Developer [role](../../user/permissions.md) can access the Alert list at **Monitor > Alerts** in your project's sidebar. The Alert list displays alerts sorted by start time, but you can change the sort order by clicking the headers in the Alert list. -([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.) The alert list displays the following information: @@ -53,6 +52,8 @@ immediately identify which alerts you should prioritize investigating: Alerts contain one of the following icons: +<!-- vale gitlab.SubstitutionWarning = NO --> + | Severity | Icon | Color (hexadecimal) | |----------|-------------------------|---------------------| | Critical | **{severity-critical}** | `#8b2615` | @@ -62,10 +63,12 @@ Alerts contain one of the following icons: | Info | **{severity-info}** | `#418cd8` | | Unknown | **{severity-unknown}** | `#bababa` | +<!-- vale gitlab.SubstitutionWarning = YES --> + ## Alert details page Navigate to the Alert details view by visiting the [Alert list](alerts.md) -and selecting an alert from the list. You need least Developer [permissions](../../user/permissions.md) +and selecting an alert from the list. You need at least the Developer [role](../../user/permissions.md) to access alerts. NOTE: @@ -108,7 +111,7 @@ To view the metrics for an alert: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201846) in GitLab Ultimate 12.8. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/217768) in GitLab 13.3. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) to GitLab Free 12.9. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25455) from GitLab Ultimate to GitLab Free in 12.9. Viewing logs from a metrics panel can be useful if you're triaging an application incident and need to [explore logs](../metrics/dashboards/index.md#chart-context-menu) diff --git a/doc/operations/incident_management/escalation_policies.md b/doc/operations/incident_management/escalation_policies.md index b82aa5e5dc5..5f132720000 100644 --- a/doc/operations/incident_management/escalation_policies.md +++ b/doc/operations/incident_management/escalation_policies.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Escalation Policies **(PREMIUM)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4638) in [GitLab Premium](https://about.gitlab.com/pricing/) 14.1. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4638) in GitLab 14.1. Escalation Policies protect your company from missed critical alerts. Escalation Policies contain time-boxed steps that automatically page the next responder in the escalation step if the responder diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 0cd0a645c15..2d9ef21f1ce 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -24,7 +24,7 @@ Incident, you have two options to do this manually. **From the Incidents List:** -> [Moved](https://gitlab.com/gitlab-org/monitor/health/-/issues/24) to GitLab Free in 13.3. +> [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. - Navigate to **Monitor > Incidents** and click **Create Incident**. - Create a new issue using the `incident` template available when creating it. @@ -44,11 +44,11 @@ Incident, you have two options to do this manually.  -### Create incidents automatically +### Create incidents automatically **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in GitLab Ultimate 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in GitLab 11.11. -With Maintainer or higher [permissions](../../user/permissions.md), you can enable +With at least the Maintainer [role](../../user/permissions.md), you can enable GitLab to create incident automatically whenever an alert is triggered: 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. @@ -56,9 +56,9 @@ With Maintainer or higher [permissions](../../user/permissions.md), you can enab 1. To customize the incident, select an [issue template](../../user/project/description_templates.md#create-an-issue-template). 1. To send [an email notification](paging.md#email-notifications) to users - with the [Developer role](../../user/permissions.md), select + with the Developer [role](../../user/permissions.md), select **Send a separate email notification to Developers**. Email notifications are - also sent to users with **Maintainer** and **Owner** permissions. + also sent to users with the **Maintainer** and **Owner** roles. 1. Click **Save changes**. ### Create incidents via the PagerDuty webhook @@ -69,7 +69,7 @@ You can set up a webhook with PagerDuty to automatically create a GitLab inciden for each PagerDuty incident. This configuration requires you to make changes in both PagerDuty and GitLab: -1. Sign in as a user with the [Maintainer role](../../user/permissions.md). +1. Sign in as a user with the Maintainer [role](../../user/permissions.md). 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Select the **PagerDuty integration** tab: @@ -114,14 +114,14 @@ in your project's sidebar. The list contains the following metrics: tooltip depending on the user's locale. - **Assignees** - The user assigned to the incident. - **Published** - Displays a green check mark (**{check-circle}**) if the incident is published - to a [Status Page](status_page.md). **(ULTIMATE)** + to a [Status Page](status_page.md). The Incident list displays incidents sorted by incident created date. -([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) to GitLab Free in 13.3.) +([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229534) in GitLab 13.3.) To see if a column is sortable, point your mouse at the header. Sortable columns display an arrow next to the column name. -Incidents share the [Issues API](../../user/project/issues/index.md). +Incidents share the [Issues API](../../api/issues.md). NOTE: For a live example of the incident list in action, visit this @@ -165,9 +165,9 @@ Beneath the highlight bar, GitLab displays a summary that includes the following Comments are displayed in threads, but can be displayed chronologically [in a timeline view](#timeline-view). -### Metrics +### Metrics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235994) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235994) in GitLab 13.8. In many cases, incidents are associated to metrics. You can upload screenshots of metric charts in the **Metrics** tab: @@ -189,7 +189,7 @@ field populated. ### Timeline view **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in GitLab 13.5. To quickly see the latest updates on an incident, click **{comments}** **Turn timeline view on** in the comment bar to display comments @@ -199,7 +199,7 @@ un-threaded and ordered chronologically, newest to oldest: ### Service Level Agreement countdown timer **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241663) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241663) in GitLab 13.5. You can enable the Service Level Agreement Countdown timer on incidents to track the Service Level Agreements (SLAs) you hold with your customers. The timer is @@ -277,7 +277,7 @@ templates. > - [Introduced for Prometheus Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13401) in GitLab 12.5. > - [Introduced for HTTP Integrations](https://gitlab.com/gitlab-org/gitlab/-/issues/13402) in GitLab 13.4. -With Maintainer or higher [permissions](../../user/permissions.md), you can enable +With at least the Maintainer [role](../../user/permissions.md), you can enable GitLab to close an incident automatically when a **Recovery Alert** is received: 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index f6c85045fa0..1426148d163 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Integrations **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in GitLab Ultimate 12.4. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) to GitLab Free in 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13203) in GitLab 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42640) from GitLab Ultimate to GitLab Free in 12.8. GitLab can accept alerts from any source via a webhook receiver. This can be configured generically or, in GitLab versions 13.1 and greater, you can configure @@ -16,9 +16,9 @@ to use this endpoint. ## Integrations list -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in GitLab Free 13.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in GitLab 13.5. -With at least the [Maintainer role](../../user/permissions.md), you can view the list of configured +With at least the Maintainer [role](../../user/permissions.md), you can view the list of configured alerts integrations by navigating to **Settings > Monitor** in your project's sidebar menu, and expanding the **Alerts** section. The list displays the integration name, type, and status (enabled or disabled): @@ -30,13 +30,13 @@ the integration name, type, and status (enabled or disabled): GitLab can receive alerts via a HTTP endpoint that you configure, or the [Prometheus integration](#external-prometheus-integration). -### Single HTTP Endpoint **(FREE)** +### Single HTTP Endpoint Enabling the HTTP Endpoint in a GitLab projects activates it to receive alert payloads in JSON format. You can always [customize the payload](#customize-the-alert-payload-outside-of-gitlab) to your liking. -1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) +1. Sign in to GitLab as a user with the Maintainer [role](../../user/permissions.md) for a project. 1. Navigate to **Settings > Monitor** in your project. 1. Expand the **Alerts** section, and in the **Select integration type** dropdown menu, @@ -47,13 +47,13 @@ receive alert payloads in JSON format. You can always ### HTTP Endpoints **(PREMIUM)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4442) in GitLab Premium 13.6. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4442) in GitLab 13.6. In [GitLab Premium](https://about.gitlab.com/pricing/), you can create multiple unique HTTP endpoints to receive alerts from any external source in JSON format, and you can [customize the payload](#customize-the-alert-payload-outside-of-gitlab). -1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) +1. Sign in to GitLab as a user with the Maintainer [role](../../user/permissions.md) for a project. 1. Navigate to **Settings > Monitor** in your project. 1. Expand the **Alerts** section. @@ -80,7 +80,7 @@ side of the integrations list. #### Map fields in custom alerts -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4443) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4443) in GitLab 13.10. You can integrate your monitoring tool's alert format with GitLab alerts. To show the correct information in the [Alert list](alerts.md) and the @@ -125,17 +125,7 @@ NOTE: Ensure your requests are smaller than the [payload application limits](../../administration/instance_limits.md#generic-alert-json-payloads). -Example request: - -```shell -curl --request POST \ - --data '{"title": "Incident title"}' \ - --header "Authorization: Bearer <authorization_key>" \ - --header "Content-Type: application/json" \ - <url> -``` - -The `<authorization_key>` and `<url>` values can be found when configuring an alert integration. +### Example request body Example payload: @@ -157,15 +147,64 @@ Example payload: } ``` +## Authorization + +The following authorization methods are accepted: + +- Bearer authorization header +- Basic authentication + +The `<authorization_key>` and `<url>` values can be found when configuring an alert integration. + +### Bearer authorization header + +The authorization key can be used as the Bearer token: + +```shell +curl --request POST \ + --data '{"title": "Incident title"}' \ + --header "Authorization: Bearer <authorization_key>" \ + --header "Content-Type: application/json" \ + <url> +``` + +### Basic authentication + +The authorization key can be used as the `password`. The `username` is left blank: + +- username: `<blank>` +- pasword: authorization_key + +```shell +curl --request POST \ + --data '{"title": "Incident title"}' \ + --header "Authorization: Basic <base_64_encoded_credentials>" \ + --header "Content-Type: application/json" \ + <url> +``` + +Basic authentication can also be used with credentials directly in the URL: + +```shell +curl --request POST \ + --data '{"title": "Incident title"}' \ + --header "Content-Type: application/json" \ + <username:password@url> +``` + +WARNING: +Using your authorization key in the URL is insecure, as it's visible in server logs. We recommend +using one of the above header options if your tooling supports it. + ## Triggering test alerts -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Free in 13.2. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab in 13.2. After a [project maintainer or owner](../../user/permissions.md) configures an integration, you can trigger a test alert to confirm your integration works properly. -1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md). +1. Sign in as a user with at least the Developer [role](../../user/permissions.md). 1. Navigate to **Settings > Monitor** in your project. 1. Click **Alerts** to expand the section. 1. Click the **{settings}** settings icon on the right side of the integration in [the list](#integrations-list). @@ -177,7 +216,7 @@ GitLab displays an error or success message, depending on the outcome of your te ## Automatic grouping of identical alerts **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in GitLab Premium 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214557) in GitLab 13.2. In GitLab versions 13.2 and greater, GitLab groups alerts based on their payload. When an incoming alert contains the same payload as another alert @@ -200,9 +239,9 @@ field is `end_time`. With custom mappings, you can select the expected field. You can also configure the associated [incident to be closed automatically](../incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) when the alert resolves. -## Link to your Opsgenie Alerts +## Link to your Opsgenie Alerts **(PREMIUM)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab Premium 13.2. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab 13.2. WARNING: We are building deeper integration with Opsgenie and other alerting tools through @@ -219,7 +258,7 @@ active at the same time. To enable Opsgenie integration: -1. Sign in as a user with the [Maintainer or Owner role](../../user/permissions.md). +1. Sign in as a user with the Maintainer or Owner [role](../../user/permissions.md). 1. Navigate to **Monitor > Alerts**. 1. In the **Integrations** select box, select **Opsgenie**. 1. Select the **Active** toggle. diff --git a/doc/operations/incident_management/oncall_schedules.md b/doc/operations/incident_management/oncall_schedules.md index 9e736f2cdec..2a8f0eac59c 100644 --- a/doc/operations/incident_management/oncall_schedules.md +++ b/doc/operations/incident_management/oncall_schedules.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # On-call Schedule Management **(PREMIUM)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4544) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.11. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4544) in GitLab 13.11. Use on-call schedule management to create schedules for responders to rotate on-call responsibilities. Maintain the availability of your software services by putting your teams on-call. @@ -31,7 +31,7 @@ To create an on-call schedule: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Monitor > On-call Schedules**. 1. Select **Add a schedule**. -1. Enter the schedule's name and description and select a timezone. +1. Enter the schedule's name and description and select a time zone. 1. Select **Add schedule**. You now have an empty schedule with no rotations. This renders as an empty state, prompting you to diff --git a/doc/operations/incident_management/status_page.md b/doc/operations/incident_management/status_page.md index db7b55424dd..241112df521 100644 --- a/doc/operations/incident_management/status_page.md +++ b/doc/operations/incident_management/status_page.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Status Page **(ULTIMATE)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2479) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.10. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2479) in GitLab 12.10. With a GitLab Status Page, you can create and deploy a static website to communicate efficiently to users during an incident. The Status Page landing page displays an @@ -138,7 +138,7 @@ you provided during setup. As part of publication, GitLab: - Anonymizes user and group mentions with `Incident Responder`. - Removes titles of non-public [GitLab references](../../user/markdown.md#gitlab-specific-references). - Publishes any files attached to incident issue descriptions, up to 5000 per issue. - ([Introduced in GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/205166).) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205166) in GitLab 13.1.) After publication, you can access the incident's details page by clicking the **Published on status page** button displayed under the Incident's title. @@ -162,7 +162,7 @@ To publish comments to the Status Page Incident: adding a microphone [award emoji](../../user/award_emojis.md) reaction (`:microphone:` 🎤) to the comment. - Any files attached to the comment (up to 5000 per issue) are also published. - ([Introduced in GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/205166).) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205166) in GitLab 13.1.) WARNING: Anyone with access to view the Issue can add an emoji award to a comment, so diff --git a/doc/operations/index.md b/doc/operations/index.md index d170e82dd7f..ec54b6c57b9 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -36,7 +36,7 @@ Are your alerts too noisy? Alerts configured on GitLab metrics can configured and fine-tuned in GitLab immediately following a fire-fight. - [Manage alerts and incidents](incident_management/index.md) in GitLab. -- [Configure alerts for metrics](metrics/alerts.md#set-up-alerts-for-prometheus-metrics) in GitLab. +- [Configure alerts for metrics](metrics/alerts.md) in GitLab. - Create a [status page](incident_management/status_page.md) to communicate efficiently to your users during an incident. @@ -94,6 +94,6 @@ an environment. ## More features - Deploy to different [environments](../ci/environments/index.md). -- Connect your project to a [Kubernetes cluster](../user/project/clusters/index.md). +- Connect your project to a [Kubernetes cluster](../user/infrastructure/clusters/index.md). - See how your application is used and analyze events with [Product Analytics](product_analytics.md). -- Create, toggle, and remove [Feature Flags](feature_flags.md). **(PREMIUM)** +- Create, toggle, and remove [Feature Flags](feature_flags.md). diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index a3d741b78ea..44999ad7cd6 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -61,7 +61,7 @@ Prometheus server to use the ## Trigger actions from alerts **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4925) in GitLab 11.11. Alerts can be used to trigger actions, like opening an issue automatically (disabled by default since `13.1`). To configure the actions: @@ -83,7 +83,7 @@ values extracted from the [`alerts` field in webhook payload](https://prometheus - `full_query`: Alert query extracted from the payload's `generatorURL` field - Optional list of attached annotations extracted from `annotations/*` - Alert [GFM](../../user/markdown.md): GitLab Flavored Markdown from the payload's `annotations/gitlab_incident_markdown` field. -- Alert Severity (introduced in GitLab version [13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871): +- Alert Severity ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50871) in GitLab version 13.9): Extracted from the alert payload field `labels/severity`. Maps case-insensitive value to [Alert's severity](../incident_management/alerts.md#alert-severity): - **Critical**: `critical`, `s1`, `p1`, `emergency`, `fatal`, or any value not in this list @@ -107,7 +107,7 @@ of the project. ### Recovery alerts -> - [From GitLab Ultimate 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it automatically closes the associated issue. +> [From GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/13401), when GitLab receives a recovery alert, it automatically closes the associated issue. The alert in GitLab will be automatically resolved when Prometheus sends a payload with the field `status` set to `resolved`. diff --git a/doc/operations/metrics/dashboards/default.md b/doc/operations/metrics/dashboards/default.md index 2ba7a4e0d87..2be26e843c4 100644 --- a/doc/operations/metrics/dashboards/default.md +++ b/doc/operations/metrics/dashboards/default.md @@ -30,7 +30,7 @@ This dashboard requires Kubernetes v1.14 or higher, due to the in Kubernetes 1.14. This dashboard displays CPU, memory, network and disk metrics for the pods in your -[connected K8s cluster](../../../user/project/clusters/index.md). It provides a +[connected Kubernetes cluster](../../../user/infrastructure/clusters/index.md). It provides a [variable selector](templating_variables.md#metric_label_values-variable-type) at the top of the dashboard to select which pod's metrics to display. diff --git a/doc/operations/metrics/dashboards/panel_types.md b/doc/operations/metrics/dashboards/panel_types.md index dc96e2556ac..140e18b5b13 100644 --- a/doc/operations/metrics/dashboards/panel_types.md +++ b/doc/operations/metrics/dashboards/panel_types.md @@ -62,12 +62,10 @@ panel_groups: query_range: 'http_requests_total' label: '# of Requests' unit: 'count' - metrics: - id: anomaly_requests_upper_limit query_range: 10000 label: 'Max # of requests' unit: 'count' - metrics: - id: anomaly_requests_lower_limit query_range: 2000 label: 'Min # of requests' diff --git a/doc/operations/metrics/dashboards/variables.md b/doc/operations/metrics/dashboards/variables.md index 4b083284819..fa79524883d 100644 --- a/doc/operations/metrics/dashboards/variables.md +++ b/doc/operations/metrics/dashboards/variables.md @@ -8,7 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Query variables -Variables can be specified using double curly braces, such as `"{{ci_environment_slug}}"` ([added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20793) in GitLab 12.7). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20793) in GitLab 12.7. + +Variables can be specified using double curly braces, such as `"{{ci_environment_slug}}"`. Support for the `"%{ci_environment_slug}"` format was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31581) in GitLab 13.0. @@ -66,5 +68,5 @@ avg(sum(container_memory_usage_bytes{container_name!="{{pod}}"}) by (job)) witho The URL for this query would be: ```plaintext -http://gitlab.com/<user>/<project>/-/environments/<environment_id>/metrics?dashboard=.gitlab%2Fdashboards%2Fcustom.yml&pod=POD +https://gitlab.com/<user>/<project>/-/environments/<environment_id>/metrics?dashboard=.gitlab%2Fdashboards%2Fcustom.yml&pod=POD ``` diff --git a/doc/operations/metrics/dashboards/yaml.md b/doc/operations/metrics/dashboards/yaml.md index 45803598a40..8068a66d5c4 100644 --- a/doc/operations/metrics/dashboards/yaml.md +++ b/doc/operations/metrics/dashboards/yaml.md @@ -14,7 +14,7 @@ Dashboards have several components: The following tables outline the details of expected properties. -## **Dashboard (top-level) properties** +## Dashboard (top-level) properties | Property | Type | Required | Description | | ------ | ------ | ------ | ------ | @@ -23,7 +23,7 @@ The following tables outline the details of expected properties. | `templating` | hash | no | Top level key under which templating related options can be added. | | `links` | array | no | Add links to display on the dashboard. | -## **Templating (`templating`) properties** +## Templating (`templating`) properties | Property | Type | Required | Description | | -------- | ---- | -------- | ----------- | @@ -31,7 +31,7 @@ The following tables outline the details of expected properties. Read the documentation on [templating](templating_variables.md). -## **Links (`links`) properties** +## Links (`links`) properties | Property | Type | Required | Description | | -------- | ---- | -------- | ----------- | @@ -41,7 +41,7 @@ Read the documentation on [templating](templating_variables.md). Read the documentation on [links](index.md#add-related-links-to-custom-dashboards). -## **Panel group (`panel_groups`) properties** +## Panel group (`panel_groups`) properties Dashboards display panel groups in the order they are listed in the dashboard YAML file. @@ -55,7 +55,7 @@ is no longer used. Panels in a panel group are laid out in rows consisting of two panels per row. An exception to this rule are single panels on a row: these panels take the full width of their containing row. -## **Panel (`panels`) properties** +## Panel (`panels`) properties Dashboards display panels in the order they are listed in the dashboard YAML file. @@ -72,7 +72,7 @@ is no longer used. | `metrics` | array | yes | The metrics which should be displayed in the panel. Any number of metrics can be displayed when `type` is `area-chart` or `line-chart`, whereas only 3 can be displayed when `type` is `anomaly-chart`. | | `links` | array | no | Add links to display on the chart's [context menu](index.md#chart-context-menu). | -## **Axis (`panels[].y_axis`) properties** +## Axis (`panels[].y_axis`) properties | Property | Type | Required | Description | | ----------- | ------ | ----------------------------- | -------------------------------------------------------------------- | @@ -80,7 +80,7 @@ is no longer used. | `format` | string | no, defaults to `engineering` | Unit format used. See the [full list of units](yaml_number_format.md). | | `precision` | number | no, defaults to `2` | Number of decimal places to display in the number. | | -## **Metrics (`metrics`) properties** +## Metrics (`metrics`) properties | Property | Type | Required | Description | | ------ | ------ | ------ | ------ | diff --git a/doc/operations/metrics/embed.md b/doc/operations/metrics/embed.md index d773d04f1cc..e84c190e08d 100644 --- a/doc/operations/metrics/embed.md +++ b/doc/operations/metrics/embed.md @@ -96,17 +96,17 @@ a chart corresponding to the query can be included if these requirements are met ## Embedding cluster health charts -> - [Introduced](<https://gitlab.com/gitlab-org/gitlab/-/issues/40997>) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. -> - [Moved](<https://gitlab.com/gitlab-org/gitlab/-/issues/208224>) to GitLab core in 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/40997) in GitLab 12.9. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) from GitLab Ultimate to GitLab Free in 13.2. -[Cluster Health Metrics](../../user/project/clusters/index.md#visualizing-cluster-health) +[Cluster Health Metrics](../../user/infrastructure/clusters/manage/clusters_health.md) can also be embedded in [GitLab-flavored Markdown](../../user/markdown.md). To embed a metric chart, include a link to that chart in the form `https://<root_url>/<project>/-/cluster/<cluster_id>?<query_params>` anywhere that GitLab-flavored Markdown is supported. To generate and copy a link to the chart, follow the instructions in the -[Cluster Health Metric documentation](../../user/project/clusters/index.md#visualizing-cluster-health). +[Cluster Health Metric documentation](../../user/infrastructure/clusters/manage/clusters_health.md). The following requirements must be met for the metric to unfurl: @@ -114,7 +114,7 @@ The following requirements must be met for the metric to unfurl: - Prometheus must be monitoring the cluster. - The user must be allowed access to the project cluster metrics. - The dashboards must be reporting data on the - [Cluster Health Page](../../user/project/clusters/index.md#visualizing-cluster-health) + [Cluster Health Page](../../user/infrastructure/clusters/manage/clusters_health.md) If the above requirements are met, then the metric unfurls as seen below. diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md index 473b335d4c5..81b1f8a3bc6 100644 --- a/doc/operations/metrics/embed_grafana.md +++ b/doc/operations/metrics/embed_grafana.md @@ -51,7 +51,7 @@ To set up the Grafana API in Grafana: section. 1. To enable the integration, check the **Active** checkbox. 1. For **Grafana URL**, enter the base URL of the Grafana instance. -1. For **API Token**, enter the Admin API token you just generated. +1. For **API Token**, enter the Administrator API token you just generated. 1. Click **Save Changes**. ### Generate a link to a panel diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index 07e71fe79ef..f09b9f35d88 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -111,7 +111,7 @@ dashboard is visible to authenticated and non-authenticated users. ## Adding custom metrics -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28527) to GitLab Free in 12.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28527) from GitLab Premium to GitLab Free in 12.10. Custom metrics can be monitored by adding them on the monitoring dashboard page. After saving them, they display on the environment metrics dashboard provided that either: diff --git a/doc/operations/product_analytics.md b/doc/operations/product_analytics.md index c89500ab92c..3ff33027042 100644 --- a/doc/operations/product_analytics.md +++ b/doc/operations/product_analytics.md @@ -50,8 +50,7 @@ Feature.disable(:product_analytics, Project.find(<project ID>)) After enabling the feature flag for Product Analytics, you can access the user interface: -1. Sign in to GitLab as a user with Reporter or greater - [permissions](../user/permissions.md). +1. Sign in to GitLab as a user with at least the Reporter [role](../user/permissions.md). 1. Navigate to **Monitor > Product Analytics**. The user interface contains: diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index 7435c0dd8a2..1593607cc98 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Tracing **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/7903) in GitLab Ultimate 11.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42645) to GitLab Free in 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/7903) in GitLab 11.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/42645) from GitLab Ultimate to GitLab Free in 13.5. Tracing provides insight into the performance and health of a deployed application, tracking each function or microservice that handles a given request. Tracing makes it easy to understand the diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index f7c64895c78..399ef40cb40 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -36,7 +36,7 @@ The following table describes the version types and their release cadence: | Version type | Description | Cadence | |:-------------|:------------|:--------| -| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 14.0 on June 22, 2021 (one month later than typical, details in [this issue](https://gitlab.com/gitlab-com/Product/-/issues/2337)). Subsequent major releases will be scheduled for May 22 each year, by default. | +| Major | For significant changes, or when any backward-incompatible changes are introduced to the public API. | Yearly. The next major release is GitLab 15.0 on May 22, 2022. GitLab schedules major releases on May 22 each year, by default. | | Minor | For when new backward-compatible functionality is introduced to the public API, a minor feature is introduced, or when a set of smaller features is rolled out. | Monthly on the 22nd. | | Patch | For backward-compatible bug fixes that fix incorrect behavior. See [Patch releases](#patch-releases). | As needed. | diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index 1b4e1e64562..e0b07abd4e9 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -1,13 +1,13 @@ --- -stage: none -group: unassigned +stage: Manage +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Project and group visibility +# Project and group visibility **(FREE)** -GitLab allows [Owners](../user/permissions.md) to set a project's or group's visibility as: +GitLab allows users with the Owner [role](../user/permissions.md) to set a project's or group's visibility as: - **Public** - **Internal** @@ -24,7 +24,7 @@ Public projects can be cloned **without any** authentication over HTTPS. They are listed in the public access directory (`/public`) for all users. -**Any signed-in user** has the [Guest role](../user/permissions.md) on the repository. +**Any signed-in user** has the Guest [role](../user/permissions.md) on the repository. NOTE: By default, `/public` is visible to unauthenticated users. However, if the @@ -39,7 +39,7 @@ Internal projects can be cloned by any signed-in user except They are also listed in the public access directory (`/public`), but only for signed-in users. Any signed-in users except [external users](../user/permissions.md#external-users) have the -[Guest role](../user/permissions.md) on the repository. +Guest [role](../user/permissions.md) on the repository. NOTE: From July 2019, the `Internal` visibility setting is disabled for new projects, groups, @@ -57,7 +57,7 @@ They appear in the public access directory (`/public`) for project members only. Prerequisite: -- You must have the Owner role for a project. +- You must have the Owner [role](../user/permissions.md) for a project. 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > General**. @@ -69,7 +69,7 @@ Prerequisite: Prerequisite: -- You must have the Owner role for a group. +- You must have the Owner [role](../user/permissions.md) for a group. 1. On the top bar, select **Menu > Groups** and find your project. 1. On the left sidebar, select **Settings > General**. @@ -77,11 +77,11 @@ Prerequisite: 1. Under **Visibility level** select either **Private**, **Internal**, or **Public**. 1. Select **Save changes**. -## Restrict use of public or internal projects +## Restrict use of public or internal projects **(FREE SELF)** You can restrict the use of visibility levels for users when they create a project or a snippet. This is useful to prevent users from publicly exposing their repositories by accident. The -restricted visibility settings do not apply to admin users. +restricted visibility settings do not apply to administrators. For details, see [Restricted visibility levels](../user/admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels). diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index 9a8c0d79395..3d0e4fdbe49 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -49,7 +49,7 @@ branch name. Your developers may not remember that policy, so they might push to various branches, and CI pipelines might not work as expected. By restricting the branch names globally in Push Rules, such mistakes are prevented. -Any branch name that doesn't match your push rule is rejected. +All branch names that don't match your push rule are rejected. Note that the name of your default branch is always allowed, regardless of the branch naming regular expression (regex) specified. GitLab is configured this way @@ -73,7 +73,7 @@ Some example regular expressions you can use in push rules: By default, GitLab restricts certain formats of branch names for security purposes. 40-character hexadecimal names, similar to Git commit hashes, are prohibited. -### Custom Push Rules **(FREE SELF)** +### Custom Push Rules **(PREMIUM SELF)** It's possible to create custom push rules rather than the push rules available in **Admin Area > Push Rules** by using more advanced server hooks. @@ -104,12 +104,12 @@ The following options are available: |---------------------------------|-------------| | Removal of tags with `git push` | Forbid users to remove Git tags with `git push`. Tags can be deleted through the web UI. | | Check whether the commit author is a GitLab user | Restrict commits to existing GitLab users (checked against their emails). | -| Reject unverified users **(PREMIUM)** | GitLab rejects any commit that was not committed by an authenticated user. | -| Check whether commit is signed through GPG **(PREMIUM)** | Reject commit when it is not signed through GPG. Read [signing commits with GPG](../user/project/repository/gpg_signed_commits/index.md). | +| Reject unverified users | GitLab rejects any commit that was not committed by an authenticated user. | +| Check whether commit is signed through GPG | Reject commit when it is not signed through GPG. Read [signing commits with GPG](../user/project/repository/gpg_signed_commits/index.md). | | Prevent pushing secret files | GitLab rejects any files that are likely to contain secrets. See the [forbidden file names](#prevent-pushing-secrets-to-the-repository). | | Require expression in commit messages | Only commit messages that match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | | Reject expression in commit messages | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | -| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow any branch name. | +| Restrict by branch name | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow all branch names. | | Restrict by commit author's email | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | | Prohibited file names | Any committed filenames that match this regular expression and do not already exist in the repository are not allowed to be pushed. Leave empty to allow any filenames. See [common examples](#prohibited-file-names). | | Maximum file size | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. Files tracked by Git LFS are exempted. | @@ -117,7 +117,7 @@ The following options are available: NOTE: GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules, and you can test them at the [regex101 regex tester](https://regex101.com/). -### Caveat to "Reject unsigned commits" push rule **(PREMIUM)** +### Caveat to "Reject unsigned commits" push rule This push rule ignores commits that are authenticated and created by GitLab (either through the UI or API). When the **Reject unsigned commits** push rule is diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index b6f772dee17..89b727cf570 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -61,7 +61,7 @@ including: - Container Registry images - GitLab Pages content - Snippets -- Group wikis **(PREMIUM)** +- [Group wikis](../user/project/wiki/index.md#group-wikis) Backups do not include: @@ -1129,14 +1129,14 @@ GitLab backup script to be too slow. If your GitLab instance has a lot of forked projects, the regular backup task also duplicates the Git data for all of them. In these cases, consider using file system snapshots as part of your backup strategy. -Example: Amazon EBS +Example: Amazon Elastic Block Store (EBS) > A GitLab server using Omnibus GitLab hosted on Amazon AWS. > An EBS drive containing an ext4 file system is mounted at `/var/opt/gitlab`. > In this case you could make an application backup by taking an EBS snapshot. > The backup includes all repositories, uploads and PostgreSQL data. -Example: LVM snapshots + rsync +Example: Logical Volume Manager (LVM) snapshots + rsync > A GitLab server using Omnibus GitLab, with an LVM logical volume mounted at `/var/opt/gitlab`. > Replicating the `/var/opt/gitlab` directory using rsync would not be reliable because too many files would change while rsync is running. @@ -1172,7 +1172,7 @@ For manually backing up the Git repository data on disk, there are multiple poss Git repositories must be copied in a consistent way. They should not be copied during concurrent write operations, as this can lead to inconsistencies or corruption issues. For more details, -[issue 270422](https://gitlab.com/gitlab-org/gitlab/-/issues/270422 "Provide documentation on preferred method of migrating Gitaly servers") +[issue #270422](https://gitlab.com/gitlab-org/gitlab/-/issues/270422 "Provide documentation on preferred method of migrating Gitaly servers") has a longer discussion explaining the potential problems. To prevent writes to the Git repository data, there are two possible approaches: @@ -1335,11 +1335,11 @@ that contain required, sensitive information. If the key is lost, GitLab can't decrypt those columns, preventing access to the following items: - [CI/CD variables](../ci/variables/index.md) -- [Kubernetes / GCP integration](../user/project/clusters/index.md) +- [Kubernetes / GCP integration](../user/infrastructure/clusters/index.md) - [Custom Pages domains](../user/project/pages/custom_domains_ssl_tls_certification/index.md) - [Project error tracking](../operations/error_tracking.md) - [Runner authentication](../ci/runners/index.md) -- [Project mirroring](../user/project/repository/repository_mirroring.md) +- [Project mirroring](../user/project/repository/mirror/index.md) - [Web hooks](../user/project/integrations/webhooks.md) In cases like CI/CD variables and runner authentication, you can experience @@ -1517,7 +1517,7 @@ err.message="unknown error" This issue is caused by the restore running as the unprivileged user `git`, which is unable to assign the correct ownership to the registry files during -the restore process ([issue 62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). +the restore process ([issue #62759](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759 "Incorrect permissions on registry filesystem after restore")). To get your registry working again: @@ -1548,7 +1548,7 @@ If this happens, examine the following: - If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values results in this error. -### `gitaly-backup` for repository backup and restore **(FREE SELF)** +### `gitaly-backup` for repository backup and restore > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.2. > - [Deployed behind a feature flag](../user/feature_flags.md), enabled by default. diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index df71b8791f8..80aa52ed5a4 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -12,7 +12,7 @@ common administration and operational processes. You can perform GitLab Rake tasks by using: -- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html) +- `gitlab-rake <raketask>` for [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html) installations. - `bundle exec rake <raketask>` for [source](../install/installation.md) installations. @@ -51,3 +51,14 @@ The following Rake tasks are available for use with GitLab: | [User management](user_management.md) | Perform user management tasks. | | [Webhooks administration](web_hooks.md) | Maintain project webhooks. | | [X.509 signatures](x509_signatures.md) | Update X.509 commit signatures, which can be useful if the certificate store changed. | + +To list all available Rake tasks: + +```shell +# Omnibus GitLab +sudo gitlab-rake -vT + +# Installations from source +cd /home/git/gitlab +sudo -u git -H bundle exec rake -vT RAILS_ENV=production +``` diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index 08f0005883b..f63c35ab475 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -32,7 +32,7 @@ sudo gitlab-rake gitlab:import:all_users_to_all_projects bundle exec rake gitlab:import:all_users_to_all_projects RAILS_ENV=production ``` -Admin users are added as maintainers. +Administrators are added as maintainers. ## Add user as a developer to all groups diff --git a/doc/security/README.md b/doc/security/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/security/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/security/asset_proxy.md b/doc/security/asset_proxy.md index d6b85eb5c9f..abeb5c401da 100644 --- a/doc/security/asset_proxy.md +++ b/doc/security/asset_proxy.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md index a8dee8f589a..801a294dd81 100644 --- a/doc/security/crime_vulnerability.md +++ b/doc/security/crime_vulnerability.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/security/index.md b/doc/security/index.md index 35e93fc2c55..832af93b95e 100644 --- a/doc/security/index.md +++ b/doc/security/index.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index 69223b5edb9..162346c8874 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts --- @@ -24,7 +24,7 @@ limitation. You can take steps to prevent unintentional sharing and information destruction. This limitation is the reason why only certain people are allowed to [add users to a project](../user/project/members/index.md) -and why only a GitLab admin can [force push a protected +and why only a GitLab administrator can [force push a protected branch](../user/project/protected_branches.md). <!-- ## Troubleshooting diff --git a/doc/security/project_import_decompressed_archive_size_limits.md b/doc/security/project_import_decompressed_archive_size_limits.md index 6510cf459be..3c5099b1f75 100644 --- a/doc/security/project_import_decompressed_archive_size_limits.md +++ b/doc/security/project_import_decompressed_archive_size_limits.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index 4894af1fa19..b0bebc5a956 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index 6045dece0c2..4585748ffc2 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -35,6 +35,8 @@ These are rate limits you can set in the Admin Area of your instance: - [User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) - [Package registry rate limits](../user/admin_area/settings/package_registry_rate_limits.md) - [Git LFS rate limits](../user/admin_area/settings/git_lfs_rate_limits.md) +- [Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md) +- [Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md) ## Non-configurable limits @@ -51,7 +53,7 @@ The **rate limit** is 5 requests per minute per user. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/commit/35bc85c3ca093fee58d60dacdc9ed1fd9a15adec) in GitLab 13.4. -There is a rate limit for [testing webhooks](../user/project/integrations/webhooks.md#testing-webhooks), which prevents abuse of the webhook functionality. +There is a rate limit for [testing webhooks](../user/project/integrations/webhooks.md#test-a-webhook), which prevents abuse of the webhook functionality. The **rate limit** is 5 requests per minute per user. diff --git a/doc/security/reset_user_password.md b/doc/security/reset_user_password.md index 344cfcae46a..8b89200e1a7 100644 --- a/doc/security/reset_user_password.md +++ b/doc/security/reset_user_password.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# How to reset user password +# How to reset user password **(FREE SELF)** There are a few ways to reset the password of a user. @@ -32,7 +32,7 @@ sudo gitlab-rake "gitlab:password:reset[johndoe]" NOTE: To reset the default admin password, run this Rake task with the username -`root`, which is the default username of that admin account. +`root`, which is the default username of that administrator account. ## Rails console @@ -110,7 +110,7 @@ password. If the username was changed to something else and has been forgotten, one possible way is to reset the password using Rails console with user ID `1` (in -almost all the cases, the first user is the default admin account). +almost all the cases, the first user is the default administrator account). <!-- ## Troubleshooting diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index 239949b5568..1f1c7457441 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -27,7 +27,7 @@ the minimum key length for each technology:  If a restriction is imposed on any key type, users cannot upload new SSH keys that don't meet the -requirement. Any existing keys that don't meet it are disabled but not removed and users cannot to +requirement. Any existing keys that don't meet it are disabled but not removed and users cannot pull or push code using them. An icon is visible to the user of a restricted key in the SSH keys section of their profile: diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index 4e72033fd77..2a971b21840 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Token overview +# GitLab Token overview **(FREE)** This document lists tokens used in GitLab, their purpose and, where applicable, security guidance. @@ -63,7 +63,7 @@ Project maintainers and owners can add or enable a deploy key for a project repo ## Runner registration tokens -Runner registration tokens are used to [register](https://docs.gitlab.com/runner/register/) a [runner](https://docs.gitlab.com/runner/) with GitLab. Group or project owners or instance admins can obtain them through the GitLab user interface. The registration token is limited to runner registration and has no further scope. +Runner registration tokens are used to [register](https://docs.gitlab.com/runner/register/) a [runner](https://docs.gitlab.com/runner/) with GitLab. Group or project owners or instance administrators can obtain them through the GitLab user interface. The registration token is limited to runner registration and has no further scope. You can use the runner registration token to add runners that execute jobs in a project or group. The runner has access to the project's code, so be careful when assigning project and group-level permissions. diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index a5b01a1b27d..61b26204599 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -44,13 +44,13 @@ Gitlab::CurrentSettings.update!('require_two_factor_authentication': false) ## Enforce 2FA for all users in a group **(FREE)** -> [Introduced in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24965) GitLab 12.0, 2FA settings for a group are also applied to subgroups. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24965) in GitLab 12.0, 2FA settings for a group are also applied to subgroups. To enforce 2FA only for certain groups: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions, LFS, 2FA** section. -1. Select the **Require all users in this group to setup two-factor authentication** option. +1. Select the **Require all users in this group to set up two-factor authentication** option. You can also specify a grace period in the **Time before enforced** option. diff --git a/doc/security/unlock_user.md b/doc/security/unlock_user.md index da451d96ef9..ceb375a9ad1 100644 --- a/doc/security/unlock_user.md +++ b/doc/security/unlock_user.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md index 09e1e09b676..48538e413b4 100644 --- a/doc/security/user_email_confirmation.md +++ b/doc/security/user_email_confirmation.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# User email confirmation at sign-up +# User email confirmation at sign-up **(FREE SELF)** GitLab can be configured to require confirmation of a user's email address when the user signs up. When this setting is enabled, the user is unable to sign in until diff --git a/doc/security/user_file_uploads.md b/doc/security/user_file_uploads.md index bce2aeb88b4..7a8a78cc5f8 100644 --- a/doc/security/user_file_uploads.md +++ b/doc/security/user_file_uploads.md @@ -5,7 +5,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# User File Uploads +# User File Uploads **(FREE)** Images that are attached to issues, merge requests, or comments do not require authentication to be viewed if they are accessed directly by URL. diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index c0e5d0695cc..89dd4f8e5fc 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, reference, howto --- @@ -74,7 +74,7 @@ allowlist: The allowed entries can be separated by semicolons, commas or whitespaces (including newlines) and be in different formats like hostnames, IP addresses and/or IP ranges. IPv6 is supported. Hostnames that contain Unicode characters should -use IDNA encoding. +use Internationalising Domain Names in Applications (IDNA) encoding. The allowlist can hold a maximum of 1000 entries. Each entry can be a maximum of 255 characters. diff --git a/doc/ssh/README.md b/doc/ssh/README.md deleted file mode 100644 index 0e6c2f63f9e..00000000000 --- a/doc/ssh/README.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-09-28' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-09-28. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ssh/index.md b/doc/ssh/index.md index 94c157697ce..2fae3512b9d 100644 --- a/doc/ssh/index.md +++ b/doc/ssh/index.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: howto, reference --- -# GitLab and SSH keys +# GitLab and SSH keys **(FREE)** Git is a distributed version control system, which means you can work locally, then share or "push" your changes to a server. In this case, the server is GitLab. @@ -213,7 +213,7 @@ To use SSH with GitLab, copy your public key to your GitLab account. which starts with `ssh-ed25519` or `ssh-rsa`, and may end with a comment. 1. In the **Title** box, type a description, like `Work Laptop` or `Home Workstation`. -1. Optional. In the **Expires at** box, select an expiration date. (Introduced in [GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/36243).) +1. Optional. In the **Expires at** box, select an expiration date. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36243) in GitLab 12.9.) In: - GitLab 13.12 and earlier, the expiration date is informational only. It doesn't prevent you from using the key. Administrators can view expiration dates and use them for @@ -326,7 +326,7 @@ If you are using [EGit](https://www.eclipse.org/egit/), you can [add your SSH ke ## Use SSH on Microsoft Windows -If you're running Windows 10, you can either use the [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install-win10) +If you're running Windows 10, you can either use the [Windows Subsystem for Linux (WSL)](https://docs.microsoft.com/en-us/windows/wsl/install) with [WSL 2](https://docs.microsoft.com/en-us/windows/wsl/install-win10#update-to-wsl-2) which has both `git` and `ssh` preinstalled, or install [Git for Windows](https://gitforwindows.org) to use SSH through Powershell. diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 327fb8887ad..bb2a2303d29 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -68,7 +68,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Full code quality reports in the code quality tab](../user/project/merge_requests/code_quality.md#code-quality-reports) - [Merge request approvals](../user/project/merge_requests/approvals/index.md) - [Multiple assignees](../user/project/merge_requests/getting_started.md#multiple-assignees) - - [Approval Rule information for Reviewers](../user/project/merge_requests/reviews/index.md#approval-rule-information-for-reviewers) **(PREMIUM)** + - [Approval Rule information for Reviewers](../user/project/merge_requests/reviews/index.md#approval-rule-information-for-reviewers) - [Required Approvals](../user/project/merge_requests/approvals/index.md#required-approvals) - [Code Owners as eligible approvers](../user/project/merge_requests/approvals/rules.md#code-owners-as-eligible-approvers) - [Approval rules](../user/project/merge_requests/approvals/rules.md) features @@ -89,14 +89,14 @@ the tiers are no longer mentioned in GitLab documentation: - Repositories: - [Repository size limit](../user/admin_area/settings/account_and_limit_settings.md#repository-size-limit) - Repository mirroring: - - [Pull mirroring](../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository) outside repositories in a GitLab repository - - [Overwrite diverged branches](../user/project/repository/repository_mirroring.md#overwrite-diverged-branches) - - [Trigger pipelines for mirror updates](../user/project/repository/repository_mirroring.md#trigger-pipelines-for-mirror-updates) - - [Hard failures](../user/project/repository/repository_mirroring.md#hard-failure) when mirroring fails - - [Trigger pull mirroring from the API](../user/project/repository/repository_mirroring.md#trigger-an-update-using-the-api) - - [Mirror only protected branches](../user/project/repository/repository_mirroring.md#mirror-only-protected-branches) - - [Bidirectional mirroring](../user/project/repository/repository_mirroring.md#bidirectional-mirroring) - - [Mirror with Perforce Helix via Git Fusion](../user/project/repository/repository_mirroring.md#mirror-with-perforce-helix-via-git-fusion) + - [Pull mirroring](../user/project/repository/mirror/pull.md) outside repositories in a GitLab repository + - [Overwrite diverged branches](../user/project/repository/mirror/pull.md#overwrite-diverged-branches) + - [Trigger pipelines for mirror updates](../user/project/repository/mirror/pull.md#trigger-pipelines-for-mirror-updates) + - [Hard failures](../user/project/repository/mirror/pull.md#hard-failure) when mirroring fails + - [Trigger pull mirroring from the API](../user/project/repository/mirror/pull.md#trigger-an-update-by-using-the-api) + - [Mirror only protected branches](../user/project/repository/mirror/index.md#mirror-only-protected-branches) + - [Bidirectional mirroring](../user/project/repository/mirror/bidirectional.md) + - [Mirror with Perforce Helix via Git Fusion](../user/project/repository/mirror/bidirectional.md#mirror-with-perforce-helix-via-git-fusion) - Runners: - Run pipelines in the parent project [for merge requests from a forked project](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project) - [Shared runners pipeline minutes quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index cecbb362cb9..a8e02251b64 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -9,115 +9,85 @@ type: index, reference GitLab SaaS is the GitLab software-as-a-service offering, which is available at GitLab.com. You don't need to install anything to use GitLab SaaS, you only need to -[sign up](https://gitlab.com/users/sign_up). +[sign up](https://gitlab.com/users/sign_up). When you sign up, you choose: -This page reviews the details of your GitLab SaaS subscription. +- [A license tier](https://about.gitlab.com/pricing/). +- [The number of seats you want](#how-seat-usage-is-determined). -## Choose a GitLab SaaS tier - -Pricing is [tier-based](https://about.gitlab.com/pricing/), so you can choose -the features that fit your budget. For information on the features available -for each tier, see the -[GitLab SaaS feature comparison](https://about.gitlab.com/pricing/gitlab-com/feature-comparison/). - -## Choose the number of users - -NOTE: -Applied only to groups. - -A GitLab SaaS subscription uses a concurrent (_seat_) model. You pay for a -subscription according to the maximum number of users enabled at once. You can -add and remove users during the subscription period, as long as the total users -at any given time doesn't exceed the subscription count. - -Every occupied seat is counted in the subscription, with the following exception: - -- Members with Guest permissions on an Ultimate subscription. - -NOTE: -To support the open source community and encourage the development of open -source projects, GitLab grants access to **Ultimate** features for all GitLab SaaS -**public** projects, regardless of the subscription. GitLab also provides qualifying -open source projects with 50,000 CI minutes and free access to the Ultimate tier for -groups through the [GitLab for Open Source program](https://about.gitlab.com/solutions/open-source/). +All GitLab SaaS public projects, regardless of the subscription, get access to features in the **Ultimate** tier. +Qualifying open source projects also get 50,000 CI minutes and free access to the **Ultimate** tier +through the [GitLab for Open Source program](https://about.gitlab.com/solutions/open-source/). ## Obtain a GitLab SaaS subscription To subscribe to GitLab SaaS: -1. Create a user account for yourself using our +1. View the [GitLab SaaS feature comparison](https://about.gitlab.com/pricing/gitlab-com/feature-comparison/) + and decide which tier you want. +1. Create a user account for yourself by using the [sign up page](https://gitlab.com/users/sign_up). -1. Create a [group](../../user/group/index.md). GitLab groups help assemble related - projects together allowing you to grant members access to several projects - at once. A group is not required if you plan on having projects inside a personal - namespace. +1. Create a [group](../../user/group/index.md#create-a-group). You use the group to grant users access to several projects + at once. A group is not required if you plan to have projects in a personal namespace instead. 1. Create additional users and [add them to the group](../../user/group/index.md#add-users-to-a-group). -1. Select the GitLab SaaS plan through the - [Customers Portal](https://customers.gitlab.com/). -1. Link your GitLab SaaS account with your Customers Portal account. - Once a plan has been selected, if your account is not - already linked, GitLab prompts you to link your account with a - **Sign in to GitLab.com** button. -1. Select the namespace from the drop-down list to associate the subscription. -1. Proceed to checkout. - -NOTE: -You can also go to the [**My Account**](https://customers.gitlab.com/customers/edit) -page to add or change the GitLab SaaS account link. +1. On the left sidebar, select **Billing** and choose a tier. +1. Fill out the form to complete your purchase. ## View your GitLab SaaS subscription -To see the status of your GitLab SaaS subscription, log in to GitLab SaaS and go -to the **Billing** section: +Prerequisite: + +- You must have the Owner [role](../../user/permissions.md) for the group. -NOTE: -You must have Owner level [permissions](../../user/permissions.md) to view the billing page. +To see the status of your GitLab SaaS subscription: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > Billing**. -The following table describes details of your subscription: +The following information is displayed: | Field | Description | |:----------------------------|:------------| | **Seats in subscription** | If this is a paid plan, represents the number of seats you've bought for this group. | -| **Seats currently in use** | Number of seats in use. Select **See usage** to see a list of the users using these seats. For more details, see [Seat usage](#seat-usage). | +| **Seats currently in use** | Number of seats in use. Select **See usage** to see a list of the users using these seats. | | **Max seats used** | Highest number of seats you've used. | -| **Seats owed** | _Seats owed_ = _Max seats used_ - _Seats in subscription_. | +| **Seats owed** | **Max seats used** minus **Seats in subscription**. | | **Subscription start date** | Date your subscription started. If this is for a Free plan, it's the date you transitioned off your group's paid plan. | | **Subscription end date** | Date your current subscription ends. Does not apply to Free plans. | -## Seat usage +## How seat usage is determined > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216899) in GitLab 13.5. > - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/292086) in GitLab 13.8 to include public email address. -The **Seat usage** page lists all users occupying seats. Details for each user include: - -- Full name -- Username -- Public email address (if they have provided one in their [user settings](../../user/profile/index.md#access-your-user-settings)) - -The seat usage listing is updated live, but the usage statistics on the billing page are updated -only once per day. For this reason there can be a minor difference between the seat usage listing -and the billing page. +A GitLab SaaS subscription uses a concurrent (_seat_) model. You pay for a +subscription according to the maximum number of users enabled at one time. You can +add and remove users during the subscription period, as long as the total users +at any given time doesn't exceed the subscription count. Every user is included in seat usage, with the following exceptions: - Users who are pending approval. - Members with the Guest role on an Ultimate subscription. -- Users without project or group memberships on an Ultimate subscription. - GitLab-created service accounts: `Ghost User` and bots ([`Support Bot`](../../user/project/service_desk.md#support-bot-user), [`Project bot users`](../../user/project/settings/project_access_tokens.md#project-bot-users), and so on.) +Seat usage is reviewed [quarterly or annually](../quarterly_reconciliation.md). + ### View seat usage To view a list of seats being used: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > Usage Quotas**. -1. Under the **Seats** tab, view usage information. +1. On the **Seats** tab, view usage information. + +The seat usage listing is updated live, but the usage statistics on the billing page are updated +only once per day. For this reason there can be a minor difference between the seat usage listing +and the billing page. ### Search seat usage @@ -149,53 +119,14 @@ To export seat usage data as a CSV file: The generated list contains all seats being used, and is not affected by the current search. -## Subscription expiry +## Seats owed -When your subscription expires, you can continue to use paid features of GitLab for 14 days. -On the 15th day, paid features are no longer available. You can -continue to use free features. +A GitLab subscription is valid for a specific number of users. -To resume paid feature functionality, purchase a new subscription. +If the number of billable users exceeds the number included in the subscription, known +as the number of **seats owed**, you must pay for the excess number of users before renewal. -## Renew your GitLab SaaS subscription - -To renew your subscription: - -1. [Prepare for renewal by reviewing your account.](#prepare-for-renewal-by-reviewing-your-account) -1. [Renew your GitLab SaaS subscription.](#renew-or-change-a-gitlab-saas-subscription) - -### Prepare for renewal by reviewing your account - -The [Customers Portal](https://customers.gitlab.com/customers/sign_in) is your -tool for renewing and modifying your subscription. Before going ahead with renewal, -log in and verify or update: - -- The invoice contact details on the **Account details** page. -- The credit card on file on the **Payment Methods** page. - -NOTE: -Contact our [support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) -if you need assistance accessing the Customers Portal or if you need to change -the contact person who manages your subscription. - -It's important to regularly review your user accounts, because: - -- A GitLab subscription is based on the number of users. You could pay more than - you should if you renew for too many users, while the renewal fails if you - attempt to renew a subscription for too few users. -- Stale user accounts can be a security risk. A regular review helps reduce this risk. - -#### Seats owed - -A GitLab subscription is valid for a specific number of users. For details, see -[Choose the number of users](#choose-the-number-of-users). - -If the number of [billable users](#view-your-gitlab-saas-subscription) exceeds the number included in the subscription, known -as the number of _seats owed_, you must pay for the excess number of users before renewal. - -##### Seats owed example - -You purchase a subscription for 10 users. +For example, if you purchase a subscription for 10 users: | Event | Billable members | Maximum users | |:---------------------------------------------------|:-----------------|:--------------| @@ -205,44 +136,7 @@ You purchase a subscription for 10 users. Seats owed = 12 - 10 (Maximum users - users in subscription) -### Renew or change a GitLab SaaS subscription - -Starting 30 days before a subscription expires, GitLab notifies group owners -of the date of expiry with a banner in the GitLab user interface. - -We recommend following these steps during renewal: - -1. Prune any [inactive or unwanted users](#remove-billable-user). -1. Determine if you have a need for user growth in the upcoming subscription. -1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select the **Renew** button. -1. Review your renewal details and complete the payment process. -1. Select **Confirm purchase**. - -Your updated subscription is applied to your namespace on the renewal period start date. - -An invoice is generated for the renewal and available for viewing or download on the [View invoices](https://customers.gitlab.com/receipts) page. If you have difficulty during the renewal process, contact our [support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. - -For details on upgrading your subscription tier, see -[Upgrade your GitLab SaaS subscription tier](#upgrade-your-gitlab-saas-subscription-tier). - -#### Automatic renewal - -To view or change automatic subscription renewal (at the same tier as the -previous period), log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in), and: - -- If you see a **Resume subscription** button, your subscription was canceled - previously. Click it to resume automatic renewal. -- If you see **Cancel subscription**, your subscription is set to automatically - renew at the end of the subscription period. Click it to cancel automatic renewal. - -With automatic renewal enabled, the subscription automatically renews on the -expiration date without a gap in available service. An invoice is -generated for the renewal and available for viewing or download in the -[View invoices](https://customers.gitlab.com/receipts) page. If you have difficulty -during the renewal process, contact our -[support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. - -## Add users to your subscription +### Add users to your subscription You can add users to your subscription at any time during the subscription period. The cost of additional users added during the subscription period is prorated from the date of purchase through @@ -264,6 +158,21 @@ The following is emailed to you: - A payment receipt. You can also access this information in the Customers Portal under [**View invoices**](https://customers.gitlab.com/receipts). +### Remove users from your subscription + +To remove a billable user from your subscription: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > Billing**. +1. In the **Seats currently in use** section, select **See usage**. +1. In the row for the user you want to remove, on the right side, select the ellipsis and **Remove user**. +1. Re-type the username and select **Remove user**. + +If you add a member to a group by using the [share a group with another group](../../user/group/index.md#share-a-group-with-another-group) feature, you can't remove the member by using this method. Instead, you can either: + +- Remove the member from the shared group. You must be a group owner to do this. +- From the group's membership page, remove access from the entire shared group. + ## Upgrade your GitLab SaaS subscription tier To upgrade your [GitLab tier](https://about.gitlab.com/pricing/): @@ -278,26 +187,73 @@ To upgrade your [GitLab tier](https://about.gitlab.com/pricing/): When the purchase has been processed, you receive confirmation of your new subscription tier. -## See your subscription and billable users in GitLab.com +## Subscription expiry -To view information about your subscription and occupied seats: +When your subscription expires, you can continue to use paid features of GitLab for 14 days. +On the 15th day, paid features are no longer available. You can +continue to use free features. -1. Go to your group's **Settings > Billing** page. -1. In the **Seats currently in use** area, select **See usage**. +To resume paid feature functionality, purchase a new subscription. -### Remove billable user +## Renew your GitLab SaaS subscription -To remove a billable user: +To renew your subscription: -1. Go to your group's **Settings > Billing** page. -1. In the **Seats currently in use** area, select **See usage**. -1. In the row for the user you want to remove, on the right side, select the ellipsis and **Remove user**. -1. Re-type the username and select **Remove user**. +1. [Prepare for renewal by reviewing your account.](#prepare-for-renewal-by-reviewing-your-account) +1. [Renew your GitLab SaaS subscription.](#renew-or-change-a-gitlab-saas-subscription) -If you add a member to a group by using the [share a group with another group](../../user/group/index.md#share-a-group-with-another-group) feature, you can't remove the member by using this method. Instead, you can either: +### Prepare for renewal by reviewing your account -- Remove the member from the shared group. You must be a group owner to do this. -- From the group's membership page, remove access from the entire shared group. +Before you renew your subscription: + +1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. On the **Account details** page, verify or update the invoice contact details. +1. On the **Payment Methods** page, verify or update the credit card on file. +1. In GitLab, review your list of user accounts and [remove inactive or unwanted users](#remove-users-from-your-subscription). + +### Renew or change a GitLab SaaS subscription + +Starting 30 days before a subscription expires, GitLab notifies group owners +of the date of expiry with a banner in the GitLab user interface. + +To renew your subscription: + +1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and beneath your existing subscription, select **Renew**. +1. Review your renewal details and complete the payment process. +1. Select **Confirm purchase**. + +Your updated subscription is applied to your namespace on the renewal period start date. + +An invoice is generated for the renewal and available for viewing or download on the [View invoices](https://customers.gitlab.com/receipts) page. +If you have difficulty during the renewal process, contact the [Support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. + +For details on upgrading your subscription tier, see +[Upgrade your GitLab SaaS subscription tier](#upgrade-your-gitlab-saas-subscription-tier). + +### Automatic renewal + +When you enable automatic renewal, the subscription automatically renews on the +expiration date without a gap in available service. An invoice is +generated for the renewal and available for viewing or download on the +[View invoices](https://customers.gitlab.com/receipts) page. + +#### Enable automatic renewal + +To view or change automatic subscription renewal (at the same tier as the +previous period), log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in), and: + +- If a **Resume subscription** button is displayed, your subscription was canceled + previously. Click it to resume automatic renewal. +- If a **Cancel subscription** button is displayed, your subscription is set to automatically + renew at the end of the subscription period. Click it to cancel automatic renewal. + +If you have difficulty during the renewal process, contact the +[Support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293) for assistance. + +## Change the contact person for your subscription + +To change the contact person who manages your subscription, +contact the GitLab [Support team](https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000071293). ## CI pipeline minutes @@ -389,7 +345,8 @@ Be aware that: be charged for less than one year because your subscription was previously created with the extra CI minutes. - After the extra CI minutes have been assigned to a Group, they can't be transferred - to a different Group. + to a different Group by themselves, but they will transfer along with a subscription when + changing the linked namespace for the subscription. - If you have used more minutes than your default quota, these minutes will be deducted from your Additional Minutes quota immediately after your purchase of additional minutes. @@ -426,11 +383,6 @@ To purchase more storage for either a personal or group namespace: The **Purchased storage available** total is incremented by the amount purchased. All locked projects are unlocked and their excess usage is deducted from the additional storage. -## Customers Portal - -The GitLab [Customers Portal](../index.md#customers-portal) enables you to manage your subscriptions -and account details. - ## Contact Support Learn more about: @@ -438,7 +390,7 @@ Learn more about: - The tiers of [GitLab Support](https://about.gitlab.com/support/). - [Submit a request via the Support Portal](https://support.gitlab.com/hc/en-us/requests/new). -We also encourage all users to search our project trackers for known issues and +We also encourage you to search our project trackers for known issues and existing feature requests in the [GitLab](https://gitlab.com/gitlab-org/gitlab/-/issues/) project. These issues are the best avenue for getting updates on specific product plans diff --git a/doc/subscriptions/quarterly_reconciliation.md b/doc/subscriptions/quarterly_reconciliation.md index f9cca079e76..cdac50748d0 100644 --- a/doc/subscriptions/quarterly_reconciliation.md +++ b/doc/subscriptions/quarterly_reconciliation.md @@ -4,22 +4,24 @@ group: Purchase info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# The quarterly subscription reconciliation process +# Quarterly reconciliation and annual true-ups **(PREMIUM)** -GitLab reviews your seat usage every quarter and sends you an invoice for -any overages. +GitLab reviews your seat usage and sends you an invoice for any overages. +This review can occur: -Annual reviews, also known as the [annual true-up process](self_managed/index.md#users-over-license), -require you to pay the full annual subscription fee for users added anytime during the year. With quarterly -reviews, you only pay for the remaining period of your subscription term. +- **Annually**, also known as the annual true-up process. This process requires you to pay the full annual subscription fee + for users added anytime during the year. +- **Quarterly**. With this process, you only pay for the remaining period of your subscription term. -For example, if you add users in the third quarter of your subscription term, you only -pay 25% of what you would have paid previously. This results in substantial savings. +## Quarterly reconciliation process + +With the quarterly reconciliation process, if you add users in the third quarter of your subscription term, for example, +you only pay 25% of what you would have paid previously. This calculation results in substantial savings. If it's not possible for you to participate in quarterly reconciliations, you can opt out of the process by using a contract amendment. In that case, you default to the annual review. -## Timeline for invoicing and payment +## Timeline for quarterly invoicing and payment At the end of each subscription quarter, GitLab notifies you about overages. The date you're notified about the overage is not the same as the date @@ -28,7 +30,7 @@ you are billed. ### GitLab SaaS Group owners receive an email **on the reconciliation date**. -The email communicates the [overage seat quantity](gitlab_com/index.md#seats-owed-example) +The email communicates the [overage seat quantity](gitlab_com/index.md#seats-owed) and expected invoice amount. **Seven days later**, the subscription is updated to include the additional diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index 72bd1c2b4f7..acdbdefc671 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -25,8 +25,8 @@ changes to their subscription. The cost of a GitLab self-managed subscription is determined by the following: -- GitLab tier -- Subscription seats +- [GitLab tier](https://about.gitlab.com/pricing/) +- [Subscription seats](#subscription-seats) ## Choose a GitLab tier @@ -108,6 +108,7 @@ GitLab has several features which can help you manage the number of users: - Enable the [**Require administrator approval for new sign ups**](../../user/admin_area/settings/sign_up_restrictions.md#require-administrator-approval-for-new-sign-ups) option. +- Enable `block_auto_created_users` for new sign-ups via [LDAP](../../administration/auth/ldap/index.md#basic-configuration-settings) or [OmniAuth](../../integration/omniauth.md#initial-omniauth-configuration). - Enable the [User cap](../../user/admin_area/settings/sign_up_restrictions.md#user-cap) option. **Available in GitLab 13.7 and later**. - [Disable new sign-ups](../../user/admin_area/settings/sign_up_restrictions.md), and instead manage new diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 7dd0701329d..fbeee7b96bc 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# System hooks +# System hooks **(FREE SELF)** Your GitLab instance can perform HTTP POST requests on the following events: @@ -173,7 +173,7 @@ Please refer to `group_rename` and `user_rename` for that case. "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, - "project_visibility": "visibilitylevel|private" + "project_visibility": "private" } ``` @@ -193,7 +193,7 @@ Please refer to `group_rename` and `user_rename` for that case. "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, - "project_visibility": "visibilitylevel|private" + "project_visibility": "private" } ``` @@ -213,7 +213,7 @@ Please refer to `group_rename` and `user_rename` for that case. "user_name": "John Smith", "user_username": "johnsmith", "user_id": 41, - "project_visibility": "visibilitylevel|private" + "project_visibility": "private" } ``` @@ -519,7 +519,7 @@ X-Gitlab-Event: System Hook } ``` -### Merge request events +## Merge request events Triggered when a new merge request is created, an existing merge request was updated/merged/closed or a commit is added in the source branch. @@ -533,101 +533,89 @@ X-Gitlab-Event: System Hook ```json { "object_kind": "merge_request", + "event_type": "merge_request", "user": { "id": 1, "name": "Administrator", "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", "email": "admin@example.com" }, "project": { - "name": "Example", - "description": "", - "web_url": "http://example.com/jsmith/example", - "avatar_url": null, - "git_ssh_url": "git@example.com:jsmith/example.git", - "git_http_url": "http://example.com/jsmith/example.git", - "namespace": "Jsmith", - "visibility_level": 0, - "path_with_namespace": "jsmith/example", - "default_branch": "master", - "ci_config_path": "", - "homepage": "http://example.com/jsmith/example", - "url": "git@example.com:jsmith/example.git", - "ssh_url": "git@example.com:jsmith/example.git", - "http_url": "http://example.com/jsmith/example.git" + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" }, "object_attributes": { - "id": 90, + "id": 99, "target_branch": "master", "source_branch": "ms-viewport", "source_project_id": 14, "author_id": 51, "assignee_id": 6, "title": "MS-Viewport", - "created_at": "2017-09-20T08:31:45.944Z", - "updated_at": "2017-09-28T12:23:42.365Z", + "created_at": "2013-12-03T17:23:34Z", + "updated_at": "2013-12-03T17:23:34Z", "milestone_id": null, "state": "opened", "merge_status": "unchecked", "target_project_id": 14, "iid": 1, "description": "", - "updated_by_id": 1, - "merge_error": null, - "merge_params": { - "force_remove_source_branch": "0" - }, - "merge_when_pipeline_succeeds": false, - "merge_user_id": null, - "merge_commit_sha": null, - "deleted_at": null, - "in_progress_merge_commit_sha": null, - "lock_version": 5, - "time_estimate": 0, - "last_edited_at": "2017-09-27T12:43:37.558Z", - "last_edited_by_id": 1, - "head_pipeline_id": 61, - "ref_fetched": true, - "merge_jid": null, "source": { - "name": "Awesome Project", - "description": "", - "web_url": "http://example.com/awesome_space/awesome_project", - "avatar_url": null, - "git_ssh_url": "git@example.com:awesome_space/awesome_project.git", - "git_http_url": "http://example.com/awesome_space/awesome_project.git", - "namespace": "root", - "visibility_level": 0, - "path_with_namespace": "awesome_space/awesome_project", - "default_branch": "master", - "ci_config_path": "", - "homepage": "http://example.com/awesome_space/awesome_project", - "url": "http://example.com/awesome_space/awesome_project.git", - "ssh_url": "git@example.com:awesome_space/awesome_project.git", - "http_url": "http://example.com/awesome_space/awesome_project.git" + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" }, "target": { - "name": "Awesome Project", - "description": "Aut reprehenderit ut est.", - "web_url": "http://example.com/awesome_space/awesome_project", - "avatar_url": null, - "git_ssh_url": "git@example.com:awesome_space/awesome_project.git", - "git_http_url": "http://example.com/awesome_space/awesome_project.git", - "namespace": "Awesome Space", - "visibility_level": 0, - "path_with_namespace": "awesome_space/awesome_project", - "default_branch": "master", - "ci_config_path": "", - "homepage": "http://example.com/awesome_space/awesome_project", - "url": "http://example.com/awesome_space/awesome_project.git", - "ssh_url": "git@example.com:awesome_space/awesome_project.git", - "http_url": "http://example.com/awesome_space/awesome_project.git" + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" }, "last_commit": { - "id": "ba3e0d8ff79c80d5b0bbb4f3e2e343e0aaa662b7", + "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "message": "fixed readme", - "timestamp": "2017-09-26T16:12:57Z", + "timestamp": "2012-01-03T23:36:29+02:00", "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "author": { "name": "GitLab dev user", @@ -635,16 +623,61 @@ X-Gitlab-Event: System Hook } }, "work_in_progress": false, - "total_time_spent": 0, - "human_total_time_spent": null, - "human_time_estimate": null + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open", + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + } }, - "labels": null, - "repository": { - "name": "git-gpg-test", - "url": "git@example.com:awesome_space/awesome_project.git", - "description": "", - "homepage": "http://example.com/awesome_space/awesome_project" + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "changes": { + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, + "labels": { + "previous": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "current": [{ + "id": 205, + "title": "Platform", + "color": "#123123", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "Platform related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + } } } ``` diff --git a/doc/topics/application_development_platform/index.md b/doc/topics/application_development_platform/index.md index f9baa8916df..1560ceeed26 100644 --- a/doc/topics/application_development_platform/index.md +++ b/doc/topics/application_development_platform/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Application Development Platform +# Application Development Platform **(FREE)** The GitLab Application Development Platform refers to the set of GitLab features used to create, configure, and manage a complete software development environment. It provides development, operations, and security teams with a robust feature set aimed at supporting best practices out of the box. diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 25e6e590f77..da96e88bb21 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -43,7 +43,7 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Personal access tokens](../../api/index.md#personalproject-access-tokens) - [Project access tokens](../../api/index.md#personalproject-access-tokens) - [Impersonation tokens](../../api/index.md#impersonation-tokens) -- [GitLab as an OAuth2 provider](../../api/oauth2.md#gitlab-as-an-oauth-20-provider) +- [OAuth 2.0 identity provider API](../../api/oauth2.md) ## Third-party resources diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index f8b63f5b41a..c01ed4a49d0 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -58,7 +58,7 @@ If your goal is to use only a single custom buildpack, you should provide the pr ## Custom `Dockerfile` -> Support for `DOCKERFILE_PATH` was [added in GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35662) +> Support for `DOCKERFILE_PATH` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35662) in GitLab 13.2 If your project has a `Dockerfile` in the root of the project repository, Auto DevOps builds a Docker image based on the Dockerfile, rather than using buildpacks. @@ -715,7 +715,7 @@ The banner can be disabled for: Feature.enable(:auto_devops_banner_disabled) ``` - - Through the REST API with an admin access token: + - Through the REST API with an administrator access token: ```shell curl --data "value=true" --header "PRIVATE-TOKEN: <personal_access_token>" "https://gitlab.example.com/api/v4/features/auto_devops_banner_disabled" diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index e232af05d50..9340f89c502 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -131,7 +131,7 @@ following levels: | Instance type | [Project](#at-the-project-level) | [Group](#at-the-group-level) | [Instance](#at-the-instance-level) (Admin Area) | |---------------------|------------------------|------------------------|------------------------| -| GitLab SaaS | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | +| GitLab SaaS | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | | GitLab self-managed | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Before enabling Auto DevOps, consider [preparing it for deployment](requirements.md). If you don't, Auto DevOps can build and test your app, diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 59c61da0c45..c84b5e4d9c7 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -103,7 +103,7 @@ to deploy this project to. [Cloud Run](../../user/project/clusters/add_gke_clusters.md#cloud-run-for-anthos), Istio, and HTTP Load Balancing add-ons for this cluster. - **GitLab-managed cluster** - Select this checkbox to - [allow GitLab to manage namespace and service accounts](../../user/project/clusters/index.md#gitlab-managed-clusters) for this cluster. + [allow GitLab to manage namespace and service accounts](../../user/project/clusters/gitlab_managed_clusters.md) for this cluster. 1. Select **Create Kubernetes cluster**. @@ -199,13 +199,13 @@ The jobs are separated into stages: vulnerabilities and is allowed to fail ([Auto Container Scanning](stages.md#auto-container-scanning)) - The `dependency_scanning` job checks if the application has any dependencies susceptible to vulnerabilities and is allowed to fail - ([Auto Dependency Scanning](stages.md#auto-dependency-scanning)) **(ULTIMATE)** + ([Auto Dependency Scanning](stages.md#auto-dependency-scanning)) - Jobs suffixed with `-sast` run static analysis on the current code to check for potential - security issues, and are allowed to fail ([Auto SAST](stages.md#auto-sast)) **(ULTIMATE)** - - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](stages.md#auto-secret-detection)) **(ULTIMATE)** + security issues, and are allowed to fail ([Auto SAST](stages.md#auto-sast)) + - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](stages.md#auto-secret-detection)) - The `license_scanning` job searches the application's dependencies to determine each of their licenses and is allowed to fail - ([Auto License Compliance](stages.md#auto-license-compliance)) **(ULTIMATE)** + ([Auto License Compliance](stages.md#auto-license-compliance)) - **Review** - Pipelines on the default branch include this stage with a `dast_environment_deploy` job. To learn more, see [Dynamic Application Security Testing (DAST)](../../user/application_security/dast/index.md). @@ -214,7 +214,7 @@ The jobs are separated into stages: Kubernetes ([Auto Deploy](stages.md#auto-deploy)). - **Performance** - Performance tests are run on the deployed application - ([Auto Browser Performance Testing](stages.md#auto-browser-performance-testing)). **(PREMIUM)** + ([Auto Browser Performance Testing](stages.md#auto-browser-performance-testing)). - **Cleanup** - Pipelines on the default branch include this stage with a `stop_dast_environment` job. @@ -323,7 +323,7 @@ and customized to fit your workflow. Here are some helpful resources for further 1. [Auto DevOps](index.md) 1. [Multiple Kubernetes clusters](multiple_clusters_auto_devops.md) -1. [Incremental rollout to production](customize.md#incremental-rollout-to-production) **(PREMIUM)** +1. [Incremental rollout to production](customize.md#incremental-rollout-to-production) 1. [Disable jobs you don't need with CI/CD variables](customize.md#cicd-variables) 1. [Use your own buildpacks to build your application](customize.md#custom-buildpacks) 1. [Prometheus monitoring](../../user/project/integrations/prometheus.md) diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 8dd7c0317d9..816cbbece4f 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -97,9 +97,8 @@ To make full use of Auto DevOps with Kubernetes, you need: To enable deployments, you need: - 1. A [Kubernetes 1.12+ cluster](../../user/project/clusters/index.md) for your - project. The easiest way is to create a - [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#create-new-cluster). + 1. A [Kubernetes 1.12+ cluster](../../user/infrastructure/clusters/index.md) for your + project. For Kubernetes 1.16+ clusters, you must perform additional configuration for [Auto Deploy for Kubernetes 1.16+](stages.md#kubernetes-116). 1. For external HTTP traffic, an Ingress controller is required. For regular @@ -161,7 +160,7 @@ To make full use of Auto DevOps with Kubernetes, you need: - **cert-manager** (optional, for TLS/HTTPS) - To enable HTTPS endpoints for your application, you can [install cert-manager](https://cert-manager.io/docs/installation/kubernetes/), + To enable HTTPS endpoints for your application, you can [install cert-manager](https://cert-manager.io/docs/installation/supported-releases/), a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster issues a [Let's Encrypt](https://letsencrypt.org/) certificate and ensures the diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 9e6f3103664..ead2e957684 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -35,7 +35,7 @@ your own `Dockerfile`, you must either: ### Auto Build using Cloud Native Buildpacks -> - Introduced in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165) in GitLab 12.10. > - Auto Build using Cloud Native Buildpacks by default was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63351) in GitLab 14.0. Auto Build builds an application using a project's `Dockerfile` if present. If no @@ -147,7 +147,7 @@ might want to use a [custom buildpack](customize.md#custom-buildpacks). ## Auto Code Quality -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) from GitLab Starter to GitLab Free in 13.2. Auto Code Quality uses the [Code Quality image](https://gitlab.com/gitlab-org/ci-cd/codequality) to run @@ -174,8 +174,8 @@ see the documentation. ## Auto Secret Detection -> - Introduced in GitLab Ultimate 13.1. -> - [Select functionality made available in all tiers](../../user/application_security/secret_detection/#making-secret-detection-available-to-all-gitlab-tiers) in 13.3 +> - Introduced in GitLab 13.1. +> - Select functionality [made available](../../user/application_security/secret_detection/#making-secret-detection-available-to-all-gitlab-tiers) in all tiers in GitLab 13.3 Secret Detection uses the [Secret Detection Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/secrets) to run Secret Detection on the current code, and checks for leaked secrets. Auto Secret Detection requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or above. @@ -202,7 +202,7 @@ see the documentation. ## Auto License Compliance **(ULTIMATE)** -> Introduced in GitLab Ultimate 11.0. +> Introduced in GitLab 11.0. License Compliance uses the [License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) @@ -310,7 +310,7 @@ You can disable DAST: ## Auto Browser Performance Testing **(PREMIUM)** -> Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. +> Introduced in GitLab 10.4. Auto [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md) measures the browser performance of a web page with the @@ -331,7 +331,7 @@ Any browser performance differences between the source and target branches are a ## Auto Load Performance Testing **(PREMIUM)** -> Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> Introduced in GitLab 13.2. Auto [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md) measures the server performance of an application with the @@ -348,7 +348,7 @@ Any load performance test result differences between the source and target branc ## Auto Deploy -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216008) in GitLab 13.6, you have the choice to deploy to [Amazon Elastic Compute Cloud (Amazon EC2)](https://aws.amazon.com/ec2/) in addition to a Kubernetes cluster. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216008) in GitLab 13.6, you have the choice to deploy to [Amazon Elastic Compute Cloud (Amazon EC2)](https://aws.amazon.com/ec2/) in addition to a Kubernetes cluster. Auto Deploy is an optional step for Auto DevOps. If the [requirements](requirements.md) are not met, the job is skipped. diff --git a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md index 7ddcdcbacb5..8c460247734 100644 --- a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md +++ b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Upgrading deployments for newer Auto Deploy dependencies +# Upgrading deployments for newer Auto Deploy dependencies **(FREE)** [Auto Deploy](stages.md#auto-deploy) is a feature that deploys your application to a Kubernetes cluster. It consists of several dependencies: diff --git a/doc/topics/build_your_application.md b/doc/topics/build_your_application.md index 150e0ec26e4..9750af5ba1c 100644 --- a/doc/topics/build_your_application.md +++ b/doc/topics/build_your_application.md @@ -13,4 +13,4 @@ code, and use CI/CD to generate your application. Include packages in your app a - [Merge requests](../user/project/merge_requests/index.md) - [CI/CD](../ci/index.md) - [Packages & Registries](../user/packages/index.md) -- [Application infrastructure](../user/project/clusters/index.md) +- [Application infrastructure](../user/infrastructure/index.md) diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index cc2631c9445..ae1667376b0 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -126,7 +126,14 @@ MaxStartups 100:30:200 `100:30:200` means up to 100 SSH sessions are allowed without restriction, after which 30% of connections are dropped until reaching an absolute maximum of 200. -Once configured, restart the SSH daemon for the change to take effect. +After you modify the value of `MaxStartups`, check for any errors in the configuration. + +```shell +sudo sshd -t -f /etc/ssh/sshd_config +``` + +If the configuration check runs without errors, it should be safe to restart the +SSH daemon for the change to take effect. ```shell # Debian/Ubuntu @@ -200,7 +207,7 @@ apply more than one: ```shell omnibus_gitconfig['system'] = { # Set the http.postBuffer size, in bytes - "http" => ["postBuffer" => 524288000] + "http" => ["postBuffer => 524288000"] } ``` diff --git a/doc/topics/plan_and_track.md b/doc/topics/plan_and_track.md index 662898e88fc..86c5287b331 100644 --- a/doc/topics/plan_and_track.md +++ b/doc/topics/plan_and_track.md @@ -1,6 +1,6 @@ --- -stage: -group: +stage: Plan +group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -11,18 +11,32 @@ with milestones and track your team's time. Learn how to save time with quick actions, see how GitLab renders Markdown text, and learn how to use Git to interact with GitLab. -- [Epics](../user/group/epics/index.md) +## Basic workflow features + +Planning features everyone needs to use day-to-day. + +- [Keyboard shortcuts](../user/shortcuts.md) +- [Markdown](../user/markdown.md) +- [Quick actions](../user/project/quick_actions.md) +- [To-Do lists](../user/todos.md) +- [Using Git](../topics/git/index.md) + +## Team Planning + +Get work done as a team. + +- [Comments and threads](../user/discussions/index.md) - [Issues](../user/project/issues/index.md) -- [Labels](../user/project/labels.md) -- [Discussions](../user/discussions/index.md) - [Iterations](../user/group/iterations/index.md) +- [Labels](../user/project/labels.md) - [Milestones](../user/project/milestones/index.md) - [Requirements](../user/project/requirements/index.md) -- [Roadmaps](../user/group/roadmap/index.md) - [Time tracking](../user/project/time_tracking.md) - [Wikis](../user/project/wiki/index.md) -- [Keyboard shortcuts](../user/shortcuts.md) -- [Quick actions](../user/project/quick_actions.md) -- [Markdown](../user/markdown.md) -- [To-Do lists](../user/todos.md) -- [Using Git](../topics/git/index.md) + +## Portfolio Management + +Align your work across teams. + +- [Epics](../user/group/epics/index.md) +- [Roadmaps](../user/group/roadmap/index.md) diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index d453c5d8336..e2af4f453c0 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -6,6 +6,16 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo # Deprecated feature removal schedule +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +<!-- vale off --> + <!-- This page is automatically generated from the YAML files in `/data/deprecations` by the rake task located at `lib/tasks/gitlab/docs/compile_deprecations.rake`. @@ -16,6 +26,24 @@ To add a deprecation, use the example.yml file in `/data/deprecations/templates` then run `bin/rake gitlab:docs:compile_deprecations`. --> +## 14.4 + +### Rename Task Runner pod to Toolbox + +The Task Runner pod is used to execute periodic housekeeping tasks within the GitLab application and is often confused with the GitLab Runner. Thus, [Task Runner will be renamed to Toolbox](https://gitlab.com/groups/gitlab-org/charts/-/epics/25). + +This will result in the rename of the sub-chart: `gitlab/task-runner` to `gitlab/toolbox`. Resulting pods will be named along the lines of `{{ .Release.Name }}-toolbox`, which will often be `gitlab-toolbox`. They will be locatable with the label `app=toolbox`. + +Announced: 2021-08-22 + +## 14.6 + +### Release CLI be distributed as a generic package + +The [release-cli](https://gitlab.com/gitlab-org/release-cli) will be released as a [generic package](https://gitlab.com/gitlab-org/release-cli/-/packages) starting in GitLab 14.2. We will continue to deploy it as a binary to S3 until GitLab 14.5 and stop distributing it in S3 in GitLab 14.6. + +Announced: 2021-08-22 + ## 15.0 ### Legacy database configuration @@ -30,19 +58,19 @@ Announced: 2021-09-22 ### Audit events for repository push events -Audit events for [repository events](../administration/audit_events.md#repository-push) are now deprecated and will be removed in GitLab 15.0. +Audit events for [repository events](https://docs.gitlab.com/ee/administration/audit_events.html#repository-push) are now deprecated and will be removed in GitLab 15.0. These events have always been disabled by default and had to be manually enabled with a feature flag. Enabling them can cause too many events to be generated which can dramatically slow down GitLab instances. For this reason, they are being removed. -Announced: 2021-09-02 +Announced: 2021-09-22 ### OmniAuth Kerberos gem The `omniauth-kerberos` gem will be removed in our next major release, GitLab 15.0. -This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](../integration/kerberos.md#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. +This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](https://docs.gitlab.com/ee/integration/kerberos.html#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. @@ -50,18 +78,24 @@ Announced: 2021-09-22 ### GitLab Serverless -[GitLab Serverless](../user/project/clusters/serverless/index.md) is a feature set to support Knative-based serverless development with automatic deployments and monitoring. +[GitLab Serverless](https://docs.gitlab.com/ee/user/project/clusters/serverless/) is a feature set to support Knative-based serverless development with automatic deployments and monitoring. We decided to remove the GitLab Serverless features as they never really resonated with our users. Besides, given the continuous development of Kubernetes and Knative, our current implementations do not even work with recent versions. Announced: 2021-09-22 -## 14.4 +## 15.2 -### Rename Task Runner pod to Toolbox +### NFS for Git repository storage deprecated -The Task Runner pod is used to execute periodic housekeeping tasks within the GitLab application and is often confused with the GitLab Runner. Thus, [Task Runner will be renamed to Toolbox](https://gitlab.com/groups/gitlab-org/charts/-/epics/25). +With the general availability of Gitaly Cluster ([introduced in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/)), we have deprecated development (bugfixes, performance improvements, etc) for NFS for Git repository storage in GitLab 14.0. We will continue to provide technical support for NFS for Git repositories throughout 14.x, but we will remove all support for NFS in GitLab 15.0. Please see our official [Statement of Support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs) for further information. -This will result in the rename of the sub-chart: `gitlab/task-runner` to `gitlab/toolbox`. Resulting pods will be named along the lines of `{{ .Release.Name }}-toolbox`, which will often be `gitlab-toolbox`. They will be locatable with the label `app=toolbox`. +Gitaly Cluster offers tremendous benefits for our customers such as: -Announced: 2021-09-22 +- [Variable replication factors](https://docs.gitlab.com/ee/administration/gitaly/index.html#replication-factor). +- [Strong consistency](https://docs.gitlab.com/ee/administration/gitaly/index.html#strong-consistency). +- [Distributed read capabilities](https://docs.gitlab.com/ee/administration/gitaly/index.html#distributed-reads). + +We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on [migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/index.html#migrate-to-gitaly-cluster). + +Announced: 2021-06-22 diff --git a/doc/update/index.md b/doc/update/index.md index fadb55684f8..b719c47ae26 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -79,6 +79,10 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). Certain major/minor releases may require different migrations to be finished before you update to the newer version. +Decrease the time required to complete these migrations by increasing the number of +[Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) +that can process jobs in the `background_migration` queue. + **For GitLab 14.0 and newer** To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): @@ -88,6 +92,16 @@ To check the status of [batched background migrations](../user/admin_area/monito  +The status of batched background migrations can also be queried directly in the database. + +1. Log into a `psql` prompt according to the directions for your instance's installation method +(for example, `sudo gitlab-psql` for Omnibus installations). +1. Run the following query in the `psql` session to see details on incomplete batched background migrations: + + ```sql + select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; + ``` + **For Omnibus installations** You can also run: @@ -136,9 +150,9 @@ pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_cla ## Dealing with running CI/CD pipelines and jobs -If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates will fail. Once GitLab is back online, then the trace updates should self-heal. However, depending on the error, the GitLab Runner will either retry or eventually terminate job handling. +If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. -As for the artifacts, the GitLab Runner will attempt to upload them three times, after which the job will eventually fail. +As for the artifacts, the GitLab Runner attempts to upload them three times, after which the job eventually fails. To address the above two scenario's, it is advised to do the following prior to upgrading: @@ -193,7 +207,7 @@ upgrade paths. | Target version | Your version | Supported upgrade path | Note | | -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| `14.1.2` | `13.9.2` | `13.9.2` -> `13.12.9` -> `14.0.7` -> `14.1.2` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1`. | +| `14.1.6` | `13.9.2` | `13.9.2` -> `13.12.12` -> `14.0.11` -> `14.1.6` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1`. | | `13.12.10` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.10` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.10`. | | `13.2.10` | `11.5.0` | `11.5.0` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.2.10` | Six intermediate versions are required: `11.11`, `12.0`, `12.1`, `12.10`, `13.0` and `13.1`, then `13.2.10`. | | `12.10.14` | `11.3.4` | `11.3.4` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` | Three intermediate versions are required: `11.11`, `12.0` and `12.1`, then `12.10.14`. | @@ -206,7 +220,7 @@ upgrade paths. Upgrading the *major* version requires more attention. Backward-incompatible changes and migrations are reserved for major versions. Follow the directions carefully as we -cannot guarantee that upgrading between major versions will be seamless. +cannot guarantee that upgrading between major versions is seamless. It is required to follow the following upgrade steps to ensure a successful *major* version upgrade: @@ -293,6 +307,11 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 14.4.0 + +Git 2.33.x and later is required. We recommend you use the +[Git version provided by Gitaly](../install/installation.md#git). + ### 14.3.0 Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) @@ -402,7 +421,7 @@ Git 2.31.x and later is required. We recommend you use the ### 13.9.0 We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) -that will prevent upgrades to GitLab 13.9.0, 13.9.1, 13.9.2 and 13.9.3 when following the zero-downtime steps. It is necessary +that prevents upgrades to GitLab 13.9.0, 13.9.1, 13.9.2, and 13.9.3 when following the zero-downtime steps. It is necessary to perform the following additional steps for the zero-downtime upgrade: 1. Before running the final `sudo gitlab-rake db:migrate` command on the deploy node, @@ -423,7 +442,7 @@ to perform the following additional steps for the zero-downtime upgrade: ``` If you have already run the final `sudo gitlab-rake db:migrate` command on the deploy node and have -encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you will +encountered the [column rename issue](https://gitlab.com/gitlab-org/gitlab/-/issues/324160), you see the following error: ```shell diff --git a/doc/update/package/downgrade.md b/doc/update/package/downgrade.md index 9a528f5ee44..96b17a7632b 100644 --- a/doc/update/package/downgrade.md +++ b/doc/update/package/downgrade.md @@ -12,7 +12,7 @@ of a package. WARNING: You must at least have a database backup created under the version you are downgrading to. Ideally, you should have a -[full backup archive](../../raketasks/backup_restore.md#back-up-gitlab) +[full backup archive](../../raketasks/backup_restore.md) on hand. The example below demonstrates the downgrade procedure when downgrading between minor diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 44be79f22fb..776a7111188 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -4,103 +4,99 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Upgrade GitLab using the GitLab Package **(FREE SELF)** +# Upgrade GitLab by using the GitLab package **(FREE SELF)** -This section describes how to upgrade GitLab to a new version using the +You can upgrade GitLab to a new version by using the GitLab package. -We recommend performing upgrades between major and minor releases no more than once per -week, to allow time for background migrations to finish. Decrease the time required to -complete these migrations by increasing the number of -[Sidekiq workers](../../administration/operations/extra_sidekiq_processes.md) -that can process jobs in the `background_migration` queue. - -If you don't follow the steps in [zero downtime upgrades](../zero_downtime.md), -your GitLab application will not be available to users while an upgrade is in progress. -They either see a "Deploy in progress" message or a "502" error in their web browser. - -Prerequisites: - -- [Supported upgrade paths](../index.md#upgrade-paths) - has suggestions on when to upgrade. Upgrade paths are enforced for version upgrades by - default. This restricts performing direct upgrades that skip major versions (for - example 10.3 to 12.7 in one jump) that **can break GitLab - installations** due to multiple reasons like deprecated or removed configuration - settings, upgrade of internal tools and libraries, and so on. -- If you are upgrading from a non-Package installation to a GitLab Package installation, see - [Upgrading from a non-Package installation to a GitLab Package installation](https://docs.gitlab.com/omnibus/convert_to_omnibus.html). -- It's important to ensure that any +## Prerequisites + +- Decide when to upgrade by viewing the [supported upgrade paths](../index.md#upgrade-paths). + You can't directly skip major versions (for example, go from 10.3 to 12.7 in one step). +- If you are upgrading from a non-package installation to a GitLab package installation, see + [Upgrading from a non-package installation to a GitLab package installation](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). +- Ensure that any [background migrations](../index.md#checking-for-background-migrations-before-upgrading) - have been fully completed before upgrading to a new major version. Upgrading - before background migrations have finished may lead to data corruption. + are fully completed. Upgrading + before background migrations have finished can lead to data corruption. + We recommend performing upgrades between major and minor releases no more than once per + week, to allow time for background migrations to finish. - Gitaly servers must be upgraded to the newer version prior to upgrading the application server. This prevents the gRPC client on the application server from sending RPCs that the old Gitaly version does not support. -You can upgrade the GitLab Package using one of the following methods: +## Downtime + +- For single node installations, GitLab is not available to users while an + upgrade is in progress. The user's web browser shows a `Deploy in progress` message or a `502` error. +- For multi-node installations, see how to perform + [zero downtime upgrades](../zero_downtime.md). + +## Version-specific changes + +Upgrading versions might need some manual intervention. For more information, +check the version your are upgrading to: + +- [GitLab 14](https://docs.gitlab.com/omnibus/update/gitlab_14_changes.html) +- [GitLab 13](https://docs.gitlab.com/omnibus/update/gitlab_13_changes.html) +- [GitLab 12](https://docs.gitlab.com/omnibus/update/gitlab_12_changes.html) +- [GitLab 11](https://docs.gitlab.com/omnibus/update/gitlab_11_changes.html) -- [Using the official repositories](#upgrade-using-the-official-repositories). -- [Using a manually-downloaded package](#upgrade-using-a-manually-downloaded-package). +## Back up before upgrading -Both automatically back up the GitLab database before installing a newer -GitLab version. You may skip this automatic database backup by creating an empty file +The GitLab database is backed up before installing a newer GitLab version. You +may skip this automatic database backup by creating an empty file at `/etc/gitlab/skip-auto-backup`: ```shell sudo touch /etc/gitlab/skip-auto-backup ``` -For safety reasons, you should maintain an up-to-date backup on your own if you plan to use this flag. - -## Version-specific changes - -Updating to major versions might need some manual intervention. For more information, -check the version your are upgrading to: - -- [GitLab 14](https://docs.gitlab.com/omnibus/gitlab_14_changes.html) -- [GitLab 13](https://docs.gitlab.com/omnibus/gitlab_13_changes.html) -- [GitLab 12](https://docs.gitlab.com/omnibus/gitlab_12_changes.html) -- [GitLab 11](https://docs.gitlab.com/omnibus/gitlab_11_changes.html) +Nevertheless, it is highly recommended to maintain a full up-to-date +[backup](../../raketasks/backup_restore.md) on your own. ## Upgrade using the official repositories All GitLab packages are posted to the GitLab [package server](https://packages.gitlab.com/gitlab/). Five repositories are maintained: -- [GitLab EE](https://packages.gitlab.com/gitlab/gitlab-ee): for official - [Enterprise Edition](https://about.gitlab.com/pricing/) releases. -- [GitLab CE](https://packages.gitlab.com/gitlab/gitlab-ce): for official Community Edition releases. -- [Unstable](https://packages.gitlab.com/gitlab/unstable): for release candidates and other unstable versions. -- [Nighty Builds](https://packages.gitlab.com/gitlab/nightly-builds): for nightly builds. -- [Raspberry Pi](https://packages.gitlab.com/gitlab/raspberry-pi2): for official Community Edition releases built for [Raspberry Pi](https://www.raspberrypi.org) packages. +- [`gitlab/gitlab-ee`](https://packages.gitlab.com/gitlab/gitlab-ee): The full + GitLab package that contains all the Community Edition features plus the + [Enterprise Edition](https://about.gitlab.com/pricing/) ones. +- [`gitlab/gitlab-ce`](https://packages.gitlab.com/gitlab/gitlab-ce): A stripped + down package that contains only the Community Edition features. +- [`gitlab/unstable`](https://packages.gitlab.com/gitlab/unstable): Release candidates and other unstable versions. +- [`gitlab/nightly-builds`](https://packages.gitlab.com/gitlab/nightly-builds): Nightly builds. +- [`gitlab/raspberry-pi2`](https://packages.gitlab.com/gitlab/raspberry-pi2): Official Community Edition releases built for [Raspberry Pi](https://www.raspberrypi.org) packages. -If you have installed Omnibus GitLab [Community Edition](https://about.gitlab.com/install/?version=ce) +If you have installed GitLab [Community Edition](https://about.gitlab.com/install/?version=ce) or [Enterprise Edition](https://about.gitlab.com/install/), then the official GitLab repository should have already been set up for you. -To upgrade to the newest GitLab version, run: +### Upgrade to the latest version using the official repositories -- For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/): +If you upgrade GitLab regularly, for example once a month, you can upgrade to +the latest version by using your package manager. - ```shell - # Debian/Ubuntu - sudo apt-get update - sudo apt-get install gitlab-ee +To upgrade to the latest GitLab version: - # Centos/RHEL - sudo yum install gitlab-ee - ``` +```shell +# Ubuntu/Debian +sudo apt update && sudo apt install gitlab-ee + +# RHEL/CentOS 6 and 7 +sudo yum install gitlab-ee -- For GitLab Community Edition: +# RHEL/CentOS 8 +sudo dnf install gitlab-ee - ```shell - # Debian/Ubuntu - sudo apt-get update - sudo apt-get install gitlab-ce +# SUSE +sudo zypper install gitlab-ee +``` - # Centos/RHEL - sudo yum install gitlab-ce - ``` +NOTE: +For the GitLab Community Edition, replace `gitlab-ee` with +`gitlab-ce`. ### Upgrade to a specific version using the official repositories @@ -113,30 +109,43 @@ versions, so you must specify the specific GitLab package with each upgrade. To specify the intended GitLab version number in your package manager's install or upgrade command: -1. First, identify the GitLab version number in your package manager: +1. Identify the version number of the installed package: ```shell # Ubuntu/Debian sudo apt-cache madison gitlab-ee + # RHEL/CentOS 6 and 7 yum --showduplicates list gitlab-ee + # RHEL/CentOS 8 - dnf search gitlab-ee* + dnf --showduplicates list gitlab-ee + + # SUSE + zypper search -s gitlab-ee ``` -1. Then install the specific GitLab package: +1. Install the specific `gitlab-ee` package by using one of the following commands + and replacing `<version>` with the version you found in the previous step: ```shell # Ubuntu/Debian - sudo apt install gitlab-ee=12.0.12-ee.0 + sudo apt install gitlab-ee=<version> + # RHEL/CentOS 6 and 7 - yum install gitlab-ee-12.0.12-ee.0.el7 + yum install gitlab-ee-<version> + # RHEL/CentOS 8 - dnf install gitlab-ee-12.0.12-ee.0.el8 + dnf install gitlab-ee-<version> + # SUSE - zypper install gitlab-ee=12.0.12-ee.0 + zypper install gitlab-ee=<version> ``` +NOTE: +For the GitLab Community Edition, replace `gitlab-ee` with +`gitlab-ce`. + ## Upgrade using a manually-downloaded package NOTE: @@ -150,34 +159,30 @@ install GitLab for the first time or update it. To download and install GitLab: 1. Visit the [official repository](#upgrade-using-the-official-repositories) of your package. -1. Browse to the repository for the type of package you would like to see the - list of packages that are available. Multiple packages exist for a - single version, one for each supported distribution type. Next to the filename - is a label indicating the distribution, as the file names may be the same. -1. Find the package version you wish to install and click on it. -1. Click the **Download** button in the upper right corner to download the package. -1. After the GitLab package is downloaded, install it using the following commands: - - - For GitLab [Enterprise Edition](https://about.gitlab.com/pricing/): +1. Filter the list by searching for the version you want to install (for example 14.1.6). + Multiple packages may exist for a single version, one for each supported distribution + and architecture. Next to the filename is a label indicating the distribution, + as the filenames may be the same. +1. Find the package version you wish to install, and select the filename from the list. +1. Select **Download** in the upper right corner to download the package. +1. After the package is downloaded, install it by using one of the + following commands and replacing `<package_name>` with the package name + you downloaded: - ```shell - # Debian/Ubuntu - dpkg -i gitlab-ee-<version>.deb - - # CentOS/RHEL - rpm -Uvh gitlab-ee-<version>.rpm - ``` + ```shell + # Debian/Ubuntu + dpkg -i <package_name> - - For GitLab Community Edition: + # CentOS/RHEL + rpm -Uvh <package_name> - ```shell - # GitLab Community Edition - # Debian/Ubuntu - dpkg -i gitlab-ce-<version>.deb + # SUSE + zypper install <package_name> + ``` - # CentOS/RHEL - rpm -Uvh gitlab-ce-<version>.rpm - ``` +NOTE: +For the GitLab Community Edition, replace `gitlab-ee` with +`gitlab-ce`. ## Troubleshooting @@ -237,8 +242,8 @@ To avoid this issue, either: ### 500 error when accessing Project > Settings > Repository -When GitLab is migrated from CE > EE > CE, and then back to EE, you -might get the following error when viewing a project's repository settings: +This error occurs when GitLab is converted from CE > EE > CE, and then back to EE. +When viewing a project's repository settings, you can view this error in the logs: ```shell Processing by Projects::Settings::RepositoryController#show as HTML diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index 406f8322218..320f900dd21 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a GitLab upgrade plan +# Create a GitLab upgrade plan **(FREE SELF)** This document serves as a guide to create a strong plan to upgrade a self-managed GitLab instance. @@ -75,7 +75,7 @@ Create a backup of GitLab and all its data (database, repos, uploads, builds, artifacts, LFS objects, registry, pages). This is vital for making it possible to roll back GitLab to a working state if there's a problem with the upgrade: -- Create a [GitLab backup](../raketasks/backup_restore.md#back-up-gitlab). +- Create a [GitLab backup](../raketasks/backup_restore.md). Make sure to follow the instructions based on your installation method. Don't forget to back up the [secrets and configuration files](../raketasks/backup_restore.md#storing-configuration-files). - Alternatively, create a snapshot of your instance. If this is a multi-node @@ -173,7 +173,7 @@ If anything doesn't go as planned: - [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) if you installed GitLab using the Helm Charts. - For support: - - [Contact GitLab Support](https://support.gitlab.com) and, + - [Contact GitLab Support](https://support.gitlab.com/hc) and, if you have one, your Technical Account Manager. - If [the situation qualifies](https://about.gitlab.com/support/#definitions-of-support-impact) and [your plan includes emergency support](https://about.gitlab.com/support/#priority-support), diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md index b8cceb8c367..cedb46a7c71 100644 --- a/doc/update/restore_after_failure.md +++ b/doc/update/restore_after_failure.md @@ -31,9 +31,7 @@ attempts. This is because some tables may have been added during the failed upgrade. If these tables are still present after you restore from the older backup it can lead to migration failures on future upgrades. -Starting in GitLab 8.6 we drop all tables prior to importing the backup to -prevent this problem. If you've restored a backup to a version prior to 8.6 you -may have to manually correct the problem next time you upgrade. +We drop all tables prior to importing the backup to prevent this problem. Example error: @@ -52,38 +50,16 @@ Copy the version from the error. In this case the version number is WARNING: Use the following steps only if you are certain you must do them. -### GitLab 8.6+ +1. Pass the version to a database Rake task to manually mark the migration as + complete. -Pass the version to a database Rake task to manually mark the migration as -complete. + ```shell + # Source install + sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production -```shell -# Source install -sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production + # Omnibus install + sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857] + ``` -# Omnibus install -sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857] -``` - -After the migration is successfully marked, run the Rake `db:migrate` task again. -Repeat this process until all failed migrations are complete. - -### GitLab < 8.6 - -```shell -# Source install -sudo -u git -H bundle exec rails console -e production - -# Omnibus install -sudo gitlab-rails console -``` - -At the Rails console, type the following commands: - -```ruby -ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')") -exit -``` - -After the migration is successfully marked, run the Rake `db:migrate` task again. -Repeat this process until all failed migrations are complete. +1. After the migration is successfully marked, run the Rake `db:migrate` task again. +1. Repeat this process until all failed migrations are complete. diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 9abf993f0fe..67559db2cf0 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -258,6 +258,10 @@ sudo systemctl daemon-reload ### 10. Install libraries, migrations, etc +Make sure you have the required +[PostgreSQL extensions](../install/requirements.md#postgresql-requirements), +then proceed to install the needed libraries: + ```shell cd /home/git/gitlab @@ -304,15 +308,18 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake "gitlab:workhorse:install[/home/git/gitlab-workhorse]" RAILS_ENV=production ``` +NOTE: +If you get any errors concerning Rack attack, see the [13.0](#1301) specific +upgrade instructions. + ### 13. Update Gitaly #### Compile Gitaly ```shell -cd /home/git/gitaly -sudo -u git -H git fetch --all --tags --prune -sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) -sudo -u git -H make +# Fetch Gitaly source with Git and compile with Go +cd /home/git/gitlab +sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly,/home/git/repositories]" RAILS_ENV=production ``` ### 14. Update GitLab Pages @@ -375,14 +382,13 @@ Additional instructions here. ### 13.0.1 -As part of [deprecating Rack Attack throttles on Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4750), Rack Attack initializer on GitLab +As part of [deprecating Rack Attack throttles on Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4750), the Rack Attack initializer on GitLab was renamed from [`config/initializers/rack_attack_new.rb` to `config/initializers/rack_attack.rb`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33072). If this file exists on your installation, consider creating a backup before updating: ```shell cd /home/git/gitlab - -cp config/initializers/rack_attack.rb config/initializers/rack_attack_backup.rb +cp config/initializers/rack_attack.rb ~/config/initializers/rack_attack_backup.rb ``` ## Troubleshooting diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index f0e6377f355..7a74435267f 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Zero downtime upgrades +# Zero downtime upgrades **(FREE SELF)** Starting with GitLab 9.1.0 it's possible to upgrade to a newer major, minor, or patch version of GitLab without having to take your GitLab instance offline. @@ -428,7 +428,7 @@ sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert setting `gitlab_rails['auto_migrate'] = false` in `/etc/gitlab/gitlab.rb` after you've completed these steps. -### Use Redis HA (using Sentinel) **(PREMIUM ONLY)** +### Use Redis HA (using Sentinel) **(PREMIUM SELF)** Package upgrades may involve version updates to the bundled Redis service. On instances using [Redis for scaling](../administration/redis/index.md), @@ -525,7 +525,7 @@ failover is complete, we can go ahead and upgrade the original primary node. Install the package for new version and follow regular package upgrade procedure. -## Geo deployment **(PREMIUM ONLY)** +## Geo deployment **(PREMIUM SELF)** The order of steps is important. While following these steps, make sure you follow them in the right order, on the correct node. @@ -645,7 +645,7 @@ sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert setting `gitlab_rails['auto_migrate'] = false` in `/etc/gitlab/gitlab.rb` after you've completed these steps. -## Multi-node / HA deployment with Geo **(PREMIUM ONLY)** +## Multi-node / HA deployment with Geo **(PREMIUM SELF)** This section describes the steps required to upgrade a multi-node / HA deployment with Geo. Some steps must be performed on a particular node. This diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index 7ddddfc5e53..b5294bb265d 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # DevOps Report **(FREE SELF)** -> [Renamed from Conversational Development Index](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) in GitLab 12.6. +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. The DevOps Report gives you an overview of your entire instance's adoption of [Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) diff --git a/doc/user/admin_area/appearance.md b/doc/user/admin_area/appearance.md index 0ffa8289d37..0b264ef22b9 100644 --- a/doc/user/admin_area/appearance.md +++ b/doc/user/admin_area/appearance.md @@ -39,7 +39,7 @@ of the page to activate it in the GitLab instance. ## System header and footer messages -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55057) to GitLab Free in 11.9. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55057) from GitLab Premium to GitLab Free in 11.9. You can add a small header message, a small footer message, or both, to the interface of your GitLab instance. These messages appear on all projects and pages of the diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index 8c0586e5851..976d8e4efcf 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -7,7 +7,7 @@ type: reference # Custom instance-level project templates **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6860) in GitLab 11.2. GitLab administrators can set a group to be the source of project templates that are selectable when a new project is created on the instance. These templates can be selected diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md index a2354e68d72..f2b899f0be9 100644 --- a/doc/user/admin_area/geo_nodes.md +++ b/doc/user/admin_area/geo_nodes.md @@ -22,45 +22,45 @@ All Geo sites have the following settings: | Setting | Description | | --------| ----------- | | Primary | This marks a Geo site as **primary** site. There can be only one **primary** site. | -| Name | The unique identifier for the Geo site. It's highly recommended to use a physical location as a name. Good examples are "London Office" or "us-east-1". Avoid words like "primary", "secondary", "Geo", or "DR". This makes the failover process easier because the physical location does not change, but the Geo site role can. All nodes in a single Geo site use the same site name. Nodes use the `gitlab_rails['geo_node_name']` setting in `/etc/gitlab/gitlab.rb` to lookup their Geo site record in the PostgreSQL database. If `gitlab_rails['geo_node_name']` is not set, then the node's `external_url` with trailing slash is used as fallback. The value of `Name` is case-sensitive, and most characters are allowed. | +| Name | The unique identifier for the Geo site. It's highly recommended to use a physical location as a name. Good examples are "London Office" or "us-east-1". Avoid words like "primary", "secondary", "Geo", or "DR". This makes the failover process easier because the physical location does not change, but the Geo site role can. All nodes in a single Geo site use the same site name. Nodes use the `gitlab_rails['geo_node_name']` setting in `/etc/gitlab/gitlab.rb` to lookup their Geo site record in the PostgreSQL database. If `gitlab_rails['geo_node_name']` is not set, the node's `external_url` with trailing slash is used as fallback. The value of `Name` is case-sensitive, and most characters are allowed. | | URL | The instance's user-facing URL. | The site you're currently browsing is indicated with a blue `Current` label, and the **primary** node is listed first as `Primary site`. -## **Secondary** site settings +## Secondary site settings **Secondary** sites have a number of additional settings available: | Setting | Description | |---------------------------|-------------| | Selective synchronization | Enable Geo [selective sync](../../administration/geo/replication/configuration.md#selective-synchronization) for this **secondary** site. | -| Repository sync capacity | Number of concurrent requests this **secondary** site will make to the **primary** site when backfilling repositories. | -| File sync capacity | Number of concurrent requests this **secondary** site will make to the **primary** site when backfilling files. | +| Repository sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling repositories. | +| File sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling files. | ## Geo backfill **Secondary** sites are notified of changes to repositories and files by the **primary** site, -and will always attempt to synchronize those changes as quickly as possible. +and always attempt to synchronize those changes as quickly as possible. Backfill is the act of populating the **secondary** site with repositories and files that -existed *before* the **secondary** site was added to the database. Since there may be -extremely large numbers of repositories and files, it's infeasible to attempt to -download them all at once, so GitLab places an upper limit on the concurrency of +existed *before* the **secondary** site was added to the database. Because there may be +extremely large numbers of repositories and files, it's not feasible to attempt to +download them all at once; so, GitLab places an upper limit on the concurrency of these operations. -How long the backfill takes is a function of the maximum concurrency, but higher +How long the backfill takes is dependent on the maximum concurrency, but higher values place more strain on the **primary** site. From [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3107), the limits are configurable. If your **primary** site has lots of surplus capacity, you can increase the values to complete backfill in a shorter time. If it's -under heavy load and backfill is reducing its availability for normal requests, +under heavy load and backfill reduces its availability for normal requests, you can decrease them. ## Using a different URL for synchronization The **primary** site's Internal URL is used by **secondary** sites to contact it (to sync repositories, for example). The name Internal URL distinguishes it from -[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab) +[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab), which is used by users. Internal URL does not need to be a private address. Internal URL defaults to external URL, but you can also customize it: @@ -79,21 +79,21 @@ terminated at the load balancer. WARNING: Starting with GitLab 13.3 and [until 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/325522), -using an internal URL that is not accessible to the users will result in the -OAuth authorization flow not working properly, as the users will get redirected +if you use an internal URL that is not accessible to the users, the +OAuth authorization flow does not work properly, because users are redirected to the internal URL instead of the external one. ## Multiple secondary sites behind a load balancer -In GitLab 11.11, **secondary** sites can use identical external URLs as long as +In GitLab 11.11, **secondary** sites can use identical external URLs if a unique `name` is set for each Geo site. The `gitlab.rb` setting `gitlab_rails['geo_node_name']` must: - Be set for each GitLab instance that runs `puma`, `sidekiq`, or `geo_logcursor`. - Match a Geo site name. -The load balancer must use sticky sessions in order to avoid authentication -failures and cross site request errors. +The load balancer must use sticky sessions to avoid authentication +failures and cross-site request errors. <!-- ## Troubleshooting diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index a5c3a2a7aeb..27d2bd8f764 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -8,7 +8,7 @@ type: reference # GitLab Admin Area **(FREE SELF)** The Admin Area provides a web UI to manage and configure some features of GitLab -self-managed instances. If you are an Admin user, you can access the Admin Area +self-managed instances. If you are an administrator, you can access the Admin Area by visiting `/admin` on your self-managed instance. You can also access it through the UI: @@ -16,7 +16,7 @@ the UI: - GitLab versions 13.12 and earlier: on the top bar, select the Admin Area icon (**{admin}**). NOTE: -Only admin users can access the Admin Area. +Only administrators can access the Admin Area. ## Admin Area sections @@ -24,7 +24,7 @@ The Admin Area is made up of the following sections: | Section | Description | |:-----------------------------------------------|:------------| -| **{overview}** [Overview](#overview-section) | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administering-projects), [users](#administering-users), [groups](#administering-groups), [jobs](#administering-jobs), [runners](#administering-runners), and [Gitaly servers](#administering-gitaly-servers). | +| **{overview}** [Overview](#overview-section) | View your GitLab [Dashboard](#admin-area-dashboard), and administer [projects](#administering-projects), [users](#administering-users), [groups](#administering-groups), [topics](#administering-topics), [jobs](#administering-jobs), [runners](#administering-runners), and [Gitaly servers](#administering-gitaly-servers). | | **{monitor}** Monitoring | View GitLab [system information](#system-information), and information on [background jobs](#background-jobs), [logs](#logs), [health checks](monitoring/health_check.md), [requests profiles](#requests-profiles), and [audit events](#audit-events). | | **{messages}** Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | | **{hook}** System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | @@ -41,7 +41,7 @@ The Admin Area is made up of the following sections: | **{appearance}** Appearance | Customize [GitLab appearance](appearance.md). | | **{settings}** Settings | Modify the [settings](settings/index.md) for your GitLab instance. | -## Admin Dashboard +## Admin Area dashboard The Dashboard provides statistics and system information about the GitLab instance. @@ -151,7 +151,7 @@ you must provide the complete email address. #### User impersonation -An administrator can "impersonate" any other user, including other administrator users. +An administrator can "impersonate" any other user, including other administrators. This allows the administrator to "see what the user sees," and take actions on behalf of the user. You can impersonate a user in the following ways: @@ -237,6 +237,26 @@ insensitive, and applies partial matching. To [Create a new group](../group/index.md#create-a-group) click **New group**. +### Administering Topics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340920) in GitLab 14.4. + +You can administer all topics in the GitLab instance from the Admin Area's Topics page. + +To access the Topics page: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Overview > Topics**. + +For each topic, the page displays their name and number of projects labeled with the topic. + +To create a new topic, select **New topic**. + +To edit a topic, select **Edit** in that topic's row. + +To search for topics by name, enter your criteria in the search box. The topic search is case +insensitive, and applies partial matching. + ### Administering Jobs You can administer all jobs in the GitLab instance from the Admin Area's Jobs page. @@ -369,7 +389,7 @@ The Sidekiq dashboard consists of the following elements: ### Logs -Since GitLab 13.0, **Log** view has been removed from the admin dashboard since the logging does not work in multi-node setups and could cause confusion for administrators by displaying partial information. +Since GitLab 13.0, **Log** view has been removed from the Admin Area dashboard since the logging does not work in multi-node setups and could cause confusion for administrators by displaying partial information. For multi-node systems we recommend ingesting the logs into services like Elasticsearch and Splunk. diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 4e97cb8e49c..d984f6f4092 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Activating GitLab EE +# Activating GitLab EE **(PREMIUM SELF)** To enable features of GitLab Enterprise Edition (EE), you need to activate your instance. Ensure you are running an enterprise edition. To verify, sign in to GitLab and browse to `/help`. The GitLab edition and version are listed at the top of the **Help** page. @@ -15,7 +15,7 @@ As of GitLab Enterprise Edition 9.4.0, a newly-installed instance without an uploaded license only has the Free features active. A trial license activates all Ultimate features, but after [the trial expires](#what-happens-when-your-license-expires), some functionality is locked. -## Activate GitLab EE with an Activation Code **(PREMIUM SELF)** +## Activate GitLab EE with an Activation Code As of GitLab Enterprise Edition 14.1, you need an activation code to activate your instance. You can obtain an activation code by [purchasing a license](https://about.gitlab.com/pricing/) or by signing up for a [free trial](https://about.gitlab.com/free-trial/). This activation code is a 24-character alphanumeric string you receive in a confirmation email. You can also sign in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) to copy the activation code to your clipboard. @@ -28,9 +28,9 @@ To begin the activation process with your activation code: 1. Read and accept the terms of service. 1. Select **Activate**. -## Activate GitLab EE with a License File **(PREMIUM SELF)** +## Activate GitLab EE with a License File -If you receive a license file from GitLab (for example a new trial), you can upload it by signing into your GitLab instance as an admin or adding it during installation. The license is a base64-encoded ASCII text file with a `.gitlab-license` extension. +If you receive a license file from GitLab (for example a new trial), you can upload it by signing into your GitLab instance as an administrator or adding it during installation. The license is a base64-encoded ASCII text file with a `.gitlab-license` extension. ## Uploading your license diff --git a/doc/user/admin_area/merge_requests_approvals.md b/doc/user/admin_area/merge_requests_approvals.md index ffa08dee10d..18151fc17d5 100644 --- a/doc/user/admin_area/merge_requests_approvals.md +++ b/doc/user/admin_area/merge_requests_approvals.md @@ -32,4 +32,7 @@ any commits to the source branch. - **Prevent editing approval rules in projects and merge requests**. Prevents users from modifying the approvers list in project settings or in individual merge requests. -Also read the [project level merge request approval rules](../project/merge_requests/approvals/index.md), which are affected by instance level rules. +See also the following, which are affected by instance-level rules: + +- [Project-level merge request approval rules](../project/merge_requests/approvals/index.md). +- [Group-level merge request approval rules](../group/index.md#group-approval-rules) available in GitLab 13.9 and later. diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index 2655d927b87..c8f160f729f 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Moderate users +# Moderate users **(FREE SELF)** GitLab administrators can moderate user access by approving, blocking, banning, or deactivating users. @@ -73,7 +73,7 @@ In order to completely prevent access of a user to the GitLab instance, administrators can choose to block the user. Users can be blocked [via an abuse report](review_abuse_reports.md#blocking-users), -or directly from the Admin Area. To do this: +by removing them in LDAP, or directly from the Admin Area. To do this: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users**. diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index 5765a530baf..da245300e1d 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. > - Enabled on GitLab.com. > - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-batched-background-migrations). **(FREE SELF)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-batched-background-migrations). There can be [risks when disabling released features](../../../administration/feature_flags.md#risks-when-disabling-released-features). Refer to this feature's version history for more details. @@ -21,12 +21,12 @@ are created by GitLab developers and run automatically on upgrade. However, such limited in scope to help with migrating some `integer` database columns to `bigint`. This is needed to prevent integer overflow for some tables. -## Check the status of background migrations **(FREE SELF)** +## Check the status of background migrations All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md). You can [check the status of existing migrations](../../../update/index.md#checking-for-background-migrations-before-upgrading). -## Enable or disable batched background migrations **(FREE SELF)** +## Enable or disable batched background migrations Batched background migrations are under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -45,21 +45,21 @@ To disable it: Feature.disable(:execute_batched_migrations_on_schedule) ``` -## Automatic batch size optimization **(FREE SELF)** +## Automatic batch size optimization > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60133) in GitLab 13.12. > - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/329511) in GitLab 13.12. > - Enabled on GitLab.com. > - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-automatic-batch-size-optimization). **(FREE SELF)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-automatic-batch-size-optimization). There can be [risks when disabling released features](../../../administration/feature_flags.md#risks-when-disabling-released-features). Refer to this feature's version history for more details. To maximize throughput of batched background migrations (in terms of the number of tuples updated per time unit), batch sizes are automatically adjusted based on how long the previous batches took to complete. -## Enable or disable automatic batch size optimization **(FREE SELF)** +## Enable or disable automatic batch size optimization Automatic batch size optimization for batched background migrations is under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. @@ -91,13 +91,95 @@ Expected batched background migration for the given configuration to be marked a {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} ``` -To fix this error: +First, check if you have followed the [version-specific upgrade instructions for 14.2](../../../update/index.md#1420). +If you have, you can [manually finish the batched background migration](#manually-finishing-a-batched-background-migration). +If you haven't, choose one of the following methods: -1. Update to either 14.0.5 or 14.1. -1. [Check the status](#check-the-status-of-background-migrations) of the batched background migration from the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, or finalize it manually using the command suggested in the error, for example: +1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required +versions before updating to 14.2+. +1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current +version and manually ensuring that the batched migrations complete successfully. + +#### Roll back and follow the required upgrade path + +1. [Rollback and restore the previously installed version](../../../update/restore_after_failure.md#roll-back-to-an-earlier-version-and-restore-a-backup) +1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+ +1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations and +make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active, +you can [manually finish them](#manually-finishing-a-batched-background-migration). + +#### Roll forward and finish the migrations on the upgraded version + +##### For a deployment with downtime + +To run all the batched background migrations, it can take a significant amount of time +depending on the size of your GitLab installation. + +1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations in the +database, and [manually run them](#manually-finishing-a-batched-background-migration) with the appropriate +arguments until the status query returns no rows. +1. When the status of all of all them is marked as complete, re-run migrations for your installation. +1. [Complete the database migrations](../../../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade: + + ```plaintext + sudo gitlab-rake db:migrate + ``` + +1. Run a reconfigure: + + ```plaintext + sudo gitlab-ctl reconfigure + ``` + +1. Finish the upgrade for your installation. + +##### For a no-downtime deployment + +As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded +version and wait for the batched background migrations to finish normally. + +1. [Check the status](#check-the-status-of-background-migrations) of the batched background migration from +the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, +or [manually finish it](#manually-finishing-a-batched-background-migration). +1. Re-run migrations for your installation, so the remaining post-deployment migrations finish. + +### Manually finishing a batched background migration + +If you need to manually finish a batched background migration due to an +error, you can run: + +```shell +sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>'] +``` + +Replace the values in angle brackets with the correct +arguments. For example, if you receive an error similar to this: + +```plaintext +StandardError: An error has occurred, all later migrations canceled: + +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': + {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]} +``` + +Plug the arguments from the error message into the command: ```shell sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]'] ``` -1. You can now update to GitLab 14.2 or higher. +If you need to manually run a batched background migration to continue an upgrade, you can +[check the status](#check-the-status-of-background-migrations) in the database and get the +arguments from the query results. For example, if the query returns this: + +```plaintext + job_class_name | table_name | column_name | job_arguments +---------------------------------------+------------+-------------+------------------------------------ + CopyColumnUsingBackgroundMigrationJob | events | id | [["id"], ["id_convert_to_bigint"]] + ``` + +The results from the query can be plugged into the command: + +```shell +sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]'] +``` diff --git a/doc/user/admin_area/review_abuse_reports.md b/doc/user/admin_area/review_abuse_reports.md index 6494934c34d..6a8b48e7ba7 100644 --- a/doc/user/admin_area/review_abuse_reports.md +++ b/doc/user/admin_area/review_abuse_reports.md @@ -12,14 +12,14 @@ View and resolve abuse reports from GitLab users. GitLab administrators can view and [resolve](#resolving-abuse-reports) abuse reports in the Admin Area. -## Receiving notifications of abuse reports +## Receive notification of abuse reports by email -To receive notifications of new abuse reports by email, follow these steps: +To receive notifications of new abuse reports by email: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Reporting**. 1. Expand the **Abuse reports** section. -1. Provide an email address. +1. Provide an email address and select **Save changes**. The notification email address can also be set and retrieved [using the API](../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls). diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 178b117d06c..1b6da0e2806 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -47,7 +47,7 @@ details:  -## Maximum artifacts size **(FREE SELF)** +## Maximum artifacts size The maximum size of the [job artifacts](../../../administration/job_artifacts.md) can be set at: @@ -216,6 +216,20 @@ of your GitLab instance (`.gitlab-ci.yml` if not set): It is also possible to specify a [custom CI/CD configuration file for a specific project](../../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file). +## Enable or disable the pipeline suggestion banner + +By default, a banner displays in merge requests with no pipeline suggesting a +walkthrough on how to add one. + + + +To enable or disable the banner: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Select or clear the **Enable pipeline suggestion banner** checkbox. +1. Select **Save changes**. + ## Required pipeline configuration **(PREMIUM SELF)** WARNING: @@ -296,7 +310,7 @@ To set the maximum file size: > - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. > - Disabled on GitLab.com. > - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to enable it. **(FREE SELF)** +> - To use in GitLab self-managed instances, ask a GitLab administrator to enable it. GitLab administrators can adjust who is allowed to register runners, by showing and hiding areas of the UI. diff --git a/doc/user/admin_area/settings/deprecated_api_rate_limits.md b/doc/user/admin_area/settings/deprecated_api_rate_limits.md new file mode 100644 index 00000000000..a2edb6da86e --- /dev/null +++ b/doc/user/admin_area/settings/deprecated_api_rate_limits.md @@ -0,0 +1,53 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Deprecated API rate limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68645) in GitLab 14.4. + +Deprecated API endpoints are those which have been replaced with alternative +functionality, but cannot be removed without breaking backward compatibility. +Setting a restrictive rate limit on these endpoints can encourage users to +switch to the alternatives. + +## Deprecated API endpoints + +Not all deprecated API endpoints are included in this rate limit - just those +that might have a performance impact: + +- [`GET /groups/:id`](../../../api/groups.md#details-of-a-group) **without** the `with_projects=0` query parameter. + +## Define Deprecated API rate limits + +Rate limits for deprecated API endpoints are disabled by default. When enabled, they supersede +the general user and IP rate limits for requests to deprecated endpoints. You can keep any general user +and IP rate limits already in place, and increase or decrease the rate limits +for deprecated API endpoints. No other new features are provided by this override. + +Prerequisites: + +- You must have the Administrator role for your instance. + +To override the general user and IP rate limits for requests to deprecated API endpoints: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Deprecated API Rate Limits**. +1. Select the check boxes for the types of rate limits you want to enable: + - **Unauthenticated API request rate limit** + - **Authenticated API request rate limit** +1. _If you enabled unauthenticated API request rate limits:_ + 1. Select the **Maximum unauthenticated API requests per period per IP**. + 1. Select the **Unauthenticated API rate limit period in seconds**. +1. _If you enabled authenticated API request rate limits:_ + 1. Select the **Maximum authenticated API requests per period per user**. + 1. Select the **Authenticated API rate limit period in seconds**. + +## Resources + +- [Rate limits](../../../security/rate_limits.md) +- [User and IP rate limits](user_and_ip_rate_limits.md) diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index c04a9a12912..6bc9e97629c 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -72,7 +72,7 @@ To add additional text to emails: 1. Enter your text in the **Additional text** field. 1. Select **Save changes**. -## User deactivation emails **(FREE SELF)** +## User deactivation emails GitLab sends email notifications to users when their account has been deactivated. diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 985f3c133d5..5f007c83e4b 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -1,13 +1,12 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # External authorization control **(FREE SELF)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27056) to GitLab Free in 11.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27056) from GitLab Premium to GitLab Free in 11.10. In highly controlled environments, it may be necessary for access policy to be controlled by an external service that permits access based on project diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md new file mode 100644 index 00000000000..f91d4b2fb0c --- /dev/null +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -0,0 +1,56 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Files API rate limits **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68561) in GitLab 14.3. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it +available, ask an administrator to [enable the `files_api_throttling` flag](../../../administration/feature_flags.md). +On GitLab.com, this feature is available but can be configured by GitLab.com +administrators only. The feature is not ready for production use. + +The [Repository files API](../../../api/repository_files.md) enables you to +fetch, create, update, and delete files in your repository. To improve the security +and durability of your web application, you can enforce +[rate limits](../../../security/rate_limits.md) on this API. Any rate limits you +create for the Files API override the [general user and IP rate limits](user_and_ip_rate_limits.md). + +## Define Files API rate limits + +Rate limits for the Files API are disabled by default. When enabled, they supersede +the general user and IP rate limits for requests to the +[Repository files API](../../../api/repository_files.md). You can keep any general user +and IP rate limits already in place, and increase or decrease the rate limits +for the Files API. No other new features are provided by this override. + +Prerequisites: + +- You must have the Administrator role for your instance. +- The `files_api_throttling` feature flag must be enabled. + +To override the general user and IP rate limits for requests to the Repository files API: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Files API Rate Limits**. +1. Select the check boxes for the types of rate limits you want to enable: + - **Unauthenticated API request rate limit** + - **Authenticated API request rate limit** +1. _If you enabled unauthenticated API request rate limits:_ + 1. Select the **Max unauthenticated API requests per period per IP**. + 1. Select the **Unauthenticated API rate limit period in seconds**. +1. _If you enabled authenticated API request rate limits:_ + 1. Select the **Max authenticated API requests per period per user**. + 1. Select the **Authenticated API rate limit period in seconds**. + +## Resources + +- [Rate limits](../../../security/rate_limits.md) +- [Repository files API](../../../api/repository_files.md) +- [User and IP rate limits](user_and_ip_rate_limits.md) diff --git a/doc/user/admin_area/settings/floc.md b/doc/user/admin_area/settings/floc.md index 17c390aef0e..6bfc6dfbebd 100644 --- a/doc/user/admin_area/settings/floc.md +++ b/doc/user/admin_area/settings/floc.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Federated Learning of Cohorts (FLoC) **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60933) in GitLab Free 13.12. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60933) in GitLab 13.12. Federated Learning of Cohorts (FLoC) is a new feature of the Chrome browser. It works by categorizing users into different cohorts, so that diff --git a/doc/user/admin_area/settings/gitaly_timeouts.md b/doc/user/admin_area/settings/gitaly_timeouts.md index 1d4f45d1f04..5da43935052 100644 --- a/doc/user/admin_area/settings/gitaly_timeouts.md +++ b/doc/user/admin_area/settings/gitaly_timeouts.md @@ -22,6 +22,6 @@ The following timeouts are available. | Timeout | Default | Description | |:--------|:-----------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the worker timeout that can be configured for [Puma](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | +| Default | 55 seconds | Timeout for most Gitaly calls (not enforced for `git` `fetch` and `push` operations, or Sidekiq jobs). For example, checking if a repository exists on disk. Makes sure that Gitaly calls made within a web request cannot exceed the entire request timeout. It should be shorter than the worker timeout that can be configured for [Puma](../../../install/requirements.md#puma-settings). If a Gitaly call timeout exceeds the worker timeout, the remaining time from the worker timeout is used to avoid having to terminate the worker. | | Fast | 10 seconds | Timeout for fast Gitaly operations used within requests, sometimes multiple times. For example, checking if a repository exists on disk. If fast operations exceed this threshold, there may be a problem with a storage shard. Failing fast can help maintain the stability of the GitLab instance. | | Medium | 30 seconds | Timeout for Gitaly operations that should be fast (possibly within requests) but preferably not used multiple times within a request. For example, loading blobs. Timeout that should be set between Default and Fast. | diff --git a/doc/user/admin_area/settings/help_page.md b/doc/user/admin_area/settings/help_page.md index cf08b9b71db..0bf5dc1d37a 100644 --- a/doc/user/admin_area/settings/help_page.md +++ b/doc/user/admin_area/settings/help_page.md @@ -68,12 +68,9 @@ You can specify a custom URL to which users are directed when they: ## Redirect `/help` pages -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43157) in GitLab 13.5. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to -[enable the `:help_page_documentation_redirect` flag](../../../administration/feature_flags.md). -On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43157) in GitLab 13.5. +> - [Feature flag `help_page_documentation_redirect`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71737) removed in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71737) in GitLab 14.4. The `/help` URL of a GitLab instance displays a basic version of the documentation sourced from the [`doc` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) of GitLab. `/help` links diff --git a/doc/user/admin_area/settings/img/suggest_pipeline_banner.png b/doc/user/admin_area/settings/img/suggest_pipeline_banner.png Binary files differnew file mode 100644 index 00000000000..9f118ccc5ed --- /dev/null +++ b/doc/user/admin_area/settings/img/suggest_pipeline_banner.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index ec5f3ca812f..688734849e5 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -98,6 +98,8 @@ To access the default page for Admin Area settings: | [User and IP rate limits](user_and_ip_rate_limits.md) | Configure limits for web and API requests. | | [Package Registry Rate Limits](package_registry_rate_limits.md) | Configure specific limits for Packages API requests that supersede the user and IP rate limits. | | [Git LFS Rate Limits](git_lfs_rate_limits.md) | Configure specific limits for Git LFS requests that supersede the user and IP rate limits. | +| [Files API Rate Limits](files_api_rate_limits.md) | Configure specific limits for Files API requests that supersede the user and IP rate limits. | +| [Deprecated API Rate Limits](deprecated_api_rate_limits.md) | Configure specific limits for deprecated API requests that supersede the user and IP rate limits. | | [Outbound requests](../../../security/webhooks.md) | Allow requests to the local network from hooks and services. | | [Protected Paths](protected_paths.md) | Configure paths to be protected by Rack Attack. | | [Incident Management](../../../operations/incident_management/index.md) Limits | Limit the number of inbound alerts that can be sent to a project. | diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index 862bf3b1652..044863729db 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -31,6 +31,8 @@ After you add templates, you can use them for the entire instance. They are available in the [Web Editor's dropdown](../../project/repository/web_editor.md#template-dropdowns) and through the [API settings](../../../api/settings.md). +## Supported file types and locations + Templates must be added to a specific subdirectory in the repository, corresponding to the kind of template. The following types of custom templates are supported: diff --git a/doc/user/admin_area/settings/protected_paths.md b/doc/user/admin_area/settings/protected_paths.md index 061e9fa05d3..dc328fe8b7c 100644 --- a/doc/user/admin_area/settings/protected_paths.md +++ b/doc/user/admin_area/settings/protected_paths.md @@ -41,7 +41,7 @@ try again. ## Configure using GitLab UI -> Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31246). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31246) in GitLab 12.4. Throttling of protected paths is enabled by default and can be disabled or customized on **Admin > Network > Protected Paths**, along with these options: diff --git a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md index a2e8a875ebb..50dd24de3fb 100644 --- a/doc/user/admin_area/settings/rate_limit_on_issues_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_issues_creation.md @@ -19,7 +19,7 @@ To can change its value: 1. Select **Save changes**. For example, if you set a limit of 300, requests using the -[Projects::IssuesController#create](https://gitlab.com/gitlab-org/gitlab/raw/master/app/controllers/projects/issues_controller.rb) +[Projects::IssuesController#create](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/projects/issues_controller.rb) action exceeding a rate of 300 per minute are blocked. Access to the endpoint is allowed after one minute. When using [epics](../../group/epics/index.md), epic creation will share this rate limit with issues. diff --git a/doc/user/admin_area/settings/sidekiq_job_limits.md b/doc/user/admin_area/settings/sidekiq_job_limits.md index c6a783beb3f..750665285b4 100644 --- a/doc/user/admin_area/settings/sidekiq_job_limits.md +++ b/doc/user/admin_area/settings/sidekiq_job_limits.md @@ -13,7 +13,6 @@ type: reference Redis. To avoid excessive memory for Redis, we: - Compress job arguments before storing them in Redis. -arguments before storing them in Redis, and rejecting jobs that exceed - Reject jobs that exceed the specified threshold limit after compression. To access Sidekiq job size limits: @@ -30,7 +29,6 @@ To access Sidekiq job size limits: |-------------------------------------------|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Limiting mode | Compress | This mode compresses the jobs at the specified threshold and rejects them if they exceed the specified limit after compression. | | Sidekiq job compression threshold (bytes) | 100 000 (100 KB) | When the size of arguments exceeds this threshold, they are compressed before being stored in Redis. | -| Sidekiq job size limit (bytes) | 0 | The jobs exceeding this size after compression are rejected. This avoids excessive memory usage in Redis leading to instability. Setting it to 0 prevents rejecting jobs. | +| Sidekiq job size limit (bytes) | 0 | The jobs exceeding this size after compression are rejected. This avoids excessive memory usage in Redis leading to instability. Setting it to 0 prevents rejecting jobs. | -After changing these values, [restart -Sidekiq](../../../administration/restart_gitlab.md). +After changing these values, [restart Sidekiq](../../../administration/restart_gitlab.md). diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index 223ffeebd44..8cee63b0ac2 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -24,6 +24,8 @@ You can restrict the password authentication for web interface and Git over HTTP - **Web interface**: When this feature is disabled, the **Standard** sign-in tab is removed and an [external authentication provider](../../../administration/auth/index.md) must be used. - **Git over HTTP(S)**: When this feature is disabled, a [Personal Access Token](../../profile/personal_access_tokens.md) must be used to authenticate. +In the event of an external authentication provider outage, use the [GitLab Rails console](../../../administration/operations/rails_console.md) to [re-enable the standard web sign-in form](../../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#re-enable-standard-web-sign-in-form). This configuration can also be changed over the [Application settings REST API](../../../api/settings.md#change-application-settings) while authenticating with an administrator account's personal access token. + ## Admin Mode > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2158) in GitLab 13.10. @@ -104,7 +106,7 @@ see [Email notification for unknown sign-ins](../../profile/unknown_sign_in_noti All users that are not logged in are redirected to the page represented by the configured **Home page URL** if value is not empty. -All users are redirected to the page represented by the configured **After sign out path** +All users are redirected to the page represented by the configured **After sign-out path** after sign out if value is not empty. In the **Sign-in restrictions** section, scroll to the **Sign-in text** field. You can add a diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index dc09d6a5132..ed80bca470e 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -31,7 +31,7 @@ To disable sign ups: > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4491) in GitLab 13.5. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/267568) in GitLab 13.6. -When this setting is enabled, any user visiting your GitLab domain and signing up for a new account +When this setting is enabled, any user visiting your GitLab domain and signing up for a new account using the registration form must be explicitly [approved](../moderate_users.md#approve-or-reject-a-user-sign-up) by an administrator before they can start using their account. In GitLab 13.6 and later, this setting is enabled by default for new GitLab instances. It is only applicable if sign ups are enabled. @@ -45,6 +45,12 @@ To require administrator approval for new sign ups: In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/273258), if an administrator disables this setting, the users in pending approval state are automatically approved in a background job. +NOTE: +This setting doesn't apply to LDAP or OmniAuth users. To enforce approvals for new users +signing up using OmniAuth or LDAP, set `block_auto_created_users` to `true` in the +[OmniAuth configuration](../../../integration/omniauth.md#initial-omniauth-configuration) or +[LDAP configuration](../../../administration/auth/ldap/index.md#basic-configuration-settings). + ## Require email confirmation You can send confirmation emails during sign up and require that users confirm @@ -56,7 +62,7 @@ To enforce confirmation of the email address used for new sign ups: 1. On the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. Select the **Enable email restrictions for sign ups** checkbox, then select **Save changes**. -## User cap **(FREE SELF)** +## User cap > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4315) in GitLab 13.7. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292600) in GitLab 13.9. @@ -117,7 +123,7 @@ the minimum number of characters a user must have in their password using the Gi You can specify an inclusive or exclusive list of email domains which can be used for user sign up. These restrictions are only applied during sign up from an external user. An administrator can add a -user through the admin panel with a disallowed domain. Also, note that the users can change their +user through the administrator panel with a disallowed domain. Also, note that the users can change their email addresses to disallowed domains after sign up. ### Allowlist email domains diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index a9c8c5d2a76..c9d48b14a6c 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -7,7 +7,7 @@ type: reference # Third-party offers **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20379) in GitLab Free 11.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20379) in GitLab 11.1. Within GitLab, we inform users of available third-party offers they might find valuable in order to enhance the development of their projects. An example is the Google Cloud Platform free credit diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index 330a25087ef..923ea9e19c1 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -12,7 +12,7 @@ to perform various actions. All usage statistics are [opt-out](#enable-or-disable-usage-statistics). -## Service Ping **(FREE SELF)** +## Service Ping Service Ping is a process that collects and sends a weekly payload to GitLab Inc. For more information, see the [Service Ping guide](../../../development/service_ping/index.md). @@ -23,7 +23,7 @@ When Service Ping is enabled, GitLab gathers data from other instances and enables certain [instance-level analytics features](../analytics/index.md) that are dependent on Service Ping. -## Version check **(FREE SELF)** +## Version check If enabled, version check informs you if a new version is available and the importance of it through a status. The status displays on the help pages (`/help`) @@ -76,7 +76,7 @@ To enable or disable Service Ping and version check: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand **Usage statistics**. -1. Select or clear the **Enable version check** and **Enable service ping** checkboxes. +1. Select or clear the **Enable version check** and **Enable Service Ping** checkboxes. 1. Select **Save changes**. <!-- ## Troubleshooting diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index 32f08801c76..ac6fe29da28 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -189,6 +189,10 @@ The possible names are: - `throttle_unauthenticated_packages_api` - `throttle_authenticated_packages_api` - `throttle_authenticated_git_lfs` +- `throttle_unauthenticated_files_api` +- `throttle_authenticated_files_api` +- `throttle_unauthenticated_deprecated_api` +- `throttle_authenticated_deprecated_api` For example, to try out throttles for all authenticated requests to non-protected paths can be done by setting diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 7f3f4b32802..075becfd32f 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -21,10 +21,7 @@ To access the visibility and access control options: With this option, you can define [branch protections](../../project/protected_branches.md) to apply to every repository's [default branch](../../project/repository/branches/default.md). -These protections specify the user roles with permission to: - -- Push to branches. -- Delete branches. +These protections specify the user roles with permission to push to default branches. This setting applies only to each repository's default branch. To protect other branches, you must configure [branch protection in the repository](../../project/protected_branches.md), @@ -37,14 +34,14 @@ To change the default branch protection for the entire instance: 1. On the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. 1. Select a **Default branch protection**: - - **Not protected** - Both developers and maintainers can push new commits, - force push, or delete the branch. + - **Not protected** - Both developers and maintainers can push new commits + and force push. - **Protected against pushes** - Developers cannot push new commits, but are allowed to accept merge requests to the branch. Maintainers can push to the branch. - **Partially protected** - Both developers and maintainers can push new commits, - but cannot force push or delete the branch. + but cannot force push. - **Fully protected** - Developers cannot push new commits, but maintainers can. - No one can force push or delete the branch. + No one can force push. 1. To allow group owners to override the instance's default branch protection, select [**Allow owners to manage default branch protection per group**](#prevent-overrides-of-default-branch-protection). 1. Select **Save changes**. @@ -295,7 +292,7 @@ For more details, see [SSH key restrictions](../../../security/ssh_keys_restrict ## Enable project mirroring This option is enabled by default. By disabling it, both -[pull and push mirroring](../../project/repository/repository_mirroring.md) no longer +[pull mirroring](../../project/repository/mirror/pull.md) and [push mirroring](../../project/repository/mirror/push.md) no longer work in every repository. They can only be re-enabled by an administrator user on a per-project basis.  diff --git a/doc/user/admin_area/user_cohorts.md b/doc/user/admin_area/user_cohorts.md index 89026e56a27..28376923810 100644 --- a/doc/user/admin_area/user_cohorts.md +++ b/doc/user/admin_area/user_cohorts.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cohorts **(FREE)** +# Cohorts **(FREE SELF)** You can analyze your users' GitLab activities over time. diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 3c80e1f5f2a..4a42133c308 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -24,7 +24,7 @@ View pipeline duration history: ## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. -> - Added support for [lead time for changes](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) in GitLab 13.10. +> - [Added support](https://gitlab.com/gitlab-org/gitlab/-/issues/291746) for lead time for changes in GitLab 13.10. Customer experience is a key metric. Users want to measure platform stability and other post-deployment performance KPIs, and set targets for customer behavior, experience, and financial @@ -56,7 +56,7 @@ The following table shows the supported metrics, at which level they are support | `change_failure_rate` | Project/Group-level | To be supported | To be supported | | | `time_to_restore_service` | Project/Group-level | To be supported | To be supported | | -### Deployment frequency charts **(ULTIMATE)** +### Deployment frequency charts > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.8. @@ -69,7 +69,7 @@ for its deployment information to appear on the graphs. These charts are available for both groups and projects. -### Lead time charts **(ULTIMATE)** +### Lead time charts > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250329) in GitLab 13.11. diff --git a/doc/user/analytics/img/project_vsa_filter_v14_3.png b/doc/user/analytics/img/project_vsa_filter_v14_3.png Binary files differnew file mode 100644 index 00000000000..d3fcad31909 --- /dev/null +++ b/doc/user/analytics/img/project_vsa_filter_v14_3.png diff --git a/doc/user/analytics/img/project_vsa_stage_table_v14_4.png b/doc/user/analytics/img/project_vsa_stage_table_v14_4.png Binary files differnew file mode 100644 index 00000000000..e65296062f6 --- /dev/null +++ b/doc/user/analytics/img/project_vsa_stage_table_v14_4.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 5b7e6e39187..c01788f4fc4 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w When we describe GitLab analytics, we use the following terms: -- **Cycle time:** The duration of only the execution work alone. Often displayed in combination with "lead time," which is longer. GitLab measures cycle time from issue first merge request creation to issue close. This approach underestimates lead time because merge request creation is always later than commit time. GitLab displays cycle time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md). +- **Cycle time:** The duration of only the execution work. Cycle time is often displayed in combination with the lead time, which is longer than the cycle time. GitLab measures cycle time from the earliest commit of a [linked issue's merge request](../project/issues/crosslinking_issues.md) to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation is always later than commit time. GitLab displays cycle time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](../analytics/value_stream_analytics.md). - **Deploys:** The total number of successful deployments to production in the given time frame (across all applicable projects). GitLab displays deploys in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). - **DORA (DevOps Research and Assessment)** ["Four Keys"](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance): - **Speed/Velocity** diff --git a/doc/user/analytics/issue_analytics.md b/doc/user/analytics/issue_analytics.md index 44b8c87ee27..af7307eab39 100644 --- a/doc/user/analytics/issue_analytics.md +++ b/doc/user/analytics/issue_analytics.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Issue Analytics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196561) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196561) in GitLab 12.9. Issue Analytics is a bar graph which illustrates the number of issues created each month. The default time span is 13 months, which includes the current month, and the 12 months diff --git a/doc/user/analytics/merge_request_analytics.md b/doc/user/analytics/merge_request_analytics.md index 44e4cd8b371..2ff15c00a3f 100644 --- a/doc/user/analytics/merge_request_analytics.md +++ b/doc/user/analytics/merge_request_analytics.md @@ -115,5 +115,5 @@ bookmark for those preferred settings in your browser. The **Merge Request Analytics** feature can be accessed only: -- On [Premium](https://about.gitlab.com/pricing/) and above. +- On [GitLab Premium](https://about.gitlab.com/pricing/) and above. - By users with [Reporter access](../permissions.md) and above. diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index 391ec5c04d9..dbe71d5f743 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Productivity Analytics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12079) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12079) in GitLab 12.3. Track development velocity with Productivity Analytics. @@ -60,5 +60,5 @@ You can filter analytics based on a date range. To filter results: The **Productivity Analytics** dashboard can be accessed only: -- On the [Premium tier](https://about.gitlab.com/pricing/) and above. +- On [GitLab Premium](https://about.gitlab.com/pricing/) and above. - By users with [Reporter access](../permissions.md) and above. diff --git a/doc/user/analytics/repository_analytics.md b/doc/user/analytics/repository_analytics.md index 9016bf6393f..e8f8acf5661 100644 --- a/doc/user/analytics/repository_analytics.md +++ b/doc/user/analytics/repository_analytics.md @@ -4,7 +4,7 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Repository Analytics +# Repository Analytics **(FREE)** Get high-level overview of the project's Git repository. diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 7057d069e95..9c1a8893f95 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -43,12 +43,59 @@ The stages tracked by Value Stream Analytics by default represent the [GitLab fl - **Staging** (Continuous Deployment) - Time between merging and deploying to production +## Filter Value Stream Analytics data + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326701) in GitLab 14.3 + +You can filter analytics based on the following parameters: + +- Milestones (Group level) +- Labels (Group level) +- Author +- Assignees + +To filter results: + +1. Select the **Filter results** text box. +1. Select a parameter. +1. Select a value. To find a value in the list, enter the value name. + + + ### Date ranges To filter analytics results based on a date range, select different **From** and **To** days from the date picker (default: last 30 days). +### Stage table + +> Sorting the stage table [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335974) in GitLab 14.4. + + + +The stage table shows a list of related workflow items for the selected stage. This can include: + +- CI/CD jobs +- Issues +- Merge requests +- Pipelines + +A little badge next to the workflow items table header shows the number of workflow items that +completed the selected stage. + +The stage table also includes the **Time** column, which shows how long it takes each item to pass +through the selected value stream stage. + +To sort the stage table by a table column, select the table header. +You can sort in ascending or descending order. To find items that spent the most time in a stage, +potentially causing bottlenecks in your value stream, sort the table by the **Time** column. +From there, select individual items to drill in and investigate how delays are happening. +To see which items most recently exited the stage, sort by the work item column on the left. + +The table displays 20 items per page. If there are more than 20 items, you can use the +**Prev** and **Next** buttons to navigate through the pages. + ## How Time metrics are measured The **Time** metrics near the top of the page are measured as follows: diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index 7940e072420..db0b2a32bcf 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -68,8 +68,12 @@ Install GitLab HAR Recorder: 1. Make sure proxy is used! 1. Stop the recorder. -To verify the HAR contains all requests, use the [HAR Viewer (online)](http://www.softwareishard.com/har/viewer/). -[Google Admin Toolbox HAR Analyzer](https://toolbox.googleapps.com/apps/har_analyzer/) +To verify the HAR contains all requests, use an online HAR viewer, for example: + +- [HAR Viewer](http://www.softwareishard.com/har/viewer/) +<!-- vale gitlab.Admin = NO --> +- [Google Admin Toolbox HAR Analyzer](https://toolbox.googleapps.com/apps/har_analyzer/) +<!-- vale gitlab.Admin = YES --> ### Insomnia API Client @@ -190,7 +194,9 @@ a text editor. Tools recommended for viewing HAR files include: - [HAR Viewer](http://www.softwareishard.com/har/viewer/) - (online) +<!-- vale gitlab.Admin = NO --> - [Google Admin Toolbox HAR Analyzer](https://toolbox.googleapps.com/apps/har_analyzer/) - (online) +<!-- vale gitlab.Admin = YES --> - [Fiddler](https://www.telerik.com/fiddler) - local - [Insomnia API Client](https://insomnia.rest/) - local diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 98e241ba3bd..674eee5e80a 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -38,7 +38,7 @@ Select **Configuration history** to see the `.gitlab-ci.yml` file's history. You can configure the following security controls: -- Static Application Security Testing (SAST) +- Static Application Security Testing (SAST) **(FREE)** - Select **Enable SAST** to configure SAST for the current project. For more details, read [Configure SAST in the UI](../sast/index.md#configure-sast-in-the-ui). - Dynamic Application Security Testing (DAST) **(ULTIMATE)** diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 2b3d4dbfc0a..22b54bf019c 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -42,10 +42,20 @@ To enable container scanning in your pipeline, you need the following: shared runners on GitLab.com, then this is already the case. - An image matching the [supported distributions](#supported-distributions). - [Build and push](../../packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) - the Docker image to your project's container registry. If using a third-party container - registry, you might need to provide authentication credentials using the `DOCKER_USER` and - `DOCKER_PASSWORD` [configuration variables](#available-cicd-variables). + the Docker image to your project's container registry. - The name of the Docker image to scan, in the `DOCKER_IMAGE` [configuration variable](#available-cicd-variables). +- If you're using a third-party container registry, you might need to provide authentication + credentials through the `DOCKER_USER` and `DOCKER_PASSWORD` [configuration variables](#available-cicd-variables). + For example, if you are connecting to AWS ECR, you might use the following: + +```yaml +export AWS_ECR_PASSWORD=$(aws ecr get-login-password --region region) + +include: + - template: Security/Container-Scanning.gitlab-ci.yml + DOCKER_USER: AWS + DOCKER_PASSWORD: "$AWS_ECR_PASSWORD" +``` ## Configuration @@ -397,7 +407,7 @@ For details on saving and transporting Docker images as a file, see Docker's doc We recommend that you set up a [scheduled pipeline](../../../ci/pipelines/schedules.md) to fetch the latest vulnerabilities database on a preset schedule. Automating this with a pipeline means you do not have to do it manually each time. You can use the -following `.gitlab-yml.ci` example as a template. +following `.gitlab-ci.yml` example as a template. ```yaml variables: @@ -420,6 +430,39 @@ The above template works for a GitLab Docker registry running on a local install you're using a non-GitLab Docker registry, you must change the `$CI_REGISTRY` value and the `docker login` credentials to match your local registry's details. +#### Scan images in external private registries + +To scan an image in an external private registry, you must configure access credentials so the +container scanning analyzer can authenticate itself before attempting to access the image to scan. + +If you use the GitLab [Container Registry](../../packages/container_registry/), +the `DOCKER_USER` and `DOCKER_PASSWORD` [configuration variables](#available-cicd-variables) +are set automatically and you can skip this configuration. + +This example shows the configuration needed to scan images in a private [Google Container Registry](https://cloud.google.com/container-registry/): + +```yaml +include: + - template: Security/Container-Scanning.gitlab-ci.yml + +container_scanning: + variables: + DOCKER_USER: _json_key + DOCKER_PASSWORD: "$GCP_CREDENTIALS" + DOCKER_IMAGE: "gcr.io/path-to-you-registry/image:tag" +``` + +Before you commit this configuration, [add a CI/CD variable](../../../ci/variables/#add-a-cicd-variable-to-a-project) +for `GCP_CREDENTIALS` containing the JSON key, as described in the +[Google Cloud Platform Container Registry documentation](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key). +Also: + +- The value of the variable may not fit the masking requirements for the **Mask variable** option, + so the value could be exposed in the job logs. +- Scans may not run in unprotected feature branches if you select the **Protect variable** option. +- Consider creating credentials with read-only permissions and rotating them regularly if the + options aren't selected. + ## Running the standalone container scanning tool It's possible to run the [GitLab container scanning tool](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning) diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 0d5eb2b6d50..cdb2e7109bf 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -7,15 +7,26 @@ type: reference, howto # Coverage-guided fuzz testing **(ULTIMATE)** +Coverage-guided fuzzing sends random inputs to an instrumented version of your application in an +effort to cause unexpected behavior. Such behavior indicates a bug that you should address. GitLab allows you to add coverage-guided fuzz testing to your pipelines. This helps you discover -bugs and potential security issues that other QA processes may miss. Coverage-guided fuzzing sends -random inputs to an instrumented version of your application in an effort to cause unexpected -behavior, such as a crash. Such behavior indicates a bug that you should address. +bugs and potential security issues that other QA processes may miss. We recommend that you use fuzz testing in addition to the other security scanners in [GitLab Secure](../index.md) and your own test processes. If you're using [GitLab CI/CD](../../../ci/index.md), -you can run your coverage-guided fuzz tests as part your CI/CD workflow. You can take advantage of -coverage-guided fuzzing by including the CI job in your existing `.gitlab-ci.yml` file. +you can run your coverage-guided fuzz tests as part your CI/CD workflow. + +## Coverage-guided fuzz testing process + +The fuzz testing process: + +1. Compiles the target application. +1. Runs the instrumented application, using the `gitlab-cov-fuzz` tool. +1. Parses and analyzes the exception information output by the fuzzer. +1. Downloads the [corpus](../terminology/index.md#corpus) and crash events from previous pipelines. +1. Outputs the parsed crash events and data to the `gl-coverage-fuzzing-report.json` file. + +The results of the coverage-guided fuzz testing are available in the CI/CD pipeline. ## Supported fuzzing engines and languages @@ -249,6 +260,8 @@ which shows an overview of all the security vulnerabilities in your groups, proj Clicking the vulnerability opens a modal that provides additional information about the vulnerability: +<!-- vale gitlab.Acronyms = NO --> + - Status: The vulnerability's status. As with any type of vulnerability, a coverage fuzzing vulnerability can be Detected, Confirmed, Dismissed, or Resolved. - Project: The project in which the vulnerability exists. @@ -262,3 +275,5 @@ vulnerability: - Scanner: The scanner that detected the vulnerability (for example, Coverage Fuzzing). - Scanner Provider: The engine that did the scan. For Coverage Fuzzing, this can be any of the engines listed in [Supported fuzzing engines and languages](#supported-fuzzing-engines-and-languages). + +<!-- vale gitlab.Acronyms = YES --> diff --git a/doc/user/application_security/cve_id_request.md b/doc/user/application_security/cve_id_request.md index 1489b250e4b..5ffd47527c5 100644 --- a/doc/user/application_security/cve_id_request.md +++ b/doc/user/application_security/cve_id_request.md @@ -5,65 +5,64 @@ group: Threat Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CVE ID Requests **(FREE SAAS)** +# CVE ID request **(FREE SAAS)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41203) in GitLab 13.4, only for public projects on GitLab.com. -As part of [our role as a CVE Numbering Authority](https://about.gitlab.com/security/cve/) -([CNA](https://cve.mitre.org/cve/cna.html)), you may request -[CVE](https://cve.mitre.org/index.html) identifiers from GitLab to track -vulnerabilities found within your project. +A [CVE](https://cve.mitre.org/index.html) identifier is assigned to a publicly-disclosed software +vulnerability. GitLab is a [CVE Numbering Authority](https://about.gitlab.com/security/cve/) +([CNA](https://cve.mitre.org/cve/cna.html)). For any public project you can request +a CVE identifier (ID). -## Overview +Assigning a CVE ID to a vulnerability in your project helps your users stay secure and informed. For +example, [dependency scanning tools](../application_security/dependency_scanning/index.md) can +detect when vulnerable versions of your project are used as a dependency. -CVE identifiers track specific vulnerabilities within projects. Having a CVE assigned to a -vulnerability in your project helps your users stay secure and informed. For example, -[dependency scanning tools](../application_security/dependency_scanning/index.md) -can detect when vulnerable versions of your project are used as a dependency. +A common vulnerability workflow is: -## Conditions +1. Request a CVE for a vulnerability. +1. Reference the assigned CVE identifier in release notes. +1. Publish the vulnerability's details after the fix is released. -If the following conditions are met, a **Request CVE ID** button appears in your issue sidebar: +## Prerequisites -- The project is hosted in GitLab.com. +To [submit a CVE ID Request](#submit-a-cve-id-request) the following prerequisites must be met: + +- The project is hosted on GitLab.com. - The project is public. - You are a maintainer of the project. -- The issue is [confidential](../project/issues/confidential_issues.md). - -## Submitting a CVE ID Request - -Clicking the **Request CVE ID** button in the issue sidebar takes you to the new issue page for -the [GitLab CVE project](https://gitlab.com/gitlab-org/cves). +- The vulnerability's issue is [confidential](../project/issues/confidential_issues.md). - +## Submit a CVE ID request -Creating the [confidential issue](../project/issues/confidential_issues.md) starts the CVE request process. +To submit a CVE ID request: - +1. Go to the vulnerability's issue and select **Create CVE ID Request**. The new issue page of + the [GitLab CVE project](https://gitlab.com/gitlab-org/cves) opens. -You are required to fill in the issue description, which includes: +  -- A description of the vulnerability -- The project's vendor and name -- Impacted versions -- Fixed versions -- The vulnerability type (a [CWE](https://cwe.mitre.org/data/index.html) identifier) -- A [CVSS v3 vector](https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator) +1. In the **Title** box, enter a brief description of the vulnerability. -## CVE Assignment +1. In the **Description** box, enter the following details: -GitLab triages your submitted CVE ID request and communicates with you throughout the CVE validation -and assignment process. + - A detailed description of the vulnerability + - The project's vendor and name + - Impacted versions + - Fixed versions + - The vulnerability class (a [CWE](https://cwe.mitre.org/data/index.html) identifier) + - A [CVSS v3 vector](https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator) - +  -Once a CVE identifier is assigned, you may use and reference it as you see fit. +GitLab updates your CVE ID request issue when: -Details of the vulnerability submitted in the CVE ID request are published according to your -schedule. It's common to request a CVE for an unpatched vulnerability, reference the assigned CVE -identifier in release notes, and later publish the vulnerability's details after the fix is -released. +- Your submission is assigned a CVE. +- Your CVE is published. +- MITRE is notified that your CVE is published. +- MITRE has added your CVE in the NVD feed. -Separate communications notify you when different stages of the publication process are complete. +## CVE assignment - +After a CVE identifier is assigned, you can reference it as required. Details of the vulnerability +submitted in the CVE ID request are published according to your schedule. diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 5094ccd2196..9c5b84f4f36 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -10,13 +10,13 @@ type: reference, howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323423) in GitLab 13.12. WARNING: -This product is an early access and is considered a [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) feature. +This product is in an early-access stage and is considered a [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) feature. GitLab DAST's new browser-based crawler is a crawl engine built by GitLab to test Single Page Applications (SPAs) and traditional web applications. Due to the reliance of modern web applications on JavaScript, handling SPAs or applications that are dependent on JavaScript is paramount to ensuring proper coverage of an application for Dynamic Application Security Testing (DAST). -The browser-based crawler works by loading the target application into a specially-instrumented Chromium browser. A snapshot of the page is taken prior to a search to find any actions that a user might perform, -such as clicking on a link or filling in a form. For each action found, the crawler will execute it, take a new snapshot and determine what in the page changed from the previous snapshot. +The browser-based crawler works by loading the target application into a specially-instrumented Chromium browser. A snapshot of the page is taken before a search to find any actions that a user might perform, +such as clicking on a link or filling in a form. For each action found, the crawler executes it, takes a new snapshot, and determines what in the page changed from the previous snapshot. Crawling continues by taking more snapshots and finding subsequent actions. The benefit of crawling by following user actions in a browser is that the crawler can interact with the target application much like a real user would, identifying complex flows that traditional web crawlers don't understand. This results in better coverage of the website. @@ -57,17 +57,17 @@ The browser-based crawler can be configured using CI/CD variables. | `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed but not reported against. | | `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, clicking a link, or filling a form. | | `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | -| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but will likely produce little benefit after five to seven instances. | +| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com, we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but are likely to produce little benefit after five to seven instances. | | `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | | `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended log level. | -| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another | -| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action | -| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis | -| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes | -| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action | -| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or navigations | -| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations | -| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis | +| `DAST_BROWSER_NAVIGATION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `15s` | The maximum amount of time to wait for a browser to navigate from one page to another. | +| `DAST_BROWSER_ACTION_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to complete an action. | +| `DAST_BROWSER_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis. | +| `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `7s` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after a navigation completes. | +| `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `800ms` | The maximum amount of time to wait for a browser to consider a page loaded and ready for analysis after completing an action. | +| `DAST_BROWSER_SEARCH_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `3s` | The maximum amount of time to allow the browser to search for new elements or navigations. | +| `DAST_BROWSER_EXTRACT_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `5s` | The maximum amount of time to allow the browser to extract newly found elements or navigations. | +| `DAST_BROWSER_ELEMENT_TIMEOUT` | [Duration string](https://golang.org/pkg/time/#ParseDuration) | `600ms` | The maximum amount of time to wait for an element before determining it is ready for analysis. | The [DAST variables](index.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, `DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, @@ -80,27 +80,27 @@ While the browser-based crawler crawls modern web applications efficiently, vuln The crawler runs the target website in a browser with DAST/ZAP configured as the proxy server. This ensures that all requests and responses made by the browser are passively scanned by DAST/ZAP. When running a full scan, active vulnerability checks executed by DAST/ZAP do not use a browser. This difference in how vulnerabilities are checked can cause issues that require certain features of the target website to be disabled to ensure the scan works as intended. -For example, for a target website that contains forms with Anti-CSRF tokens, a passive scan will scan as intended because the browser displays pages/forms as if a user is viewing the page. -However, active vulnerability checks run in a full scan will not be able to submit forms containing Anti-CSRF tokens. In such cases we recommend you disable Anti-CSRF tokens when running a full scan. +For example, for a target website that contains forms with Anti-CSRF tokens, a passive scan works as intended because the browser displays pages and forms as if a user is viewing the page. +However, active vulnerability checks that run in a full scan cannot submit forms containing Anti-CSRF tokens. In such cases, we recommend you disable Anti-CSRF tokens when running a full scan. ## Managing scan time -It is expected that running the browser-based crawler will result in better coverage for many web applications, when compared to the normal GitLab DAST solution. +It is expected that running the browser-based crawler results in better coverage for many web applications, when compared to the normal GitLab DAST solution. This can come at a cost of increased scan time. You can manage the trade-off between coverage and scan time with the following measures: - Limit the number of actions executed by the browser with the [variable](#available-cicd-variables) `DAST_BROWSER_MAX_ACTIONS`. The default is `10,000`. - Limit the page depth that the browser-based crawler will check coverage on with the [variable](#available-cicd-variables) `DAST_BROWSER_MAX_DEPTH`. The crawler uses a breadth-first search strategy, so pages with smaller depth are crawled first. The default is `10`. -- Vertically scaling the runner and using a higher number of browsers with [variable](#available-cicd-variables) `DAST_BROWSER_NUMBER_OF_BROWSERS`. The default is `3`. +- Vertically scale the runner and use a higher number of browsers with [variable](#available-cicd-variables) `DAST_BROWSER_NUMBER_OF_BROWSERS`. The default is `3`. ## Timeouts Due to poor network conditions or heavy application load, the default timeouts may not be applicable to your application. -Browser-based scans offer the ability to adjust various timeouts to ensure it continues smoothly as it transitions from one page to the next. These values are configured using a [Duration string](https://golang.org/pkg/time/#ParseDuration) which allow you to configure durations with a prefix: `m` for minutes, `s` for seconds, and `ms` for milliseconds. +Browser-based scans offer the ability to adjust various timeouts to ensure it continues smoothly as it transitions from one page to the next. These values are configured using a [Duration string](https://golang.org/pkg/time/#ParseDuration), which allow you to configure durations with a prefix: `m` for minutes, `s` for seconds, and `ms` for milliseconds. -Navigations, or the act of loading a new page, usually require the most amount of time as they are +Navigations, or the act of loading a new page, usually require the most amount of time because they are loading multiple new resources such as JavaScript or CSS files. Depending on the size of these resources, or the speed at which they are returned, the default `DAST_BROWSER_NAVIGATION_TIMEOUT` may not be sufficient. Stability timeouts, such as those configurable with `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT`, `DAST_BROWSER_STABILITY_TIMEOUT`, and `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` can also be configured. Stability timeouts determine when browser-based scans consider @@ -110,11 +110,11 @@ a page fully loaded. Browser-based scans consider a page loaded when: 1. There are no open or outstanding requests that are deemed important, such as JavaScript and CSS. Media files are usually deemed unimportant. 1. Depending on whether the browser executed a navigation, was forcibly transitioned, or action: - - There are no new Document Object Model (DOM) modification events after the `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT`, `DAST_BROWSER_STABILITY_TIMEOUT` or `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` durations + - There are no new Document Object Model (DOM) modification events after the `DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT`, `DAST_BROWSER_STABILITY_TIMEOUT`, or `DAST_BROWSER_ACTION_STABILITY_TIMEOUT` durations. -After these events have occurred, browser-based scans consider the page loaded and ready and attempt the next action. +After these events have occurred, browser-based scans consider the page loaded and ready, and attempt the next action. -If your application experiences latency or returns many navigation failures, consider adjusting the timeout values such in this example: +If your application experiences latency or returns many navigation failures, consider adjusting the timeout values such as in this example: ```yaml include: @@ -132,7 +132,7 @@ dast: ``` NOTE: -Adjusting these values may impact scan time as they adjust how long each browser waits for various activities to complete. +Adjusting these values may impact scan time because they adjust how long each browser waits for various activities to complete. ## Debugging scans using logging @@ -168,7 +168,7 @@ The modules that can be configured for logging are as follows: | Log module | Component overview | | ---------- | ----------- | | `AUTH` | Used for creating an authenticated scan. | -| `BROWS` | Used for querying the state/page of the browser. | +| `BROWS` | Used for querying the state or page of the browser. | | `BPOOL` | The set of browsers that are leased out for crawling. | | `CRAWL` | Used for the core crawler algorithm. | | `DATAB` | Used for persisting data to the internal database. | diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md index f771bc82d58..9969526c906 100644 --- a/doc/user/application_security/dast/dast_troubleshooting.md +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -20,7 +20,7 @@ A DAST job has two executing processes: Enable the `DAST_DEBUG` CI/CD variable to debug scripts. This can help when troubleshooting the job, and outputs statements indicating what percentage of the scan is complete. -For details on using variables, see [Overriding the DAST template](index.md#customizing-the-dast-settings). +For details on using variables, see [Overriding the DAST template](index.md#customize-dast-settings). Debug mode of the ZAP server can be enabled using the `DAST_ZAP_LOG_CONFIGURATION` variable. The following table outlines examples of values that can be set and the effect that they have on the output that is logged. diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 0d60701b030..09b55e7b395 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -269,7 +269,7 @@ image. Using the `DAST_VERSION` variable, you can choose how DAST updates: - Only update fixes by pinning to a minor version (such as `1.6`). - Prevent all updates by pinning to a specific version (such as `1.6.4`). -Find the latest DAST versions on the [Releases](https://gitlab.com/security-products/dast/-/releases) +Find the latest DAST versions on the [Releases](https://gitlab.com/gitlab-org/security-products/dast/-/releases) page. #### Configure DAST using the UI @@ -483,6 +483,13 @@ When using `DAST_PATHS` and `DAST_PATHS_FILE`, note the following: To perform a [full scan](#full-scan) on the listed paths, use the `DAST_FULL_SCAN_ENABLED` CI/CD variable. +### List URLs scanned + +When DAST completes scanning, the merge request page states the number of URLs scanned. +Click **View details** to view the web console output which includes the list of scanned URLs. + + + ### View details of a vulnerability detected by DAST > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. @@ -515,15 +522,20 @@ To view details of vulnerabilities detected by DAST: | Links | Links to further details of the detected vulnerability. | | Solution | Details of a recommended solution to the vulnerability (optional). | -### Customizing the DAST settings +## Customize DAST settings + +You can customize the behavior of DAST using both CI/CD variables and command-line options. Use of CI/CD +variables overrides the values contained in the DAST template. + +### Customize DAST using CI/CD variables WARNING: Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) -is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/index.md#rules) instead. +is no longer supported. You must use [`rules`](../../../ci/yaml/index.md#rules) instead. The DAST settings can be changed through CI/CD variables by using the -[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. -These variables are documented in [available variables](#available-cicd-variables). +[`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. For details of +all DAST CI/CD variables, read [Available CI/CD variables](#available-cicd-variables). For example: @@ -539,10 +551,10 @@ variables: Because the template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline configuration, the last mention of the variable takes precedence. -#### Enabling and disabling rules +#### Enable or disable rules A complete list of the rules that DAST uses to scan for vulnerabilities can be -found in the [ZAP docs](https://www.zaproxy.org/docs/alerts/). +found in the [ZAP documentation](https://www.zaproxy.org/docs/alerts/). `DAST_EXCLUDE_RULES` disables the rules with the given IDs. @@ -559,7 +571,7 @@ can be found in [exclude_rules.yml](https://gitlab.com/gitlab-org/security-produ The lists for `DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` **must** be enclosed in double quotes (`"`), otherwise they are interpreted as numeric values. -### Hide sensitive information +#### Hide sensitive information > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. @@ -573,7 +585,105 @@ authorization credentials. By default, the following headers are masked: Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the headers whose values you want masked. For details on how to mask headers, see -[Customizing the DAST settings](#customizing-the-dast-settings). +[Customizing the DAST settings](#customize-dast-settings). + +#### Available CI/CD variables + +These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. + +| CI/CD variable | Type | Description | +|:-------------------------------------------------|:--------------|:------------------------------| +| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | +| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | +| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | +| `DAST_API_OPENAPI` | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290241) in GitLab 13.12 and replaced by `DAST_API_OPENAPI`. To be removed in GitLab 15.0. The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | +| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | +| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | +| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | +| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | +| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | +| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that will be clicked on prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | +| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | +| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | +| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when clicked submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | +| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | +| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | +| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | +| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | +| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | +| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | +| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | +| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | +| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | +| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when clicked submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | +| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | +| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | +| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_OPENAPI` must be specified if this is omitted. | +| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `log4j.logger.org.parosproxy.paros.network.HttpSender=DEBUG;log4j.logger.com.crawljax=DEBUG` | +| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | + +1. Available to an on-demand DAST scan. +1. Used for authentication. + +### Customize DAST using command-line options + +Not all DAST configuration is available via CI/CD variables. To find out all +possible options, run the following configuration. +Available command-line options are printed to the job log: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - /analyze --help +``` + +You must then overwrite the `script` command to pass in the appropriate +argument. For example, vulnerability definitions in alpha can be included with +`-a`. The following configuration includes those definitions: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + script: + - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} + - /analyze -a -t $DAST_WEBSITE +``` + +### Custom ZAProxy configuration + +The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). +Many key/values for `-config` remain undocumented, but there is an untested list of +[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). +Note that these options are not supported by DAST, and may break the DAST scan +when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" +``` ## Authentication @@ -747,59 +857,6 @@ dast: when: always ``` -## Available CI/CD variables - -These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. - -| CI/CD variable | Type | Description | -|:-------------------------------------------------|:--------------|:------------------------------| -| `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | -| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | -| `DAST_API_HOST_OVERRIDE` <sup>1</sup> | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080`. | -| `DAST_API_OPENAPI` | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_API_SPECIFICATION` <sup>1</sup> | URL or string | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290241) in GitLab 13.12 and replaced by `DAST_API_OPENAPI`. To be removed in GitLab 15.0. The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. The variable `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_AUTH_REPORT` <sup>2</sup> | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | -| `DAST_AUTH_EXCLUDE_URLS` <sup>2</sup> | URLs | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/289959)** in GitLab 14.0. Replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | -| `DAST_AUTH_URL` <sup>1,2</sup> | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. Example: `https://login.example.com`. | -| `DAST_AUTH_VERIFICATION_LOGIN_FORM` <sup>2</sup> | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | -| `DAST_AUTH_VERIFICATION_SELECTOR` <sup>2</sup> | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo`. | -| `DAST_AUTH_VERIFICATION_URL` <sup>1,2</sup> | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. Example: `"http://example.com/loggedin_page"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | -| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false`. | -| `DAST_BROWSER_PATH_TO_LOGIN_FORM` <sup>1,2</sup> | selector | Comma-separated list of selectors that will be clicked on prior to attempting to enter `DAST_USERNAME` and `DAST_PASSWORD` into the login form. Example: `"css:.navigation-menu,css:.login-menu-item"`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326633) in GitLab 14.1. | -| `DAST_DEBUG` <sup>1</sup> | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. The whole list **must** be enclosed in double quotes (`"`). Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | -| `DAST_EXCLUDE_URLS` <sup>1,2</sup> | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. Example, `http://example.com/sign-out`. | -| `DAST_FIRST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when clicked submits the username form of a multi-page login process. For example, `css:button[type='user-submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293595)** in GitLab 14.0. Set to `true` to require domain validation when running DAST full scans. Not supported for API scans. Default: `false` | -| `DAST_FULL_SCAN_ENABLED` <sup>1</sup> | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | -| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | -| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. The whole list **must** be enclosed in double quotes (`"`). Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | -| `DAST_PASSWORD` <sup>1,2</sup> | string | The password to authenticate to in the website. Example: `P@55w0rd!` | -| `DAST_PASSWORD_FIELD` <sup>1,2</sup> | string | The selector of password field at the sign-in HTML form. Example: `id:password` | -| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | -| `DAST_REQUEST_HEADERS` <sup>1</sup> | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | -| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | -| `DAST_SPIDER_MINS` <sup>1</sup> | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | -| `DAST_SUBMIT_FIELD` <sup>2</sup> | string | The `id` or `name` of the element that when clicked submits the login form or the password form of a multi-page login process. For example, `css:button[type='submit']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_TARGET_AVAILABILITY_TIMEOUT` <sup>1</sup> | number | Time limit in seconds to wait for target availability. | -| `DAST_USE_AJAX_SPIDER` <sup>1</sup> | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_USERNAME` <sup>1,2</sup> | string | The username to authenticate to in the website. Example: `admin` | -| `DAST_USERNAME_FIELD` <sup>1,2</sup> | string | The selector of username field at the sign-in HTML form. Example: `name:username` | -| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_WEBSITE` <sup>1</sup> | URL | The URL of the website to scan. The variable `DAST_API_OPENAPI` must be specified if this is omitted. | -| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. Example: `log4j.logger.org.parosproxy.paros.network.HttpSender=DEBUG;log4j.logger.com.crawljax=DEBUG` | -| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | - -1. Available to an on-demand DAST scan. -1. Used for authentication. - ### Selectors Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. @@ -848,51 +905,6 @@ When using selectors to locate specific fields we recommend you avoid searching - XPath searches as they are less performant than other selector searches. - Unscoped searches, such as those beginning with `css:*` and `xpath://*`. -### DAST command-line options - -Not all DAST configuration is available via CI/CD variables. To find out all -possible options, run the following configuration. -Available command-line options are printed to the job log: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - /analyze --help -``` - -You must then overwrite the `script` command to pass in the appropriate -argument. For example, vulnerability definitions in alpha can be included with -`-a`. The following configuration includes those definitions: - -```yaml -include: - template: DAST.gitlab-ci.yml - -dast: - script: - - export DAST_WEBSITE=${DAST_WEBSITE:-$(cat environment_url.txt)} - - /analyze -a -t $DAST_WEBSITE -``` - -### Custom ZAProxy configuration - -The ZAProxy server contains many [useful configurable values](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_245801885). -Many key/values for `-config` remain undocumented, but there is an untested list of -[possible keys](https://gitlab.com/gitlab-org/gitlab/-/issues/36437#note_244981023). -Note that these options are not supported by DAST, and may break the DAST scan -when used. An example of how to rewrite the Authorization header value with `TOKEN` follows: - -```yaml -include: - template: DAST.gitlab-ci.yml - -variables: - DAST_ZAP_CLI_OPTIONS: "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=TOKEN" -``` - ### Bleeding-edge vulnerability definitions ZAP first creates rules in the `alpha` class. After a testing period with @@ -1002,12 +1014,15 @@ The on-demand DAST scan runs, and the project's dashboard shows the results. #### Schedule an on-demand scan -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per user, -ask an administrator to [disable the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md). -The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature, ask an +administrator to [disable the feature flag](../../../administration/feature_flags.md) named +`dast_on_demand_scans_scheduler`. +On GitLab.com, this feature is available. To schedule a scan: @@ -1185,11 +1200,9 @@ If a validated site profile's target URL is edited, the site's validation status #### Retry a failed validation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an -administrator to [disable the `dast_failed_site_validations` flag](../../../administration/feature_flags.md). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322609) in GitLab 14.3. +> - [Deployed behind the `dast_failed_site_validations` flag](../../../administration/feature_flags.md), enabled by default. +> - [Feature flag `dast_failed_site_validations` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323961) in GitLab 14.4. If a site profile's validation fails, you can retry it by selecting the **Retry validation** button in the profiles list. @@ -1308,7 +1321,7 @@ If a scanner profile is linked to a security policy, a user cannot delete the pr page. See [Scan Execution Policies](../policies/index.md#scan-execution-policy-editor) for more information. -### Auditing +## Auditing > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217872) in GitLab 14.1. @@ -1320,13 +1333,6 @@ and DAST site profiles are included in the [audit log](../../../administration/a The DAST tool outputs a report file in JSON format by default. However, this tool can also generate reports in Markdown, HTML, and XML. For more information, see the [schema for DAST reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json). -### List of URLs scanned - -When DAST completes scanning, the merge request page states the number of URLs scanned. -Click **View details** to view the web console output which includes the list of scanned URLs. - - - ### JSON WARNING: diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 055a2ceb161..3b1c91b0be4 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -635,7 +635,7 @@ can be added, removed, and modified by creating a custom configuration. | `DAST_API_TARGET_URL` | Base URL of API testing target. | |[`DAST_API_CONFIG`](#configuration-files) | DAST API configuration file. Defaults to `.gitlab-dast-api.yml`. | |[`DAST_API_PROFILE`](#configuration-files) | Configuration profile to use during testing. Defaults to `Quick`. | -|[`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | +|[`DAST_API_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | |[`DAST_API_OPENAPI`](#openapi-specification) | OpenAPI specification file or URL. | |[`DAST_API_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | |[`DAST_API_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | @@ -899,7 +899,7 @@ variables: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. -When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the `FUZZAPI_EXCLUDE_PATHS` CI/CD variable . This variable is specified in your `.gitlab-ci.yml` file. To exclude multiple paths, separate entries using the `;` character. In the provided paths you can use a single character wildcard `?` and `*` for a multiple character wildcard. +When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the `DAST_API_EXCLUDE_PATHS` CI/CD variable . This variable is specified in your `.gitlab-ci.yml` file. To exclude multiple paths, separate entries using the `;` character. In the provided paths you can use a single character wildcard `?` and `*` for a multiple character wildcard. To verify the paths are excluded, review the `Tested Operations` and `Excluded Operations` portion of the job output. You should not see any excluded paths listed under `Tested Operations`. diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md index fae0f457a20..8559d5af02e 100644 --- a/doc/user/application_security/dependency_scanning/analyzers.md +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -47,7 +47,7 @@ In `.gitlab-ci.yml` define: ```yaml include: - template: Dependency-Scanning.gitlab-ci.yml + template: Security/Dependency-Scanning.gitlab-ci.yml variables: SECURE_ANALYZERS_PREFIX: my-docker-registry/gl-images @@ -64,7 +64,7 @@ In `.gitlab-ci.yml` define: ```yaml include: - template: Dependency-Scanning.gitlab-ci.yml + template: Security/Dependency-Scanning.gitlab-ci.yml variables: DS_EXCLUDED_ANALYZERS: "bundler-audit, gemnasium" @@ -77,7 +77,7 @@ In `.gitlab-ci.yml` define: ```yaml include: - template: Dependency-Scanning.gitlab-ci.yml + template: Security/Dependency-Scanning.gitlab-ci.yml variables: DS_EXCLUDED_ANALYZERS: "gemnasium, gemnasium-maven, gemnasium-python, bundler-audit, retire.js" diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index d903ce58982..1f840c96663 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -82,6 +82,7 @@ table.supported-languages tr td:last-child { } table.supported-languages ul { + font-size: 1em; list-style-type: none; padding-left: 0px; margin-bottom: 0px; @@ -346,7 +347,7 @@ Add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml ``` The included template creates dependency scanning jobs in your CI/CD @@ -380,7 +381,7 @@ For example: ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml variables: SECURE_LOG_LEVEL: error @@ -402,7 +403,7 @@ the `gemnasium` analyzer: ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml gemnasium-dependency_scanning: variables: @@ -413,7 +414,7 @@ To override the `dependencies: []` attribute, add an override job as above, targ ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml gemnasium-dependency_scanning: dependencies: ["build"] @@ -712,7 +713,7 @@ value of `GEMNASIUM_DB_REMOTE_URL` to the location of your offline Git copy of t ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml variables: SECURE_ANALYZERS_PREFIX: "docker-registry.example.com/analyzers" @@ -867,7 +868,7 @@ to the supported `requirements.txt` as follows. ```yaml include: - - template: Dependency-Scanning.gitlab-ci.yml + - template: Security/Dependency-Scanning.gitlab-ci.yml stages: - test @@ -926,3 +927,37 @@ gemnasium-maven-dependency_scanning: - for i in `ls cert*`; do keytool -v -importcert -alias "custom-cert-$i" -file $i -trustcacerts -noprompt -storepass changeit -keystore /opt/asdf/installs/java/adoptopenjdk-11.0.7+10.1/lib/security/cacerts 1>/dev/null 2>&1 || true; done # import each certificate using keytool (note the keystore location is related to the Java version being used and should be changed accordingly for other versions) - unset ADDITIONAL_CA_CERT_BUNDLE # unset the variable so that the analyzer doesn't duplicate the import ``` + +### Dependency Scanning job fails with message `strconv.ParseUint: parsing "0.0": invalid syntax` + +Invoking Docker-in-Docker is the likely cause of this error. Docker-in-Docker is: + +- Disabled by default in GitLab 13.0 and later. +- Unsupported from GitLab 13.4 and later. + +To fix this error, disable Docker-in-Docker for dependency scanning. Individual +`<analyzer-name>-dependency_scanning` jobs are created for each analyzer that runs in your CI/CD +pipeline. + +```yaml +include: + - template: Dependency-Scanning.gitlab-ci.yml + +variables: + DS_DISABLE_DIND: "true" +``` + +### Message `<file> does not exist in <commit SHA>` + +When the `Location` of a dependency in a file is shown, the path in the link goes to a specific Git +SHA. + +If the lock file that our dependency scanning tools reviewed was cached, however, selecting that +link redirects you to the repository root, with the message: +`<file> does not exist in <commit SHA>`. + +The lock file is cached during the build phase and passed to the dependency scanning job before the +scan occurs. Because the cache is downloaded before the analyzer run occurs, the existence of a lock +file in the `CI_BUILDS_DIR` directory triggers the dependency scanning job. + +We recommend committing the lock files, which prevents this warning. diff --git a/doc/user/application_security/img/cve_request_communication.png b/doc/user/application_security/img/cve_request_communication.png Binary files differdeleted file mode 100644 index 5c58df463ef..00000000000 --- a/doc/user/application_security/img/cve_request_communication.png +++ /dev/null diff --git a/doc/user/application_security/img/cve_request_communication_publication.png b/doc/user/application_security/img/cve_request_communication_publication.png Binary files differdeleted file mode 100644 index 9eb6f2f8d9f..00000000000 --- a/doc/user/application_security/img/cve_request_communication_publication.png +++ /dev/null diff --git a/doc/user/application_security/img/new_cve_request_issue.png b/doc/user/application_security/img/new_cve_request_issue.png Binary files differindex 6ea7ca4a2ab..5f9deeb7bd9 100644 --- a/doc/user/application_security/img/new_cve_request_issue.png +++ b/doc/user/application_security/img/new_cve_request_issue.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index bd143d8608a..e1ddb3167ff 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -1,18 +1,14 @@ --- stage: Protect group: Container Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Policies **(ULTIMATE)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5329) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. Deployed behind a feature flag, disabled by default. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 14.3. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the `security_orchestration_policies_configuration` flag](../../../administration/feature_flags.md). -On GitLab.com, this feature is available. +> - Introduced in GitLab Ultimate 13.10 [with a flag](https://gitlab.com/groups/gitlab-org/-/epics/5329) named `security_orchestration_policies_configuration`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in GitLab Ultimate 14.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in GitLab 14.4. Policies in GitLab provide security teams a way to require scans of their choice to be run whenever a project pipeline runs according to the configuration specified. Security teams can @@ -28,8 +24,8 @@ GitLab supports the following security policies: The Policies page displays deployed policies for all available environments. You can check a -policy's information (for example description, enforcement -status, etc.), and create and edit deployed policies: +policy's information (for example, description or enforcement +status), and create and edit deployed policies: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Policies**. @@ -53,7 +49,7 @@ users must make changes by following the ## Policy editor -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3403) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3403) in GitLab Ultimate 13.4. You can use the policy editor to create, edit, and delete policies: @@ -83,7 +79,7 @@ mode to fix your policy before Rule mode is available again. ## Container Network Policy -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in GitLab Ultimate 12.9. The **Container Network Policy** section provides packet flow metrics for your application's Kubernetes namespace. This section has the following @@ -158,7 +154,7 @@ at the bottom of the editor. ### Configure a Network Policy Alert -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.9. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab Ultimate 13.9. > - The feature flag was removed and the Threat Monitoring Alerts Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab 14.0. You can use policy alerts to track your policy's impact. Alerts are only available if you've @@ -255,6 +251,10 @@ The policy editor currently only supports the YAML mode. The Rule mode is tracke The YAML file with Scan Execution Policies consists of an array of objects matching Scan Execution Policy Schema nested under the `scan_execution_policy` key. You can configure a maximum of 5 policies under the `scan_execution_policy` key. +When you save a new policy, GitLab validates its contents against [this JSON schema](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/validators/json_schemas/security_orchestration_policy.json). +If you're not familiar with how to read [JSON schemas](https://json-schema.org/), +the following sections and tables provide an alternative. + | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| | `scan_execution_policy` | `array` of Scan Execution Policy | | List of scan execution policies (maximum 5) | @@ -291,6 +291,8 @@ This rule enforces the defined actions and schedules a scan on the provided date #### `cluster` schema +Use this schema to define `clusters` objects in the [`schedule` rule type](#schedule-rule-type). + | Field | Type | Possible values | Description | |--------------|---------------------|--------------------------|-------------| | `containers` | `array` of `string` | | The container name that will be scanned (only the first value is currently supported). | @@ -323,13 +325,16 @@ Note the following: - For a secret detection scan, only rules with the default ruleset are supported. [Custom rulesets](../secret_detection/index.md#custom-rulesets) are not supported. - A secret detection scan runs in `normal` mode when executed as part of a pipeline, and in - [`historic`](../secret_detection/index.md#full-history-secret-scan) + [`historic`](../secret_detection/index.md#full-history-secret-detection) mode when executed as part of a scheduled scan. - A container scanning and cluster image scanning scans configured for the `pipeline` rule type will ignore the cluster defined in the `clusters` object. They will use predefined CI/CD variables defined for your project. Cluster selection with the `clusters` object is supported for the `schedule` rule type. Cluster with name provided in `clusters` object must be created and configured for the project. To be able to successfully perform the `container_scanning`/`cluster_image_scanning` scans for the cluster you must follow instructions for the [Cluster Image Scanning feature](../cluster_image_scanning/index.md#prerequisites). -Here's an example: +### Example security policies project + +You can use this example in a `.gitlab/security-policies/policy.yml`, as described in +[Security policies project](#security-policies-project). ```yaml --- @@ -398,6 +403,24 @@ In this example: - Cluster Image Scanning scan runs every 24h. The scan runs on the `production-cluster` cluster and fetches vulnerabilities from the container with the name `database` configured for deployment with the name `production-application` in the `production-namespace` namespace. +### Example for scan execution policy editor + +You can use this example in the YAML mode of the [Scan Execution Policy editor](#scan-execution-policy-editor). +It corresponds to a single object from the previous example. + +```yaml +name: Enforce Secret Detection and Container Scanning in every default branch pipeline +description: This policy enforces pipeline configuration to have a job with Secret Detection and Container Scanning scans for the default branch +enabled: true +rules: + - type: pipeline + branches: + - main +actions: + - scan: secret_detection + - scan: container_scanning +``` + ## Roadmap See the [Category Direction page](https://about.gitlab.com/direction/protect/container_network_security/) diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 3caa1771a5b..7ffefd34e40 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -27,6 +27,8 @@ analysis are available in the [security dashboards](../security_dashboard/index. The results are sorted by the priority of the vulnerability: +<!-- vale gitlab.SubstitutionWarning = NO --> + 1. Critical 1. High 1. Medium @@ -34,6 +36,8 @@ The results are sorted by the priority of the vulnerability: 1. Info 1. Unknown +<!-- vale gitlab.SubstitutionWarning = YES --> + A pipeline consists of multiple jobs, including SAST and DAST scanning. If any job fails to finish for any reason, the security dashboard does not show SAST scanner output. For example, if the SAST job finishes but the DAST job fails, the security dashboard does not show SAST results. On failure, @@ -71,10 +75,11 @@ You can also [view our language roadmap](https://about.gitlab.com/direction/secu | .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | | .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | | Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | -| C | [Semgrep](https://semgrep.dev) | 14.2 | +| C | [Semgrep](https://semgrep.dev) | 14.2 | | C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | | Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | | Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Go | [Semgrep](https://semgrep.dev) | 14.4 | | Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | | Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | | Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | @@ -184,26 +189,60 @@ The results are saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest SAST artifact available. -### Configure SAST in the UI **(ULTIMATE)** +### Configure SAST in the UI + +You can enable and configure SAST in the UI, either with default settings, or with customizations. +Use the method that best meets your needs. + +- [Configure SAST in the UI with default settings](#configure-sast-in-the-ui-with-default-settings) +- [Configure SAST in the UI with customizations](#configure-sast-in-the-ui-with-customizations) + +### Configure SAST in the UI with default settings **(FREE)** + +> [Introduced](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#security-configuration-page-for-all-users) in GitLab 13.9 + +To enable and configure SAST with default settings: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance** > **Configuration**. +1. In the SAST section, select `Enable via MR`. +1. Review the draft MR that enables SAST with the default recommended settings in the + `.gitlab-ci.yml` file. +1. Merge the MR to enable SAST. You should see SAST jobs run in that MR's pipeline. + +NOTE: +The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal +configuration file. If you have a complex GitLab configuration file it may not be parsed +successfully, and an error may occur. + +### Configure SAST in the UI with customizations **(ULTIMATE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3659) in GitLab Ultimate 13.3. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in GitLab Ultimate 13.4. > - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/3635) in GitLab Ultimate 13.5. -You can enable and configure SAST with a basic configuration using the **SAST Configuration** -page: +To enable and configure SAST with customizations: -1. From the project's home page, go to **Security & Compliance** > **Configuration** in the - left sidebar. -1. If the project does not have a `.gitlab-ci.yml` file, click **Enable** in the Static Application Security Testing (SAST) row, otherwise click **Configure**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. If the project does not have a `.gitlab-ci.yml` file, select **Enable** in the Static Application + Security Testing (SAST) row, otherwise select **Configure**. 1. Enter the custom SAST values. - Custom values are stored in the `.gitlab-ci.yml` file. For CI/CD variables not in the SAST Configuration page, their values are left unchanged. Default values are inherited from the GitLab SAST template. + Custom values are stored in the `.gitlab-ci.yml` file. For CI/CD variables not in the SAST + Configuration page, their values are left unchanged. Default values are inherited from the GitLab + SAST template. -1. Optionally, expand the **SAST analyzers** section, select individual [SAST analyzers](analyzers.md) and enter custom analyzer values. -1. Click **Create Merge Request**. +1. Optionally, expand the **SAST analyzers** section, select individual + [SAST analyzers](analyzers.md) and enter custom analyzer values. +1. Select **Create Merge Request**. 1. Review and merge the merge request. +NOTE: +The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal +configuration file. If you have a complex GitLab configuration file it may not be parsed +successfully, and an error may occur. + ### Customizing the SAST settings The SAST settings can be changed through [CI/CD variables](#available-cicd-variables) @@ -250,12 +289,16 @@ versions are pulled, there are certain cases where it can be beneficial to pin an analyzer to a specific release. To do so, override the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable in the job template directly. -In the example below, we are pinning to a specific patch version of the `spotbugs` analyzer: +In the example below, we pin to a specific patch version of the `spotbugs` analyzer and minor version of the `semgrep` analyzer: ```yaml include: - template: Security/SAST.gitlab-ci.yml +semgrep-sast: + variables: + SAST_ANALYZER_IMAGE_TAG: "2.12" + spotbugs-sast: variables: SAST_ANALYZER_IMAGE_TAG: "2.28.1" @@ -361,9 +404,6 @@ To create a custom ruleset: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292686) in GitLab 14.2. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the `vulnerability_flags` flag](../../../administration/feature_flags.md). On GitLab.com, this feature is available. - Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. ### Using CI/CD variables to pass credentials for private repositories @@ -536,9 +576,11 @@ all [custom variables](../../../ci/variables/index.md#custom-cicd-variables) are to the underlying SAST analyzer images if [the SAST vendored template](#configuration) is used. -WARNING: -Variables having names starting with these prefixes are **not** propagated to the SAST Docker container and/or -analyzer containers: `DOCKER_`, `CI`, `GITLAB_`, `FF_`, `HOME`, `PWD`, `OLDPWD`, `PATH`, `SHLVL`, `HOSTNAME`. +NOTE: +In [GitLab 13.3 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/220540), +variables whose names started with the following prefixes are **not** propagated to either the +analyzer containers or SAST Docker container: `DOCKER_`, `CI`, `GITLAB_`, `FF_`, `HOME`, `PWD`, +`OLDPWD`, `PATH`, `SHLVL`, `HOSTNAME`. ### Experimental features @@ -807,3 +849,55 @@ This occurs when Flawfinder encounters an invalid UTF-8 character. To fix this, ### Semgrep slowness, unexpected results, or other errors If Semgrep is slow, reports too many false positives or false negatives, crashes, fails, or is otherwise broken, see the Semgrep docs for [troubleshooting GitLab SAST](https://semgrep.dev/docs/troubleshooting/gitlab-sast/). + +### SAST job fails with message `strconv.ParseUint: parsing "0.0": invalid syntax` + +Invoking Docker-in-Docker is the likely cause of this error. Docker-in-Docker is: + +- Disabled by default in GitLab 13.0 and later. +- Unsupported from GitLab 13.4 and later. + +Several workarounds are available. From GitLab version 13.0 and later, you must not use +Docker-in-Docker. + +#### Workaround 1: Pin analyzer versions (GitLab 12.1 and earlier) + +Set the following variables for the SAST job. This pins the analyzer versions to the last known +working version, allowing SAST with Docker-in-Docker to complete as it did previously: + +```yaml +sast: + variables: + SAST_DEFAULT_ANALYZERS: "" + SAST_ANALYZER_IMAGES: "registry.gitlab.com/gitlab-org/security-products/analyzers/bandit:2.9.6, registry.gitlab.com/gitlab-org/security-products/analyzers/brakeman:2.11.0, registry.gitlab.com/gitlab-org/security-products/analyzers/eslint:2.10.0, registry.gitlab.com/gitlab-org/security-products/analyzers/flawfinder:2.11.1, registry.gitlab.com/gitlab-org/security-products/analyzers/gosec:2.14.0, registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan:2.11.0, registry.gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit:2.9.1, registry.gitlab.com/gitlab-org/security-products/analyzers/pmd-apex:2.9.0, registry.gitlab.com/gitlab-org/security-products/analyzers/secrets:3.12.0, registry.gitlab.com/gitlab-org/security-products/analyzers/security-code-scan:2.13.0, registry.gitlab.com/gitlab-org/security-products/analyzers/sobelow:2.8.0, registry.gitlab.com/gitlab-org/security-products/analyzers/spotbugs:2.13.6, registry.gitlab.com/gitlab-org/security-products/analyzers/tslint:2.4.0" +``` + +Remove any analyzers you don't need from the `SAST_ANALYZER_IMAGES` list. Keep +`SAST_DEFAULT_ANALYZERS` set to an empty string `""`. + +#### Workaround 2: Disable Docker-in-Docker for SAST and Dependency Scanning (GitLab 12.3 and later) + +Disable Docker-in-Docker for SAST. Individual `<analyzer-name>-sast` jobs are created for each +analyzer that runs in your CI/CD pipeline. + +```yaml +include: + - template: SAST.gitlab-ci.yml + +variables: + SAST_DISABLE_DIND: "true" +``` + +#### Workaround 3: Upgrade to GitLab 13.x and use the defaults + +From GitLab 13.0, SAST defaults to not using Docker-in-Docker. In GitLab 13.4 and later, SAST using +Docker-in-Docker is [no longer supported](https://gitlab.com/gitlab-org/gitlab/-/issues/220540). +If you have this problem on GitLab 13.x and later, you have customized your SAST job to +use Docker-in-Docker. To resolve this, comment out any customizations you've made to +your SAST CI job definition and [follow the documentation](index.md#configuration) +to reconfigure, using the new and improved job definition default values. + +```yaml +include: + - template: Security/SAST.gitlab-ci.yml +``` diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index cd1014d36a6..5933496ea00 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -138,7 +138,7 @@ The results are saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. -### Enable Secret Detection via an automatic merge request **(ULTIMATE SELF)** +### Enable Secret Detection via an automatic merge request **(FREE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4496) in GitLab 13.11, behind a feature flag, enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/329886) in GitLab 14.1. @@ -151,7 +151,12 @@ from the Security Configuration page. 1. In the **Secret Detection** row, select **Configure via Merge Request**. This automatically creates a merge request with the changes necessary to enable Secret Detection -that you can review and merge to complete the configuration. +that you can review and merge to complete the configuration. + +NOTE: +The configuration tool works best with no existing `.gitlab-ci.yml` file, or with a minimal +configuration file. If you have a complex GitLab configuration file it may not be parsed +successfully, and an error may occur. ### Customizing settings @@ -167,12 +172,12 @@ WARNING: Beginning in GitLab 13.0, the use of [`only` and `except`](../../../ci/yaml/index.md#only--except) is no longer supported. When overriding the template, you must use [`rules`](../../../ci/yaml/index.md#rules) instead. -#### GIT_DEPTH +#### `GIT_DEPTH` variable The [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning) affects Secret Detection. The Secret Detection analyzer relies on generating patches between commits to scan content for secrets. If you override the default, ensure the value is greater than 1. If the number of commits -in an MR is greater than the GIT_DEPTH value, Secret Detection will [fail to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). +in an MR is greater than the `GIT_DEPTH` value, Secret Detection will [fail to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). #### Custom settings example @@ -285,20 +290,20 @@ sequenceDiagram Cloud Vendor-->>+RevocationAPI: ACCEPTED ``` -## Full History Secret Scan +## Full History Secret Detection GitLab 12.11 introduced support for scanning the full history of a repository. This new functionality is particularly useful when you are enabling Secret Detection in a repository for the first time and you -want to perform a full secret scan. Running a secret scan on the full history can take a long time, +want to perform a full secret detection scan. Running a secret detection scan on the full history can take a long time, especially for larger repositories with lengthy Git histories. We recommend not setting this CI/CD variable as part of your normal job definition. A new configuration variable ([`SECRET_DETECTION_HISTORIC_SCAN`](#available-cicd-variables)) can be set to change the behavior of the GitLab Secret Detection scan to run on the entire Git history of a repository. -We have created a [short video walkthrough](https://youtu.be/wDtc_K00Y0A) showcasing how you can perform a full history secret scan. +We have created a [short video walkthrough](https://youtu.be/wDtc_K00Y0A) showcasing how you can perform a full history secret detection scan. <div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=wDtc_K00Y0A">Walkthrough of historical secret scan</a>. + See the video: <a href="https://www.youtube.com/watch?v=wDtc_K00Y0A">Walkthrough of historical secret detection scan</a>. </div> <figure class="video-container"> <iframe src="https://www.youtube.com/embed/wDtc_K00Y0A" frameborder="0" allowfullscreen="true"> </iframe> diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index 8b811c62ec3..f5b0194c320 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -45,6 +45,8 @@ From the Vulnerability Report you can: You can filter the vulnerabilities table by: +<!-- vale gitlab.SubstitutionWarning = NO --> + | Filter | Available options | |:---------|:------------------| | Status | Detected, Confirmed, Dismissed, Resolved. | @@ -53,6 +55,8 @@ You can filter the vulnerabilities table by: | Project | For more details, see [Project filter](#project-filter). | | Activity | For more details, see [Activity filter](#activity-filter). | +<!-- vale gitlab.SubstitutionWarning = YES --> + ### Filter the list of vulnerabilities To filter the list of vulnerabilities: diff --git a/doc/user/asciidoc.md b/doc/user/asciidoc.md index 8313b20e795..cd166666ad6 100644 --- a/doc/user/asciidoc.md +++ b/doc/user/asciidoc.md @@ -1,8 +1,7 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, howto +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # AsciiDoc **(FREE)** @@ -85,7 +84,7 @@ I believe I shall--no, actually I won't. **Macros** ```plaintext -// where c=specialchars, q=quotes, a=attributes, r=replacements, m=macros, p=post_replacements, etc. +// where c=specialchars, q=quotes, a=attributes, r=replacements, m=macros, p=post_replacements The European icon:flag[role=blue] is blue & contains pass:[************] arranged in a icon:circle-o[role=yellow]. The pass:c[->] operator is often referred to as the stabby lambda. Since `pass:[++]` has strong priority in AsciiDoc, you can rewrite pass:c,a,r[C++ => C{pp}]. @@ -151,7 +150,7 @@ This paragraph has a footnote.footnote:[This is the text of the footnote.] ** level 2 *** level 3 **** level 4 -***** etc. +***** level 5 * back at level 1 + Attach a block or paragraph to a list item using a list continuation (which you can enclose in an open block). @@ -240,10 +239,10 @@ include::basics.adoc[] include::https://example.org/installation.adoc[] ``` -To guarantee good system performance and prevent malicious documents causing -problems, GitLab enforces a **maximum limit** on the number of include directives -processed in any one document. Currently a total of 32 documents can be -included, a number that is inclusive of transitive dependencies. +To guarantee good system performance and prevent malicious documents from causing +problems, GitLab enforces a maximum limit on the number of include directives +processed in any one document. You can include up to 32 documents, which is +inclusive of transitive dependencies. ### Blocks @@ -428,7 +427,7 @@ If you're new to using Mermaid or need help identifying issues in your Mermaid c the [Mermaid Live Editor](https://mermaid-js.github.io/mermaid-live-editor/) is a helpful tool for creating and resolving issues within Mermaid diagrams. -In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block: +To generate a diagram or flowchart, enter your text in a `mermaid` block: ```plaintext [mermaid] @@ -447,7 +446,7 @@ Kroki supports more than a dozen diagram libraries. To make Kroki available in GitLab, a GitLab administrator needs to enable it first. Read more in the [Kroki integration](../administration/integration/kroki.md) page. -Once Kroki is enabled, you can create a wide variety of diagrams in AsciiDoc and Markdown documents. +After Kroki is enabled, you can create diagrams in AsciiDoc and Markdown documents. Here's an example using a GraphViz diagram: **AsciiDoc** @@ -476,7 +475,7 @@ digraph G { To make PlantUML available in GitLab, a GitLab administrator needs to enable it first. Read more in [PlantUML & GitLab](../administration/integration/plantuml.md). -Once enabled, you should write your text inside the `plantuml` block: +After PlantUML is enabled, enter your text in a `plantuml` block: ```plaintext [plantuml] diff --git a/doc/user/award_emojis.md b/doc/user/award_emojis.md index de250bc5fb4..88651688779 100644 --- a/doc/user/award_emojis.md +++ b/doc/user/award_emojis.md @@ -6,11 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Award emoji **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/1825) in GitLab 8.2. -> - GitLab 9.0 [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/9570) the usage of native emoji if the platform -> supports them and falls back to images or CSS sprites. This change greatly -> improved award emoji performance overall. - When you're collaborating online, you get fewer opportunities for high-fives and thumbs-ups. Emoji can be awarded to [issues](project/issues/index.md), [merge requests](project/merge_requests/index.md), [snippets](snippets.md), and anywhere you can have a thread. @@ -24,8 +19,6 @@ For information on the relevant API, see [Award Emoji API](../api/award_emoji.md ## Sort issues and merge requests on vote count -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/2781) in GitLab 8.5. - You can quickly sort issues and merge requests by the number of votes they have received. The sort options can be found in the dropdown menu as "Most popular" and "Least popular". @@ -38,8 +31,6 @@ downvotes. ## Award emoji for comments -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4291) in GitLab 8.9. - Award emoji can also be applied to individual comments when you want to celebrate an accomplishment or agree with an opinion. diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md index 1ea5168f30c..6c8b7c95771 100644 --- a/doc/user/clusters/agent/ci_cd_tunnel.md +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -10,6 +10,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - The pre-configured `KUBECONFIG` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324275) in GitLab 14.2. > - The ability to authorize groups was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +WARNING: +The CI/CD Tunnel is not supported for GitLab self-managed instances installed via Omnibus. We +plan to [add support for Omnibus](https://gitlab.com/gitlab-org/gitlab/-/issues/324272) in the future. + The CI/CD Tunnel enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster. GitLab Runner does not have to be running in the same cluster. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index d2dc57c0849..557d389147d 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -15,7 +15,7 @@ The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/ is an active in-cluster component for solving GitLab and Kubernetes integration tasks in a secure and cloud-native way. It enables: -- Integrating GitLab with a Kubernetes cluster behind a firewall or NAT +- GitLab integration with a Kubernetes cluster behind a firewall or NAT (network address translation). - Pull-based GitOps deployments. - [Inventory object](../../infrastructure/clusters/deploy/inventory_object.md) to keep track of objects applied to your cluster. @@ -28,7 +28,7 @@ and [our development documentation](https://gitlab.com/gitlab-org/cluster-integr ## GitLab Agent GitOps workflow -The GitLab Agent uses multiple GitLab projects to provide a flexible workflow +The GitLab Agent, herein _Agent_, uses multiple GitLab projects to provide a flexible workflow that can suit various needs. This diagram shows these repositories and the main actors involved in a deployment: @@ -54,10 +54,10 @@ There are several components that work in concert for the Agent to accomplish Gi - A properly-configured Kubernetes cluster where the Agent is running. - A configuration repository that contains a `config.yaml` file, which tells the - Agent which repositories to synchronize with the cluster. + Agent the repositories to synchronize with the cluster. - A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster. -You can use the same GitLab project or separate projects for configuration and manifest files, as follows: +You can use the same GitLab project or projects for configuration and manifest files, as follows: - Single GitLab project (recommended): when you use a single repository to hold both the manifest and the configuration files, these projects can be either private or public, as you prefer. - Two GitLab projects: when you opt to use two different GitLab projects, one for manifest files, and another for configuration files, the manifests project must be public, while the configuration project can be either private or public. Our backlog contains issues for adding support for @@ -73,8 +73,8 @@ The setup process involves a few steps to enable GitOps deployments: 1. [Set up the Kubernetes Agent Server](#set-up-the-kubernetes-agent-server) for your GitLab instance. 1. [Define a configuration repository](#define-a-configuration-repository). 1. [Create an Agent record in GitLab](#create-an-agent-record-in-gitlab). -1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). 1. [Install the Agent into the cluster](#install-the-agent-into-the-cluster). +1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). 1. [Create manifest files](#create-manifest-files). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a GitLab 14.2 [walking-through video](https://www.youtube.com/watch?v=XuBpKtsgGkE) with this process. @@ -200,7 +200,7 @@ For more advanced configurations, we recommend to use [the `kpt` based installat Otherwise, follow the manual installation steps described below. -##### Create the Kubernetes secret +### Create the Kubernetes secret After generating the token, you must apply it to the Kubernetes cluster. @@ -256,7 +256,7 @@ NAMESPACE NAME READY S gitlab-kubernetes-agent gitlab-kubernetes-agent-77689f7dcb-5skqk 1/1 Running 0 51s ``` -##### Example `resources.yml` file +#### Example `resources.yml` file ```yaml --- @@ -403,7 +403,7 @@ The following example projects can help you get started with the Kubernetes Agen - [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) - This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) -### Deploying GitLab Runner with the Agent +### GitLab Runner Deployment with the Agent You can use the Kubernetes Agent to [deploy GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). @@ -444,6 +444,41 @@ the current project, and the configuration directory for each agent: Additional management interfaces are planned for the GitLab Kubernetes Agent. [Provide more feedback in the related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). +## Remove the GitLab Kubernetes Agent + +1. Remove an Agent record with GraphQL by deleting the `clusterAgent` and the `clusterAgentToken`. + + ```graphql + mutation deleteAgent { + clusterAgentDelete(input: { id: "<cluster-agent-id>" } ) { + errors + } + } + + mutation deleteToken { + clusterAgentTokenDelete(input: { id: "<cluster-agent-token-id>" }) { + errors + } + } + ``` + +1. Verify whether the removal occurred successfully. If the output in the Pod logs includes `unauthenticated`, it means that the agent was successfully removed: + + ```json + { + "level": "warn", + "time": "2021-04-29T23:44:07.598Z", + "msg": "GetConfiguration.Recv failed", + "error": "rpc error: code = Unauthenticated desc = unauthenticated" + } + ``` + +1. Delete the GitLab Kubernetes Agent in your cluster: + + ```shell + kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml + ``` + ## Troubleshooting If you face any issues while using GitLab Kubernetes Agent, you can read the @@ -455,10 +490,17 @@ kubectl logs -f -l=app=gitlab-kubernetes-agent -n gitlab-kubernetes-agent GitLab administrators can additionally view the [Kubernetes Agent Server logs](../../../administration/clusters/kas.md#troubleshooting). -### Agent logs - Transport: Error while dialing failed to WebSocket dial +### Agent logs + +#### Transport: Error while dialing failed to WebSocket dial ```json -{"level":"warn","time":"2020-11-04T10:14:39.368Z","msg":"GetConfiguration failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\""} +{ + "level": "warn", + "time": "2020-11-04T10:14:39.368Z", + "msg": "GetConfiguration failed", + "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab-kas:443/-/kubernetes-agent\\\": dial tcp: lookup gitlab-kas on 10.60.0.10:53: no such host\"" +} ``` This error is shown if there are some connectivity issues between the address @@ -466,27 +508,45 @@ specified as `kas-address`, and your Agent pod. To fix it, make sure that you specified the `kas-address` correctly. ```json -{"level":"error","time":"2021-06-25T21:15:45.335Z","msg":"Reverse tunnel","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\""} +{ + "level": "error", + "time": "2021-06-25T21:15:45.335Z", + "msg": "Reverse tunnel", + "mod_name": "reverse_tunnel", + "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc= \"transport: Error while dialing failed to WebSocket dial: expected handshake response status code 101 but got 301\"" +} ``` This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the `wss` or `ws` URL ends with a training slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. -### Agent logs - ValidationError(Deployment.metadata) +#### ValidationError(Deployment.metadata) ```json -{"level":"info","time":"2020-10-30T08:56:54.329Z","msg":"Synced","project_id":"root/kas-manifest001","resource_key":"apps/Deployment/kas-test001/nginx-deployment","sync_result":"error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]"} +{ + "level": "info", + "time": "2020-10-30T08:56:54.329Z", + "msg": "Synced", + "project_id": "root/kas-manifest001", + "resource_key": "apps/Deployment/kas-test001/nginx-deployment", + "sync_result": "error validating data: [ValidationError(Deployment.metadata): unknown field \"replicas\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"selector\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, ValidationError(Deployment.metadata): unknown field \"template\" in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta]" +} ``` This error is shown if a manifest file is malformed, and Kubernetes can't create specified objects. Make sure that your manifest files are valid. You may try using them to create objects in Kubernetes directly for more troubleshooting. -### Agent logs - Error while dialing failed to WebSocket dial: failed to send handshake request +#### Error while dialing failed to WebSocket dial: failed to send handshake request ```json -{"level":"warn","time":"2020-10-30T09:50:51.173Z","msg":"GetConfiguration failed","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\""} +{ + "level": "warn", + "time": "2020-10-30T09:50:51.173Z", + "msg": "GetConfiguration failed", + "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent\\\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \\\"\\\\x00\\\\x00\\\\x06\\\\x04\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x00\\\\x05\\\\x00\\\\x00@\\\\x00\\\"\"" +} ``` This error is shown if you configured `wss` as `kas-address` on the agent side, @@ -499,19 +559,30 @@ issue is in progress, directly edit the deployment with the `kubectl edit deployment gitlab-kas` command, and change `--listen-websocket=true` to `--listen-websocket=false`. After running that command, you should be able to use `grpc://gitlab-kas.<YOUR-NAMESPACE>:8150`. -### Agent logs - Decompressor is not installed for grpc-encoding +#### Decompressor is not installed for grpc-encoding ```json -{"level":"warn","time":"2020-11-05T05:25:46.916Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\""} +{ + "level": "warn", + "time": "2020-11-05T05:25:46.916Z", + "msg": "GetConfiguration.Recv failed", + "error": "rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"gzip\"" +} ``` This error is shown if the version of the agent is newer that the version of KAS. To fix it, make sure that both `agentk` and KAS use the same versions. -### Agent logs - Certificate signed by unknown authority +#### Certificate signed by unknown authority ```json -{"level":"error","time":"2021-02-25T07:22:37.158Z","msg":"Reverse tunnel","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\""} +{ + "level": "error", + "time": "2021-02-25T07:22:37.158Z", + "msg": "Reverse tunnel", + "mod_name": "reverse_tunnel", + "error": "Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://GitLabhost.tld:443/-/kubernetes-agent/\\\": x509: certificate signed by unknown authority\"" +} ``` This error is shown if your GitLab instance is using a certificate signed by an internal CA that @@ -580,34 +651,3 @@ Alternatively, you can mount the certificate file at a different location and in mountPath: /tmp/myCA.pem subPath: myCA.pem ``` - -## Remove the GitLab Kubernetes Agent - -1. Remove an Agent record with GraphQL by deleting the `clusterAgent` and the `clusterAgentToken`. - - ```graphql - mutation deleteAgent { - clusterAgentDelete(input: { id: "<cluster-agent-id>" } ) { - errors - } - } - - mutation deleteToken { - clusterAgentTokenDelete(input: { id: "<cluster-agent-token-id>" }) { - errors - } - } - ``` - -1. Verify whether the removal occurred successfully. If the output in the Pod logs includes `unauthenticated`, it means that the agent was successfully removed: - - ```json - {"level":"warn","time":"2021-04-29T23:44:07.598Z","msg":"GetConfiguration.Recv failed","error":"rpc error: - code = Unauthenticated desc = unauthenticated"} - ``` - -1. Delete the GitLab Kubernetes Agent in your cluster: - - ```shell - kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml - ``` diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index ea57ded3320..cbb27a3f343 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -152,6 +152,11 @@ gitops: > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the `group_authorized_agents` flag](../../../administration/feature_flags.md). +On GitLab.com, this feature is available. + If you use the same cluster across multiple projects, you can set up the CI/CD Tunnel to grant the Agent access to one or more groups. This way, all the projects that belong to the authorized groups can access the same Agent. This enables you to save resources and @@ -168,8 +173,8 @@ An Agent can only authorize groups in the same group hierarchy as the Agent's co To authorize a group: -1. Edit your `.config.yaml` file under the `.gitlab/agents/<agent name>` directory. -1. Add the `ci_access` attribute. +1. Edit your `config.yaml` file under the `.gitlab/agents/<agent name>` directory. +1. Add the `ci_access` root attribute. 1. Add the `groups` attribute into `ci_access`. 1. Add the group `id` into `groups`, identifying the authorized group through its path. diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 2da8396cfd7..31dd503a0cf 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -485,7 +485,7 @@ config: agent: monitor: - eventTypes: ["drop"] + eventTypes: ["drop"] # Note: possible values are documented at https://docs.cilium.io/en/stable/cmdref/cilium_monitor/ ``` The Cilium monitor log for traffic is logged out by the diff --git a/doc/user/clusters/environments.md b/doc/user/clusters/environments.md index cad55f0cf0b..470f65db61b 100644 --- a/doc/user/clusters/environments.md +++ b/doc/user/clusters/environments.md @@ -33,7 +33,7 @@ owners](../permissions.md#group-members-permissions) In order to: - Track environments for the cluster, you must - [deploy to a Kubernetes cluster](../project/clusters/index.md#deploying-to-a-kubernetes-cluster) + [deploy to a Kubernetes cluster](../project/clusters/deploy_to_cluster.md) successfully. - Show pod usage correctly, you must [enable deploy boards](../project/deploy_boards.md#enabling-deploy-boards). diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index 9e2b00a0f54..305f66c5ec5 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w With a [cluster management project](management_project.md) you can manage your cluster's deployment and applications through a repository in GitLab. -The Custer Management project template provides you a baseline to get +The Cluster Management project template provides you a baseline to get started and flexibility to customize your project to your cluster's needs. For instance, you can: @@ -91,10 +91,6 @@ the pipeline runs, Helmfile tries to either install or update your apps accordin cluster and Helm releases. If you change this attribute to `installed: false`, Helmfile tries try to uninstall this app from your cluster. [Read more](https://github.com/roboll/helmfile) about how Helmfile works. -Furthermore, each app has an `applications/{app}/values.yaml` file (`applicaton/{app}/values.yaml.gotmpl` in case of GitLab Runner). This is the -place where you can define default values for your app's Helm chart. Some apps already have defaults -pre-defined by GitLab. - ### Built-in applications The [built-in supported applications](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/tree/master/applications) are: @@ -110,3 +106,9 @@ The [built-in supported applications](https://gitlab.com/gitlab-org/project-temp - [Prometheus](../infrastructure/clusters/manage/management_project_applications/prometheus.md) - [Sentry](../infrastructure/clusters/manage/management_project_applications/sentry.md) - [Vault](../infrastructure/clusters/manage/management_project_applications/vault.md) + +#### How to customize your applications + +Each app has an `applications/{app}/values.yaml` file (`applicaton/{app}/values.yaml.gotmpl` in case of GitLab Runner). This is the +place where you can define default values for your app's Helm chart. Some apps already have defaults +pre-defined by GitLab. diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index d30cedfb3cd..63dd06bcfdb 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -30,7 +30,7 @@ Prerequisites: To view the compliance report: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance**. +1. On the left sidebar, select **Security & Compliance > Compliance report**. NOTE: The compliance report shows only the latest merge request on each project. @@ -87,7 +87,7 @@ Depending on the merge strategy, the merge commit SHA can be a merge commit, squ To download the Chain of Custody report: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance**. +1. On the left sidebar, select **Security & Compliance > Compliance report**. 1. Select **List of all merge commits**. ### Commit-specific Chain of Custody Report **(ULTIMATE)** @@ -97,7 +97,7 @@ To download the Chain of Custody report: You can generate a commit-specific Chain of Custody report for a given commit SHA. 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Compliance**. +1. On the left sidebar, select **Security & Compliance > Compliance report**. 1. At the top of the compliance report, to the right of **List of all merge commits**, select the down arrow (**{angle-down}**). 1. Enter the merge commit SHA, and then select **Export commit custody report**. SHA and then select **Export commit custody report**. diff --git a/doc/user/compliance/license_compliance/img/policies_maintainer_add_v13_2.png b/doc/user/compliance/license_compliance/img/policies_maintainer_add_v13_2.png Binary files differdeleted file mode 100644 index 2d5946503cf..00000000000 --- a/doc/user/compliance/license_compliance/img/policies_maintainer_add_v13_2.png +++ /dev/null diff --git a/doc/user/compliance/license_compliance/img/policies_maintainer_add_v14_3.png b/doc/user/compliance/license_compliance/img/policies_maintainer_add_v14_3.png Binary files differnew file mode 100644 index 00000000000..7a27899f8c9 --- /dev/null +++ b/doc/user/compliance/license_compliance/img/policies_maintainer_add_v14_3.png diff --git a/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_2.png b/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_2.png Binary files differdeleted file mode 100644 index 2ad08919f86..00000000000 --- a/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_2.png +++ /dev/null diff --git a/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png b/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png Binary files differnew file mode 100644 index 00000000000..85b2a52e04b --- /dev/null +++ b/doc/user/compliance/license_compliance/img/policies_maintainer_edit_v14_3.png diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 165150a58a1..5318f4deed1 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -47,7 +47,7 @@ When GitLab detects a **Denied** license, you can view it in the [license list](  You can view and modify existing policies from the [policies](#policies) tab. - + ## License expressions @@ -742,8 +742,9 @@ which enables a designated approver that can approve and then merge a merge requ The **Policies** tab in the project's license compliance section displays your project's license policies. Project maintainers can specify policies in this section. - - + + + Developers of the project can view the policies configured in a project. @@ -753,6 +754,10 @@ Developers of the project can view the policies configured in a project. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13067) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.3. +Prerequisites: + +- Maintainer or Owner [role](../../permissions.md#project-members-permissions). + `License-Check` is a [merge request approval](../../project/merge_requests/approvals/index.md) rule you can enable to allow an individual or group to approve a merge request that contains a `denied` license. diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 23765de8f46..e0cc79fe0fb 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -139,7 +139,7 @@ the related documentation. | [Max pipelines per schedule](../../administration/instance_limits.md#limit-the-number-of-pipelines-created-by-a-pipeline-schedule-per-day) | `24` for Free tier, `288` for all paid tiers | Unlimited | | [Scheduled Job Archival](../../user/admin_area/settings/continuous_integration.md#archive-jobs) | 3 months | Never | | Max test cases per [unit test report](../../ci/unit_test_reports.md) | `500_000` | Unlimited | -| [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | `50` per-project and per-group for Free tier,<br/>`1_000` per-group for all paid tiers / `1_000` per-project for all paid tiers | `1_000` per-group / `1_000` per-project | +| [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | Free tier: `50` per-group / `50` per-project <br/> All paid tiers: `1_000` per-group / `1_000` per-project | `1_000` per-group / `1_000` per-project | ## Account and limit settings @@ -270,7 +270,7 @@ for `shared_buffers` is quite high, and we are ## Puma -GitLab.com uses the default of 60 seconds for [Puma request timeouts](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). +GitLab.com uses the default of 60 seconds for [Puma request timeouts](../../administration/operations/puma.md#worker-timeout). ## GitLab.com-specific rate limits diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 0d885183a41..25eff076d8d 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -14,8 +14,10 @@ Similar to [project-level](../../project/clusters/index.md) and group-level Kubernetes clusters allow you to connect a Kubernetes cluster to your group, enabling you to use the same cluster across multiple projects. -To view your group level Kubernetes clusters, navigate to your project and select -**Kubernetes** from the left-hand menu. +To view your group-level Kubernetes clusters: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Kubernetes**. ## Cluster management project @@ -58,7 +60,7 @@ differentiate the new cluster from your other clusters. You can choose to allow GitLab to manage your cluster for you. If GitLab manages your cluster, resources for your projects are automatically created. See the -[Access controls](../../project/clusters/add_remove_clusters.md#access-controls) +[Access controls](../../project/clusters/cluster_access.md) section for details on which resources GitLab creates for you. For clusters not managed by GitLab, project-specific resources aren't created @@ -82,10 +84,11 @@ your cluster, which can cause deployment jobs to fail. To clear the cache: -1. Navigate to your group's **Kubernetes** page, - and select your cluster. -1. Expand the **Advanced settings** section. -1. Click **Clear cluster cache**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Kubernetes**. +1. Select your cluster. +1. Expand **Advanced settings**. +1. Select **Clear cluster cache**. ## Base domain @@ -169,7 +172,7 @@ documentation for project-level clusters. ## More information For information on integrating GitLab and Kubernetes, see -[Kubernetes clusters](../../project/clusters/index.md). +[Kubernetes clusters](../../infrastructure/clusters/index.md). <!-- ## Troubleshooting diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index a2d7f358cd9..b13dd3f63cb 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -11,15 +11,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w With Contribution Analytics you can get an overview of the [contribution actions](../../../api/events.md#action-types) in your group. -To view the Contribution Analytics, go to your group and select **Analytics > Contribution**. - -## Use cases - - Analyze your team's contributions over a period of time, and offer a bonus for the top contributors. - Identify opportunities for improvement with group members who may benefit from additional support. +## View Contribution Analytics + +To view Contribution Analytics: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Contribution**. + ## Using Contribution Analytics There are three main bar graphs that illustrate the number of contributions per group diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 41046477b90..a9c56139b4d 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -17,6 +17,12 @@ in the group and select the **Group** tab. Every project in the subgroup, but not nested subgroups, can be selected by members of the group when a new project is created. +- Public projects can be selected by any signed-in user as a template for a new project, + if all enabled [project features](../project/settings/index.md#sharing-and-permissions) + except for **GitLab Pages** and **Security & Compliance** are set to **Everyone With Access**. + The same applies to internal projects. +- Private projects can be selected only by users who are members of the projects. + Repository and database information that is copied over to each new project is identical to the data exported with the [GitLab Project Import/Export](../project/settings/import_export.md). diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 554d01039ad..e3b858aff7d 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -14,6 +14,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Dependency Scanning metrics [added](https://gitlab.com/gitlab-org/gitlab/-/issues/328034) in GitLab 14.2. > - Multiselect [added](https://gitlab.com/gitlab-org/gitlab/-/issues/333586) in GitLab 14.2. > - Overview table [added](https://gitlab.com/gitlab-org/gitlab/-/issues/335638) in GitLab 14.3. +> - Adoption over time chart [added](https://gitlab.com/gitlab-org/gitlab/-/issues/337561) in GitLab 14.4. Prerequisites: @@ -69,6 +70,13 @@ Each group appears as a separate row in the table. For each row, a feature is considered "adopted" if it has been used in a project in the given group during the time period (including projects in any subgroups of the given group). +## Adoption over time + +The **Adoption over time** chart in the **Overview** tab displays DevOps Adoption over time. The chart displays the total number of adopted features from the previous twelve months, +from when you enabled DevOps Adoption for the group. + +The tooltip displays information about the features tracked for individual months. + ## When is a feature considered adopted A feature is considered "adopted" if it has been used anywhere in the group in the specified time. diff --git a/doc/user/group/epics/img/epic_view_roadmap_v12_9.png b/doc/user/group/epics/img/epic_view_roadmap_v12_9.png Binary files differindex e8224ced7e9..3a33a6c4cf8 100644 --- a/doc/user/group/epics/img/epic_view_roadmap_v12_9.png +++ b/doc/user/group/epics/img/epic_view_roadmap_v12_9.png diff --git a/doc/user/group/import/img/import_panel_v14_1.png b/doc/user/group/import/img/import_panel_v14_1.png Binary files differindex 28417383b6c..52791a82c3c 100644 --- a/doc/user/group/import/img/import_panel_v14_1.png +++ b/doc/user/group/import/img/import_panel_v14_1.png diff --git a/doc/user/group/import/img/new_group_navigation_v13_8.png b/doc/user/group/import/img/new_group_navigation_v13_8.png Binary files differindex 307175727c7..40be3dd41d2 100644 --- a/doc/user/group/import/img/new_group_navigation_v13_8.png +++ b/doc/user/group/import/img/new_group_navigation_v13_8.png diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 31c3a8e774e..1f5de36303e 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import groups from another instance of GitLab **(FREE)** +# Migrate groups from another instance of GitLab **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7. > - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. @@ -102,8 +102,10 @@ This might involve reconfiguring your firewall to prevent blocking connection on ### Connect to the remote GitLab instance -1. Navigate to the New Group page, either via the `+` button in the top navigation bar, or the **New subgroup** button -on an existing group's page. +1. Go to the New Group page: + + - On the top bar, select `+` and then **New group**. + - Or, on an existing group's page, in the top right, select **New subgroup**.  @@ -111,21 +113,20 @@ on an existing group's page.  -1. Fill in source URL of your GitLab. -1. Fill in [personal access token](../../../user/profile/personal_access_tokens.md) for remote GitLab instance. -1. Click "Connect instance". +1. Enter the source URL of your GitLab instance. +1. Generate or copy a [personal access token](../../../user/profile/personal_access_tokens.md) + with the `api` and `read_repository` scopes on your remote GitLab instance. +1. Enter the [personal access token](../../../user/profile/personal_access_tokens.md) for your remote GitLab instance. +1. Select **Connect instance**. ### Selecting which groups to import After you have authorized access to the GitLab instance, you are redirected to the GitLab Group -Migration importer page. Listed are the remote GitLab groups to which you have the Owner role. +Migration importer page. The remote groups you have the Owner role for are listed. 1. By default, the proposed group namespaces match the names as they exist in remote instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. - -1. Select the **Import** button next to any number of groups. - -1. The **Status** column shows the import status of each group. You can choose to leave the page open and it updates in real-time. - -1. Once a group has been imported, click its GitLab path to open its GitLab URL. +1. Next to the groups you want to import, select **Import**. +1. The **Status** column shows the import status of each group. If you leave the page open, it updates in real-time. +1. After a group has been imported, select its GitLab path to open its GitLab URL.  diff --git a/doc/user/group/index.md b/doc/user/group/index.md index caf874cfe28..15d2da50012 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -571,10 +571,11 @@ To restrict group access by IP address: ## Restrict group access by domain **(PREMIUM)** ->- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7297) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. ->- Support for specifying multiple email domains [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33143) added in GitLab 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7297) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. +> - Support for specifying multiple email domains [added](https://gitlab.com/gitlab-org/gitlab/-/issues/33143) in GitLab 13.1. +> - Support for restricting access to projects within the group [added](https://gitlab.com/gitlab-org/gitlab/-/issues/14004) in GitLab 14.1.2. -You can prevent users with email addresses in specific domains from being added to a group. +You can prevent users with email addresses in specific domains from being added to a group and its projects. To restrict group access by domain: @@ -593,9 +594,6 @@ Some domains cannot be restricted. These are the most popular public email domai - `hotmail.com`, `hotmail.co.uk`, `hotmail.fr` - `msn.com`, `live.com`, `outlook.com` -NOTE: -Domain restrictions apply to groups only. They do not prevent users from being added as members of projects owned by the restricted group. - ## Group file templates **(PREMIUM)** Use group file templates to share a set of templates for common file @@ -732,6 +730,27 @@ The group's new subgroups have push rules set for them based on either: - The closest parent group with push rules defined. - Push rules set at the instance level, if no parent groups have push rules defined. +## Group approval rules **(PREMIUM)** + +> Introduced in GitLab 13.9. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md), disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +per group, ask an administrator to [enable the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md). +The feature is not ready for production use. + +Group approval rules are an in-development feature that provides an interface for managing +[project merge request approval rules](../project/merge_requests/approvals/index.md) at the +top-level group level. When rules are configured [at the instance level](../admin_area/merge_requests_approvals.md), +you can't edit locked rules. + +To view the merge request approval rules UI for a group: + +1. Go to the top-level group's **Settings > General** page. +1. Expand the **Merge request approvals** section. +1. Select the settings you want. +1. Select **Save changes**. + ## Related topics - [Group wikis](../project/wiki/index.md) @@ -769,3 +788,22 @@ If a user sees a 404 when they would normally expect access, and the problem is In viewing the log entries, compare the `remote.ip` with the list of [allowed IPs](#restrict-group-access-by-ip-address) for the group. + +### Validation errors on namespaces and groups + +[GitLab 14.4 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70365) performs +the following checks when creating or updating namespaces or groups: + +- Namespaces must not have parents. +- Group parents must be groups and not namespaces. + +You can disable the validation if GitLab shows the following errors: + +- `A user namespace cannot have a parent`. +- `A group cannot have a user namespace as its parent`. + +To disable the validation, +[disable the `validate_namespace_parent_type` flag](../../administration/feature_flags.md). + +In the unlikely event that you had to disable this feature flag to prevent errors, +[contact Support](https://about.gitlab.com/support/) so that we can improve this validation. diff --git a/doc/user/group/insights/img/insights_group_configuration.png b/doc/user/group/insights/img/insights_group_configuration.png Binary files differdeleted file mode 100644 index d181a1e94c3..00000000000 --- a/doc/user/group/insights/img/insights_group_configuration.png +++ /dev/null diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 18177d656ab..9f841691eb8 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -9,34 +9,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. -Configure the Insights that matter for your groups to explore data such as -triage hygiene, issues created/closed per a given period, average time for merge -requests to be merged and much more. +Configure the Insights that matter for your groups. Explore data such as +triage hygiene, issues created or closed for a given period, average time for merge +requests to be merged, and much more.  ## View your group's Insights -You can access your group's Insights by clicking the **Analytics > Insights** -link in the left sidebar. +To access your group's Insights: -## Configure your Insights - -Navigate to your group's **Settings > General**, expand **Insights**, and choose -the project that holds your `.gitlab/insights.yml` configuration file: +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Insights**. - +## Configure your Insights -If no configuration was set, a [default configuration file]( -https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml) -will be used. +GitLab reads Insights from the [default configuration file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml). +If you want to customize it: -See the [Project's Insights documentation](../../project/insights/index.md) for -more details about the `.gitlab/insights.yml` configuration file. +1. Create a new file [`.gitlab/insights.yml`](../../project/insights/index.md) +in a project that belongs to your group. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand **Insights**. +1. Select the project that contains your `.gitlab/insights.yml` configuration file. +1. Select **Save changes**. ## Permissions -If you have access to view a group, then you have access to view their Insights. +If you have access to view a group, then you have access to view its Insights. NOTE: Issues or merge requests that you don't have access to (because you don't have diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index 685c4601ac4..c6cd763355b 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -7,16 +7,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Repositories Analytics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215104) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215104) in GitLab 13.4.  ## Current group code coverage -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/263478) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/263478) in GitLab 13.7. The **Analytics > Repositories** group page displays the overall test coverage of all your projects in your group. In the **Overall activity** section, you can see: @@ -27,46 +24,47 @@ In the **Overall activity** section, you can see: ## Average group test coverage from the last 30 days -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215140) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215140) in GitLab 13.9. The **Analytics > Repositories** group page displays the average test coverage of all your projects in your group in a graph for the last 30 days. ## Latest project test coverage list -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267624) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267624) in GitLab 13.6. To see the latest code coverage for each project in your group: -1. Go to **Analytics > Repositories** in the group (not from a project). -1. In the **Latest test coverage results** section, use the **Select projects** dropdown to choose the projects you want to check. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Repositories**. +1. In the **Latest test coverage results** section, from the **Select projects** dropdown list, choose the projects you want to check. You can download code coverage data for specific projects using [code coverage history](../../../ci/pipelines/settings.md#view-code-coverage-history). ## Download historic test coverage data -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215104) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215104) in GitLab 13.4. You can get a CSV of the code coverage data for all of the projects in your group. This report has a maximum of 1000 records. The code coverage data is from the default branch in each project. To get the report: -1. Go to your group's **Analytics > Repositories** page -1. Click **Download historic test coverage data (`.csv`)**, -1. In the popup, select the projects you want to include in the report. -1. Select the date range for the report from the preset options. -1. Click **Download test coverage data (`.csv`)**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Repositories**. +1. Select **Download historic test coverage data (.csv)**. +1. Select the projects and date range you want to include in the report. +1. Select **Download test coverage data (.csv)**. The projects dropdown shows up to 100 projects from your group. If the project you want to check is not in the dropdown list, you can select **All projects** to download the report for all projects in your group, including any projects that are not listed. There is a plan to improve this behavior in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/250684). -For each day that a coverage report was generated by a job in a project's pipeline, there will be a row in the CSV which includes: +For each day that a coverage report was generated by a job in a project's pipeline, a row in the CSV includes: -- The date when the coverage job ran +- The date the coverage job ran - The name of the job that generated the coverage report - The name of the project - The coverage value -If the project's code coverage was calculated more than once in a day, we will take the last value from that day. +If the project's code coverage was calculated more than once in a day, the last value from that day is used. NOTE: [In GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/270102), group code coverage diff --git a/doc/user/group/roadmap/img/roadmap_view_v14_3.png b/doc/user/group/roadmap/img/roadmap_view_v14_3.png Binary files differindex ca4b87b9fdd..ea5e870680e 100644 --- a/doc/user/group/roadmap/img/roadmap_view_v14_3.png +++ b/doc/user/group/roadmap/img/roadmap_view_v14_3.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 8f6b3e7244a..1c894550a14 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > Introduced in GitLab 11.0. -This page describes SAML for Groups. For instance-wide SAML on self-managed GitLab instances, see [SAML OmniAuth Provider](../../../integration/saml.md). +This page describes SAML for groups. For instance-wide SAML on self-managed GitLab instances, see [SAML OmniAuth Provider](../../../integration/saml.md). [View the differences between SaaS and Self-Managed Authentication and Authorization Options](../../../administration/auth/index.md#saas-vs-self-managed-comparison). SAML on GitLab.com allows users to sign in through their SAML identity provider. If the user is not already a member, the sign-in process automatically adds the user to the appropriate group. @@ -23,7 +23,8 @@ If required, you can find [a glossary of common terms](../../../integration/saml ## Configuring your identity provider -1. Navigate to the GitLab group and select **Settings > SAML SSO**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. 1. Configure your SAML identity provider using the **Assertion consumer service URL**, **Identifier**, and **GitLab single sign-on URL**. Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). See [specific identity provider documentation](#providers) for more details. @@ -42,7 +43,7 @@ GitLab.com uses the SAML NameID to identify users. The NameID element: - Is a required field in the SAML response. - Must be unique to each user. -- Must be a persistent value that will never change, such as a randomly generated unique user ID. +- Must be a persistent value that never changes, such as a randomly generated unique user ID. - Is case sensitive. The NameID must match exactly on subsequent login attempts, so should not rely on user input that could change between upper and lower case. - Should not be an email address or username. We strongly recommend against these as it's hard to guarantee it doesn't ever change, for example, when a person's name changes. Email addresses are @@ -66,15 +67,15 @@ the user details need to be passed to GitLab as SAML assertions. At a minimum, the user's email address *must* be specified as an assertion named `email` or `mail`. See [the assertions list](../../../integration/saml.md#assertions) for other available claims. - -NOTE: -The `username` assertion is not supported for GitLab.com SaaS integrations. +In addition to the attributes in the linked assertions list, GitLab.com supports `username` +and `nickname` attributes. ### Metadata configuration GitLab provides metadata XML that can be used to configure your identity provider. -1. Navigate to the group and select **Settings > SAML SSO**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. 1. Copy the provided **GitLab metadata URL**. 1. Follow your identity provider's documentation and paste the metadata URL when it's requested. @@ -82,7 +83,8 @@ GitLab provides metadata XML that can be used to configure your identity provide After you set up your identity provider to work with GitLab, you must configure GitLab to use it for authentication: -1. Navigate to the group's **Settings > SAML SSO**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. 1. Find the SSO URL from your identity provider and enter it the **Identity provider single sign-on URL** field. 1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. 1. Select the access level to be applied to newly added users in the **Default membership role** field. The default access level is 'Guest'. @@ -110,7 +112,7 @@ However, users are not prompted to sign in through SSO on each visit. GitLab che has authenticated through SSO. If it's been more than 1 day since the last sign-in, GitLab prompts the user to sign in again through SSO. -We intend to add a similar SSO requirement for [API activity](https://gitlab.com/gitlab-org/gitlab/-/issues/9152). +We intend to add a similar SSO requirement for [API activity](https://gitlab.com/gitlab-org/gitlab/-/issues/297389). SSO has the following effects when enabled: @@ -131,7 +133,7 @@ When SCIM updates, the user's access is immediately revoked. ## Providers -The SAML standard means that a wide range of identity providers will work with GitLab. Your identity provider may have relevant documentation. It may be generic SAML documentation, or specifically targeted for GitLab. +The SAML standard means that you can use a wide range of identity providers with GitLab. Your identity provider might have relevant documentation. It can be generic SAML documentation or specifically targeted for GitLab. When [configuring your identity provider](#configuring-your-identity-provider), please consider the notes below for specific providers to help avoid common issues and as a guide for terminology used. @@ -224,7 +226,7 @@ When a user tries to sign in with Group SSO, GitLab attempts to find or create a To link SAML to your existing GitLab.com account: -1. Sign in to your GitLab.com account. +1. Sign in to your GitLab.com account. [Reset your password](https://gitlab.com/users/password/new) if necessary. 1. Locate and visit the **GitLab single sign-on URL** for the group you're signing in to. A group owner can find this on the group's **Settings > SAML SSO** page. If the sign-in URL is configured, users can connect to the GitLab app from the identity provider. 1. Select **Authorize**. 1. Enter your credentials on the identity provider if prompted. @@ -265,6 +267,9 @@ convert the information to XML. An example SAML response is shown here. <saml2:Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"> <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">user.email</saml2:AttributeValue> </saml2:Attribute> + <saml2:Attribute Name="username" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">user.nickName</saml2:AttributeValue> + </saml2:Attribute> <saml2:Attribute Name="first_name" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">user.firstName</saml2:AttributeValue> </saml2:Attribute> @@ -309,7 +314,7 @@ group owner, and then you can unlink the account. For example, to unlink the `MyOrg` account: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. 1. In the **Social sign-in** section, select **Disconnect** next to the connected account. @@ -346,8 +351,8 @@ a SAML identity provider group name to a GitLab Access Level. This can be done f To link the SAML groups from the `saml:AttributeStatement` example above: -1. Enter the value of `saml:AttributeValue` in the `SAML Group Name` field. -1. Choose the desired `Access Level`. +1. In the **SAML Group Name** box, enter the value of `saml:AttributeValue`. +1. Choose the desired **Access Level**. 1. **Save** the group link. 1. Repeat to add additional group links if desired. @@ -356,7 +361,14 @@ To link the SAML groups from the `saml:AttributeStatement` example above: If a user is a member of multiple SAML groups mapped to the same GitLab group, the user gets the highest access level from the groups. For example, if one group is linked as `Guest` and another `Maintainer`, a user in both groups gets `Maintainer` -access. +access. + +Users granted: + +- A higher role with Group Sync are displayed as having + [direct membership](../../project/members/#display-direct-members) of the group. +- A lower or the same role with Group Sync are displayed as having + [inherited membership](../../project/members/#display-inherited-members) of the group. ### Automatic member removal @@ -480,7 +492,7 @@ If you receive a `404` during setup when using "verify configuration", make sure [SHA-1 generated fingerprint](../../../integration/saml.md#notes-on-configuring-your-identity-provider). If a user is trying to sign in for the first time and the GitLab single sign-on URL has not [been configured](#configuring-your-identity-provider), they may see a 404. -As outlined in the [user access section](#linking-saml-to-your-existing-gitlabcom-account), a group Owner will need to provide the URL to users. +As outlined in the [user access section](#linking-saml-to-your-existing-gitlabcom-account), a group Owner needs to provide the URL to users. ### Message: "SAML authentication failed: Extern UID has already been taken" @@ -502,13 +514,13 @@ Here are possible causes and solutions: | Cause | Solution | | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | -| When a user account with the email address already exists in GitLab, but the user does not have the SAML identity tied to their account. | The user will need to [link their account](#user-access-and-management). | +| When a user account with the email address already exists in GitLab, but the user does not have the SAML identity tied to their account. | The user needs to [link their account](#user-access-and-management). | ### Message: "SAML authentication failed: Extern UID has already been taken, User has already been taken" Getting both of these errors at the same time suggests the NameID capitalization provided by the identity provider didn't exactly match the previous value for that user. -This can be prevented by configuring the [NameID](#nameid) to return a consistent value. Fixing this for an individual user involves [unlinking SAML in the GitLab account](#unlinking-accounts), although this will cause group membership and to-dos to be lost. +This can be prevented by configuring the [NameID](#nameid) to return a consistent value. Fixing this for an individual user involves [unlinking SAML in the GitLab account](#unlinking-accounts), although this causes group membership and to-do items to be lost. ### Message: "Request to link SAML account must be authorized" @@ -541,7 +553,7 @@ Otherwise, to change the SAML app used for sign in, users need to [unlink the cu Many SAML terms can vary between providers. It is possible that the information you are looking for is listed under another name. -For more information, start with your identity provider's documentation. Look for their options and examples to see how they configure SAML. This can provide hints on what you'll need to configure GitLab to work with these providers. +For more information, start with your identity provider's documentation. Look for their options and examples to see how they configure SAML. This can provide hints on what you need to configure GitLab to work with these providers. It can also help to look at our [more detailed docs for self-managed GitLab](../../../integration/saml.md). SAML configuration for GitLab.com is mostly the same as for self-managed instances. diff --git a/doc/user/group/settings/img/import_panel_v14_1.png b/doc/user/group/settings/img/import_panel_v14_1.png Binary files differdeleted file mode 100644 index 28417383b6c..00000000000 --- a/doc/user/group/settings/img/import_panel_v14_1.png +++ /dev/null diff --git a/doc/user/group/settings/img/new_group_navigation_v13_1.png b/doc/user/group/settings/img/new_group_navigation_v13_1.png Binary files differdeleted file mode 100644 index 307175727c7..00000000000 --- a/doc/user/group/settings/img/new_group_navigation_v13_1.png +++ /dev/null diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index 516f3504a98..3f9d94f044e 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -4,6 +4,7 @@ stage: Manage group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- + # Group import/export **(FREE)** > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. @@ -23,9 +24,9 @@ Users with the [Owner role](../../permissions.md) for a group can enable import and export for that group: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > General > Visibility and access controls**. -1. Scroll to **Import sources**. -1. Enable the desired **Import sources**. +1. On the left sidebar, select **Settings > General**. +1. Expand **Visibility and access controls**. +1. In the **Import sources** section, select the checkboxes for the sources you want. ## Important Notes @@ -65,19 +66,23 @@ For more details on the specific data persisted in a group export, see the ## Export a group +WARNING: +This feature will be [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/4619) +in GitLab 14.6 and replaced by [GitLab Migration](../import/). + Users with the [Owner role](../../permissions.md) for a group can export the contents of that group: -1. On the top bar, select **Menu >** **Groups** and find your group. -1. On the left sidebar, select **Settings**. -1. Scroll to the **Advanced** section, and select **Export Group**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. In the **Advanced** section, select **Export Group**. 1. After the export is generated, you should receive an email with a link to the [exported contents](#exported-contents) in a compressed tar archive, with contents in NDJSON format. 1. Alternatively, you can download the export from the UI: 1. Return to your group's **Settings > General** page. - 1. Scroll to the **Advanced** section, and select **Download export**. - You can also generate a new file by clicking **Regenerate export**. + 1. In the **Advanced** section, select **Download export**. + You can also generate a new file by selecting **Regenerate export**. NOTE: The maximum import file size can be set by the Administrator, default is `0` (unlimited). @@ -91,24 +96,18 @@ The Enterprise Edition retains some group data that isn't part of the Community ## Importing the group -1. Navigate to the New Group page, either via the `+` button in the top navigation bar, or the **New subgroup** button -on an existing group's page. - -  - -1. On the New Group page, select the **Import group**. - -  +1. Create a new group: + - On the top bar, select **New** (**{plus}**) and then **New group**. + - On an existing group's page, select the **New subgroup** button. +1. Select **Import group**. 1. Enter your group name. - 1. Accept or modify the associated group URL. - -1. Click **Choose file** - +1. Select **Choose file**. 1. Select the file that you exported in the [Export a group](#export-a-group) section. +1. To begin importing, select **Import group**. -1. Click **Import group** to begin importing. Your newly imported group page appears after the operation completes. +Your newly imported group page appears after the operation completes. ## Version history diff --git a/doc/user/group/subgroups/img/create_new_group.png b/doc/user/group/subgroups/img/create_new_group.png Binary files differdeleted file mode 100644 index 9d011ec709a..00000000000 --- a/doc/user/group/subgroups/img/create_new_group.png +++ /dev/null diff --git a/doc/user/group/subgroups/img/create_subgroup_button_v13_6.png b/doc/user/group/subgroups/img/create_subgroup_button_v13_6.png Binary files differdeleted file mode 100644 index 013aee6b0b4..00000000000 --- a/doc/user/group/subgroups/img/create_subgroup_button_v13_6.png +++ /dev/null diff --git a/doc/user/group/subgroups/img/group_members_13_7.png b/doc/user/group/subgroups/img/group_members_13_7.png Binary files differdeleted file mode 100644 index ab22bcb932c..00000000000 --- a/doc/user/group/subgroups/img/group_members_13_7.png +++ /dev/null diff --git a/doc/user/group/subgroups/img/group_members_v14_4.png b/doc/user/group/subgroups/img/group_members_v14_4.png Binary files differnew file mode 100644 index 00000000000..695564a8b74 --- /dev/null +++ b/doc/user/group/subgroups/img/group_members_v14_4.png diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index aaff0574ef0..49032d0a2ef 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -73,51 +73,41 @@ subgroups, the behavior is the same as when performing these actions at the ## Creating a subgroup +Users can create subgroups if they are explicitly added as an Owner (or +Maintainer, if that setting is enabled) to an immediate parent group, even if group +creation is disabled by an administrator in their settings. + +To create a subgroup: + +1. On the top bar, select **Menu > Groups** and find the parent group. +1. In the top right, select **New subgroup**. +1. Select **Create group**. +1. Fill in the fields. View a list of [reserved names](../../reserved_names.md) + that cannot be used as group names. +1. Select **Create group**. + +### Change who can create subgroups + To create a subgroup you must either be an Owner or a Maintainer of the group, depending on the group's setting. -By default, groups created in: +By default: -- GitLab 12.2 or later allow both Owners and Maintainers to create subgroups. -- GitLab 12.1 or earlier only allow Owners to create subgroups. +- In GitLab 12.2 or later, both Owners and Maintainers can create subgroups. +- In GitLab 12.1 or earlier, only Owners can create subgroups. -The setting can be changed for any group by: +You can change this setting: -- A group owner: +- As group owner: 1. Select the group. 1. On the left sidebar, select **Settings > General**. - 1. Expand the **Permissions, LFS, 2FA** section. -- An administrator: + 1. Expand **Permissions, LFS, 2FA**. +- As an administrator: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Groups**. 1. Select the group, and select **Edit**. -For: - -- More information, check the [permissions table](../../permissions.md#group-members-permissions). -- A list of words that are not allowed to be used as group names, see the - [reserved names](../../reserved_names.md). - -Users can always create subgroups if they are explicitly added as an Owner (or -Maintainer, if that setting is enabled) to an immediate parent group, even if group -creation is disabled by an administrator in their settings. - -To create a subgroup: - -1. In the group's dashboard click the **New subgroup** button. - -  - -1. Create a new group like you would normally do. Notice that the immediate parent group - namespace is fixed under **Group path**. The visibility level can differ from - the immediate parent group. - -  - -1. Click the **Create group** button to be redirected to the new group's - dashboard page. - -Follow the same process to create any subsequent groups. +For more information, view the [permissions table](../../permissions.md#group-members-permissions). ## Membership @@ -136,7 +126,7 @@ the **Members** page of the group the member was added. You can tell if a member has inherited the permissions from a parent group by looking at the group's **Members** page. - + From the image above, we can deduce the following things: diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 68ae5e0df2d..5c8393f30ab 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -284,9 +284,9 @@ To sort the stage table by a table column, select the table header. You can sort in ascending or descending order. To find items that spent the most time in a stage, potentially causing bottlenecks in your value stream, sort the table by the **Time** column. From there, select individual items to drill in and investigate how delays are happening. -To see which items the stage most recently, sort by the work item column on the left. +To see which items most recently exited the stage, sort by the work item column on the left. -The table displays up to 20 items at a time. If there are more than 20 items, you can use the +The table displays 20 items per page. If there are more than 20 items, you can use the **Prev** and **Next** buttons to navigate through the pages. ### Creating a value stream @@ -302,11 +302,12 @@ best practices. You can customize this flow by adding, hiding or re-ordering sta To create a value stream: -1. Navigate to your group's **Analytics > Value Stream**. -1. Click the Value stream dropdown and select **Create new Value Stream** -1. Fill in a name for the new Value Stream - - You can [customize the stages](#creating-a-value-stream-with-stages) -1. Click the **Create Value Stream** button. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Value Stream**. +1. In the top right, select the dropdown list and then **Create new Value Stream**. +1. Enter a name for the new Value Stream. + - You can [customize the stages](#creating-a-value-stream-with-stages). +1. Select **Create Value Stream**.  @@ -324,18 +325,19 @@ add stages as desired. To create a value stream with stages: -1. Go to your group and select **Analytics > Value Stream**. -1. Select the Value Stream dropdown and select **Create new Value Stream**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Value Stream**. +1. In the top right, select the dropdown list and then **Create new Value Stream**. 1. Select either **Create from default template** or **Create from no template**. - - Default stages in the value stream can be hidden or re-ordered. + - You can hide or re-order default stages in the value stream.  - - New stages can be added by clicking the 'Add another stage' button. - - The name, start and end events for the stage can be selected + - You can add new stages by selecting **Add another stage**. + - You can select the name and start and end events for the stage.  -1. Select the **Create Value Stream** button to save the value stream. +1. Select **Create Value Stream**. #### Label-based stages @@ -358,8 +360,9 @@ In this example, we'd like to measure times for deployment from a staging enviro After you create a value stream, you can customize it to suit your purposes. To edit a value stream: -1. Go to your group and select **Analytics > Value Stream**. -1. Find and select the relevant value stream from the value stream dropdown. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Value Stream**. +1. In the top right, select the dropdown list and then select the relevant value stream. 1. Next to the value stream dropdown, select **Edit**. The edit form is populated with the value stream details. 1. Optional: @@ -377,10 +380,11 @@ After you create a value stream, you can customize it to suit your purposes. To To delete a custom value stream: -1. Navigate to your group's **Analytics > Value Stream**. -1. Click the Value stream dropdown and select the value stream you would like to delete. -1. Click the **Delete (name of value stream)**. -1. Click the **Delete** button to confirm. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Value Stream**. +1. In the top right, select the dropdown list and then select the value stream you would like to delete. +1. Select **Delete (name of value stream)**. +1. To confirm, select **Delete**.  @@ -398,15 +402,6 @@ from the chart itself. The chart data is limited to the last 500 items. -### Disabling chart - -This chart is enabled by default. If you have a self-managed instance, an -administrator can open a Rails console and disable it with the following command: - -```ruby -Feature.disable(:cycle_analytics_scatterplot_enabled) -``` - ## Type of work - Tasks by type chart > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32421) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.10. @@ -422,7 +417,7 @@ select up to a total of 15 labels. ## Permissions -To access Group-level Value Stream Analytics, users must have Reporter access or above. +To access Group-level Value Stream Analytics, users must have at least the Reporter role. You can [read more about permissions](../../permissions.md) in general. diff --git a/doc/user/index.md b/doc/user/index.md index d6eaad469c1..c9f20af9668 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -13,10 +13,10 @@ Welcome to GitLab! We're glad to have you here! As a GitLab user you have access to all the features your [subscription](https://about.gitlab.com/pricing/) includes, except [GitLab administrator](../administration/index.md) -settings, unless you have admin privileges to install, configure, +settings, unless you have administrator privileges to install, configure, and upgrade your GitLab instance. -Admin privileges for [GitLab.com](https://gitlab.com/) are restricted to the GitLab team. +Administrator privileges for [GitLab.com](https://gitlab.com/) are restricted to the GitLab team. For more information on configuring GitLab self-managed instances, see the [Administrator documentation](../administration/index.md). @@ -63,7 +63,7 @@ With GitLab Enterprise Edition, you can also: - Use [Burndown Charts](project/milestones/burndown_and_burnup_charts.md) to track progress during a sprint or while working on a new version of their software. - Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Search](search/advanced_search.md) for faster, more advanced code search across your entire GitLab instance. - [Authenticate users with Kerberos](../integration/kerberos.md). -- [Mirror a repository](project/repository/repository_mirroring.md) from elsewhere on your local server. +- [Mirror a repository](project/repository/mirror/index.md) from elsewhere on your local server. - View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](../ci/pipelines/multi_project_pipelines.md). - [Lock files](project/file_lock.md) to prevent conflicts. - View the current health and status of each CI environment running on Kubernetes with [deploy boards](project/deploy_boards.md). @@ -78,7 +78,7 @@ There are several types of users in GitLab: - Regular users and GitLab.com users. <!-- Note: further description TBA --> - [Groups](group/index.md) of users. -- GitLab [admin area](admin_area/index.md) user. +- GitLab [administrator area](admin_area/index.md) user. - [GitLab Administrator](../administration/index.md) with full access to self-managed instances' features and settings. - [Internal users](../development/internal_users.md). diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index ada278292a9..636cb1bb457 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -63,16 +63,6 @@ You can also use the [certificate-based method](../../../project/clusters/multip but, for [security implications](#security-implications-for-clusters-connected-with-certificates), we don't recommend using this method. -## Cluster levels - -Choose your cluster's level according to its purpose: - -| Level | Purpose | -|--|--| -| [Project level](../../../project/clusters/index.md) | Use your cluster for a single project. | -| [Group level](../../../group/clusters/index.md) | Use the same cluster across multiple projects within your group. | -| [Instance level](../../../instance/clusters/index.md) **(FREE SELF)** | Use the same cluster across groups and projects within your instance. | - ## Supported cluster versions GitLab is committed to support at least two production-ready Kubernetes minor @@ -86,29 +76,42 @@ version. The range of supported versions is based on the evaluation of: GitLab supports the following Kubernetes versions, and you can upgrade your Kubernetes version to any supported version at any time: +- 1.20 (support ends on July 22, 2022) - 1.19 (support ends on February 22, 2022) - 1.18 (support ends on November 22, 2021) - 1.17 (support ends on September 22, 2021) +[Adding support to other versions of Kubernetes is managed under this epic](https://gitlab.com/groups/gitlab-org/-/epics/4827). + Some GitLab features may support versions outside the range provided here. -## View your clusters +## Cluster levels + +Choose your cluster's level according to its purpose: + +| Level | Purpose | +|--|--| +| [Project level](../../../project/clusters/index.md) | Use your cluster for a single project. | +| [Group level](../../../group/clusters/index.md) | Use the same cluster across multiple projects within your group. | +| [Instance level](../../../instance/clusters/index.md) | Use the same cluster across groups and projects within your instance. | + +### View your clusters To view the Kubernetes clusters connected to your project, -group, or instance, open the cluster's page according to the -[level](#cluster-levels) of your cluster. +group, or instance, open the cluster's page according to +your cluster's level. **Project-level clusters:** -1. Go to your project's homepage. +1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. **Group-level clusters:** -1. Go to your group's homepage. +1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Kubernetes**. -**Instance-level clusters:** **(FREE SELF)** +**Instance-level clusters:** 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Kubernetes**. diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 90044fa74e1..3c934b72886 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -26,25 +26,46 @@ You can then modify the project files according to your needs. **Steps:** 1. [Import the example project](#import-the-example-project). -1. [Add your GCP credentials to GitLab](#add-your-gcp-credentials-to-gitlab). +1. [Create your GCP and GitLab credentials](#create-your-gcp-and-gitlab-credentials). 1. [Configure your project](#configure-your-project). 1. [Deploy your cluster](#deploy-your-cluster). ## Import the example project +To create a new group-level cluster from GitLab using Infrastructure as Code, it is necessary +to create a project to manage the cluster from. In this tutorial, we import a pre-configured +sample project to help you get started. + Start by [importing the example project by URL](../../../project/import/repo_by_url.md). Use `https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke.git` as URL. -## Add your GCP credentials to GitLab +This project provides you with the following resources: + +- A [cluster on Google Cloud Platform (GCP)](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke/-/blob/master/gke.tf) +with defaults for name, location, node count, and Kubernetes version. +- A [`gitlab-admin` K8s service account](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke/-/blob/master/gitlab-admin.tf) with `cluster-admin` privileges. +- The new group-level cluster connected to GitLab. +- Pre-configures Terraform files: -After importing the project, you need to set up [CI environment variables](../../../../ci/variables/index.md) to associate your cluster on GCP to your group in GitLab. + ```plaintext + ├── backend.tf # State file Location Configuration + ├── gke.tf # Google GKE Configuration + ├── gitlab-admin.tf # Adding kubernetes service account + └── group_cluster.tf # Registering kubernetes cluster to GitLab `apps` Group + ``` -We advise that you [set environment variables through the GitLab UI](../../../../ci/variables/index.md#add-a-cicd-variable-to-a-project) -so that your credentials are not exposed through the code. To do so, follow the steps below. +## Create your GCP and GitLab credentials -### Prepare your credentials on GCP +To set up your project to communicate to GCP and the GitLab API: -1. Create a [GCP service account](https://cloud.google.com/docs/authentication/getting-started) to authenticate GCP with GitLab. It needs the following roles: `Computer Network Viewer`, `Kubernetes Engine Admin`, and `Service Account User`. -1. Download the JSON file with the service account key. +1. Create a [GitLab personal access token](../../../profile/personal_access_tokens.md) with + `api` scope. The Terraform script uses it to connect the cluster to your GitLab group. Take note of the generated token. You will + need it when you [configure your project](#configure-your-project). +1. To authenticate GCP with GitLab, create a [GCP service account](https://cloud.google.com/docs/authentication/getting-started) +with following roles: `Compute Network Viewer`, `Kubernetes Engine Admin`, `Service Account User`, and `Service Account Admin`. Both User and Admin +service accounts are necessary. The User role impersonates the [default service account](https://cloud.google.com/compute/docs/access/service-accounts#default_service_account) +when [creating the node pool](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/using_gke_with_terraform#node-pool-management). +The Admin role creates a service account in the `kube-system` namespace. +1. Download the JSON file with the service account key you created in the previous step. 1. On your computer, encode the JSON file to `base64` (replace `/path/to/sa-key.json` to the path to your key): ```shell @@ -53,36 +74,38 @@ so that your credentials are not exposed through the code. To do so, follow the 1. Use the output of this command as the `BASE64_GOOGLE_CREDENTIALS` environment variable in the next step. -### Add your credentials to GitLab as environment variables +## Configure your project -1. On GitLab, from your project's sidebar, go to **Settings > CI/CD** and expand **Variables**. -1. Add your `GITLAB_TOKEN` ([personal access token](../../../profile/personal_access_tokens.md)). -1. Add the variable `BASE64_GOOGLE_CREDENTIALS` from the previous step. +**Required configuration:** -## Configure your project +Use CI/CD environment variables to configure your project as detailed below. -After authenticating with GCP, replace the project's defaults from the example -project with your own. To do so, edit the files as described below. +**Required configuration:** -Edit `gke.tf`: +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Variables**. +1. Set the variable `TF_VAR_gitlab_token` to the GitLab personal access token you just created. +1. Set the variable `BASE64_GOOGLE_CREDENTIALS` to the `base64` encoded JSON file you just created. +1. Set the variable `TF_VAR_gcp_project` to your GCP's `project` name. +1. Set the variable `TF_VAR_gitlab_group` to the name of the group you want to connect your cluster to. If your group's URL is `https://gitlab.example.com/my-example-group`, `my-example-group` is your group's name. -1. **(Required)** Replace the GCP `project` with a unique project name. -1. **(Optional)** Choose the `name` of your cluster. -1. **(Optional)** Choose the `region` and `zone` that you would like to deploy your cluster to. -1. Push the changes to your project's default branch. +**Optional configuration:** -Edit `group_cluster.tf`: +The file [`variables.tf`](https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-gke/-/blob/master/variables.tf) +contains other variables that you can override according to your needs: -1. **(Required)**: Replace the `full_path` with the path to your group. -1. **(Optional)**: Choose your cluster base domain through `domain`. -1. **(Optional)**: Choose your environment through `environment_scope`. -1. Push the changes to your project's default branch. +- `TF_VAR_gcp_region`: Set your cluster's region. +- `TF_VAR_cluster_name`: Set your cluster's name. +- `TF_VAR_machine_type`: Set the machine type for the Kubernetes nodes. +- `TF_VAR_cluster_description`: Set a description for the cluster. We recommend setting this to `$CI_PROJECT_URL` to create a reference to your GitLab project on your GCP cluster detail page. This way you know which project was responsible for provisioning the cluster you see on the GCP dashboard. +- `TF_VAR_base_domain`: Set to the base domain to provision resources under. +- `TF_VAR_environment_scope`: Set to the environment scope for your cluster. Refer to the [GitLab Terraform provider](https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs) and the [Google Terraform provider](https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference) documentation for further resource options. ## Deploy your cluster -After adjusting the files in the previous step, manually trigger the deployment of your cluster. In GitLab: +After configuring your project, manually trigger the deployment of your cluster. In GitLab: 1. From your project's sidebar, go to **CI/CD > Pipelines**. 1. Select the dropdown icon (**{angle-down}**) next to the play icon (**{play}**). diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md new file mode 100644 index 00000000000..16ca6d02865 --- /dev/null +++ b/doc/user/infrastructure/clusters/index.md @@ -0,0 +1,66 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Kubernetes clusters **(FREE)** + +> - Project-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. +> - Group-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in GitLab 11.6. +> - Instance-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. + +Kubernetes is a container orchestration platform to deploy applications +in a cluster without downtime and that scales as you need. + +With the GitLab integration with Kubernetes, you can: + +1. [Connect your cluster](#connect-your-cluster-to-gitlab). +1. [Manage your cluster](#manage-your-cluster). +1. [Deploy your cluster](#deploy-to-your-cluster). + +See the [Kubernetes clusters versions supported by GitLab](connect/index.md#supported-cluster-versions). + +## Connect your cluster to GitLab + +Learn how to [create new and connect existing clusters to GitLab](connect/index.md). + +## Manage your cluster + +- [Cluster Management Project](../../clusters/management_project.md): +create a project to manage your cluster's shared resources requiring +`cluster-admin` privileges such as an Ingress controller. + - [Cluster Management Project Template](../../clusters/management_project_template.md): start a cluster management project directly from a template. + - [Migrate to Cluster Management Project](../../clusters/migrating_from_gma_to_project_template.md): migrate from the deprecated GitLab Managed Apps to Cluster Management Projects. + - [GitLab Managed Apps](../../clusters/applications.md) (deprecated in favor of Cluster Management Projects): configure applications in your cluster directly from GitLab. +- [Cluster integrations](../../clusters/integrations.md): install +third-party applications into your cluster and manage them from GitLab. +- [GitLab-managed clusters](../../project/clusters/gitlab_managed_clusters.md): +enable GitLab to automatically create resources for your clusters. +- [Cost management](../../clusters/cost_management.md): see insights into your cluster's resource usage. +- [Crossplane integration](../../clusters/crossplane.md): manage your cluster's resources and cloud infrastructure with Crossplane. + +### Monitor your cluster + +- [Prometheus monitoring](../../project/integrations/prometheus_library/kubernetes.md): detect and monitor Kubernetes metrics with Prometheus. +- [NGINX monitoring](../../project/integrations/prometheus_library/nginx.md): automatically monitor NGINX Ingress. +- [Clusters health](manage/clusters_health.md): monitor your cluster's health, such as CPU and memory usage. + +### Secure your cluster + +- [Container Host Security](../../project/clusters/protect/container_host_security/index.md): monitor and block activity inside a container and enforce security policies across the cluster. +- [Container Network security](../../project/clusters/protect/container_network_security/index.md): filter traffic going in and out of the cluster and traffic between pods through a firewall with Cilium NetworkPolicies. + +## Deploy to your cluster + +- [CI/CD Tunnel](../../clusters/agent/ci_cd_tunnel.md): use the CI/CD Tunnel to run Kubernetes commands from different projects. +- [Inventory object](deploy/inventory_object.md): track objects applied to a cluster configured with the Kubernetes Agent. +- [Auto DevOps](../../../topics/autodevops/index.md): enable Auto DevOps +to allow GitLab automatically detect, build, test, and deploy applications. +- [Cluster environments](../../clusters/environments.md): view CI/CD environments deployed to Kubernetes clusters. +- [Canary Deployments](../../project/canary_deployments.md): deploy app updates to a small portion of the fleet with this Continuous Delivery strategy. +- [Deploy to your cluster](../../project/clusters/deploy_to_cluster.md): +deploy applications into your cluster using cluster certificates. +- [Deploy Boards](../../project/deploy_boards.md): view the current health and status of each CI/CD environment running on your cluster, and the status of deployment pods. +- [Pod logs](../../project/clusters/kubernetes_pod_logs.md): view the logs of your cluster's running pods. +- [Serverless](../../project/clusters/serverless/index.md) (deprecated): deploy Serverless applications in Kubernetes environments and cloud Function as a Service (FaaS) environments. diff --git a/doc/user/infrastructure/clusters/manage/clusters_health.md b/doc/user/infrastructure/clusters/manage/clusters_health.md new file mode 100644 index 00000000000..009945589ad --- /dev/null +++ b/doc/user/infrastructure/clusters/manage/clusters_health.md @@ -0,0 +1,14 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Clusters health **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) to GitLab Free in 13.2. + +When [the Prometheus cluster integration is enabled](../../../clusters/integrations.md#prometheus-cluster-integration), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. + + diff --git a/doc/user/project/clusters/img/k8s_cluster_monitoring.png b/doc/user/infrastructure/clusters/manage/img/k8s_cluster_monitoring.png Binary files differindex 0a8c5043c65..0a8c5043c65 100644 --- a/doc/user/project/clusters/img/k8s_cluster_monitoring.png +++ b/doc/user/infrastructure/clusters/manage/img/k8s_cluster_monitoring.png diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index 9f28a40474e..e99dc691774 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -35,7 +35,7 @@ DevSecOps pipeline by default targeted at Kubernetes based deployments. To suppo all the GitLab features, GitLab offers a cluster management project for easy onboarding. The deploy boards provide quick insights into your cluster, including pod logs tailing. -Learn more about the [GitLab integration with Kubernetes](../project/clusters/index.md). +Learn more about the [GitLab integration with Kubernetes](clusters/index.md). ## Runbooks in GitLab diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 20cd30b6110..de09cd4845e 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -4,7 +4,7 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance-level Kubernetes clusters +# Instance-level Kubernetes clusters **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. @@ -39,4 +39,4 @@ are deployed to the Kubernetes cluster, see the documentation for ## More information For information on integrating GitLab and Kubernetes, see -[Kubernetes clusters](../../project/clusters/index.md). +[Kubernetes clusters](../../infrastructure/clusters/index.md). diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 4149307c45a..5e600b6e0d1 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -524,7 +524,7 @@ GitLab Flavored Markdown recognizes the following: | specific user | `@user_name` | | | | specific group | `@group_name` | | | | entire team | `@all` | | | -| project | `namespace/project` | | | +| project | `namespace/project>` | | | | issue | ``#123`` | `namespace/project#123` | `project#123` | | merge request | `!123` | `namespace/project!123` | `project!123` | | snippet | `$123` | `namespace/project$123` | `project$123` | @@ -955,6 +955,10 @@ Do not change to a reference style link.  +In the rare case where you need to set a specific height or width for an image, +you can use the `img` HTML tag instead of Markdown and set its `height` and +`width` parameters. + #### Videos If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#videos). diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 2787aefdeca..6e3af45df17 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -10,6 +10,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. > - Support for Composer 2.0 [added](https://gitlab.com/gitlab-org/gitlab/-/issues/259840) in GitLab Free 13.10. +WARNING: +The Composer package registry for GitLab is under development and isn't ready for production use due to +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6817) details the remaining +work and timelines to make it production ready. + Publish [Composer](https://getcomposer.org/) packages in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. @@ -120,6 +125,7 @@ You can publish a Composer package to the Package Registry as part of your CI/CD deploy: stage: deploy script: + - apk add curl - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "${CI_API_V4_URL}/projects/$CI_PROJECT_ID/packages/composer"' ``` @@ -132,7 +138,7 @@ To view the published package, go to **Packages & Registries > Package Registry* A more detailed Composer CI/CD file is also available as a `.gitlab-ci.yml` template: 1. On the left sidebar, select **Project information**. -1. Above the file list, click **Set up CI/CD**. If this button is not available, select **CI/CD Configuration** and then **Edit**. +1. Above the file list, select **Set up CI/CD**. If this button is not available, select **CI/CD Configuration** and then **Edit**. 1. From the **Apply a template** list, select **Composer**. WARNING: diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index dc08baaf731..33c48478921 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -9,6 +9,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in GitLab Premium 12.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +WARNING: +The Conan package registry for GitLab is under development and isn't ready for production use due to +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6816) details the remaining +work and timelines to make it production ready. + Publish Conan packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -358,7 +363,7 @@ There are two ways to remove a Conan package from the GitLab Package Registry. - From the GitLab user interface: Go to your project's **Packages & Registries > Package Registry**. Remove the - package by clicking the red trash icon. + package by selecting **Remove repository** (**{remove}**). ## Search for Conan packages in the Package Registry diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 1db2165cd5d..c39552c1edb 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -648,7 +648,11 @@ For self-managed instances, those settings can be updated in the [Rails console] ApplicationSetting.last.update(container_registry_expiration_policies_worker_capacity: 3) ``` -Alternatively, once the limits are [enabled](#enable-or-disable-cleanup-policy-limits), they are available in the [admin area](../../admin_area/index.md) **Settings > CI/CD > Container Registry**. +Alternatively, once the limits are [enabled](#enable-or-disable-cleanup-policy-limits), +they are available in the [administrator area](../../admin_area/index.md): + +1. On the top bar, select **Menu > Admin**. +1. Go to **Settings > CI/CD > Container Registry**. #### Enable or disable cleanup policy limits @@ -714,15 +718,6 @@ Check the regex patterns to ensure they are valid. GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/). View some common [regex pattern examples](#regex-pattern-examples). -## Use the Container Registry to store Helm Charts - -With the launch of [Helm v3](https://helm.sh/docs/topics/registries/), -you can use the Container Registry to store Helm Charts. However, due to the way metadata is passed -and stored by Docker, it is not possible for GitLab to parse this data and meet performance standards. -[This epic](https://gitlab.com/groups/gitlab-org/-/epics/2313) updates the architecture of the Container Registry to support Helm Charts. - -[Read more about the above challenges](https://gitlab.com/gitlab-org/gitlab/-/issues/38047#note_298842890). - ## Limitations - Moving or renaming existing Container Registry repositories is not supported @@ -911,10 +906,10 @@ these steps: while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs ``` -### Troubleshoot as a GitLab server admin +### Troubleshoot as a GitLab server administrator Troubleshooting the GitLab Container Registry, most of the times, requires -administrator access to the GitLab server. +you to log in to GitLab server with the Administrator role. [Read how to troubleshoot the Container Registry](../../../administration/packages/container_registry.md#troubleshooting). diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index 29641380753..36d85d94161 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -12,7 +12,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: The Debian package registry for GitLab is under development and isn't ready for production use due to -limited functionality. +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6057) details the remaining +work and timelines to make it production ready. Publish Debian packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -138,7 +139,7 @@ To install a package: 1. Configure the repository: - If you are using a private project, add your [credentials](#authenticate-to-the-package-registry) to your apt config: + If you are using a private project, add your [credentials](#authenticate-to-the-package-registry) to your apt configuration: ```shell echo 'machine gitlab.example.com login <username> password <your_access_token>' \ diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index ad25ec7edbf..5d482a15460 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -33,6 +33,14 @@ The following images and packages are supported. For a list of planned additions, view the [direction page](https://about.gitlab.com/direction/package/#dependency-proxy). +## Enable or disable the Dependency Proxy for a group + +To enable or disable the Dependency Proxy for a group: + +1. Go to your group's **Settings > Packages & Registries**. +1. Expand the **Dependency Proxy** section. +1. To enable the proxy, turn on **Enable Proxy**. To disable it, turn the toggle off. + ## View the Dependency Proxy To view the Dependency Proxy: @@ -66,7 +74,7 @@ has disrupted your existing Dependency Proxy usage. Because the Dependency Proxy is storing Docker images in a space associated with your group, you must authenticate against the Dependency Proxy. -Follow the [instructions for using images from a private registry](../../../ci/docker/using_docker_images.md#define-an-image-from-a-private-container-registry), +Follow the [instructions for using images from a private registry](../../../ci/docker/using_docker_images.md#access-an-image-from-a-private-container-registry), but instead of using `registry.example.com:5000`, use your GitLab domain with no port `gitlab.example.com`. For example, to manually log in: @@ -171,8 +179,53 @@ from the GitLab server. Blobs are kept forever on the GitLab server, and there is no hard limit on how much data can be stored. +### Using the API to clear the cache + To reclaim disk space used by image blobs that are no longer needed, use -the [Dependency Proxy API](../../../api/dependency_proxy.md). +the [Dependency Proxy API](../../../api/dependency_proxy.md) to clear the entire +cache. + +If you clear the cache, the next time a pipeline runs it must pull an image or tag from Docker Hub. + +### Cleanup policies + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab Free 14.4. + +The cleanup policy is a scheduled job you can use to clear cached images that are no longer used, +freeing up additional storage space. The policies use time-to-live (TTL) logic: + +- The number of days is configured. +- All cached dependency proxy files that have not been pulled in that many days are deleted. + +Use the [GraphQL API](../../../api/graphql/reference/index.md#mutationupdatedependencyproxyimagettlgrouppolicy) +to enable and configure cleanup policies: + +```graphql +mutation { + updateDependencyProxyImageTtlGroupPolicy(input: + { + groupPath: "<your-full-group-path>", + enabled: true, + ttl: 90 + } + ) { + dependencyProxyImageTtlPolicy { + enabled + ttl + } + errors + } +} +``` + +See the [Getting started with GraphQL](../../../api/graphql/getting_started.md) +guide to learn how to make GraphQL queries. Support for enabling and configuring cleanup policies in +the UI is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340777). + +When the policy is initially enabled, the default TTL setting is 90 days. Once enabled, stale +dependency proxy files are queued for deletion each day. Deletion may not occur right away due to +processing time. If the image is pulled after the cached files are marked as expired, the expired +files are ignored and new files are downloaded and cached from the external registry. ## Docker Hub rate limits and the Dependency Proxy @@ -227,7 +280,7 @@ script: ```shell # Note, you must have jq installed to run this command -TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 | grep RateLimit +TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 | grep --ignore-case RateLimit ... ``` @@ -256,10 +309,6 @@ hub_docker_quota_check: TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq --raw-output .token) && curl --head --header "Authorization: Bearer $TOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest" 2>&1 ``` -## Use the NPM Dependency Proxy for NPM packages - -For information on this, see [Dependency Proxy](../npm_registry/#dependency-proxy). - ## Troubleshooting ### Dependency Proxy Connection Failure diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index 0c04c4a23d0..f17910cd46d 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -9,10 +9,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27376) in GitLab Premium 13.1. > - It's deployed behind a feature flag, disabled by default. > - It's disabled for GitLab.com. -> - It's not recommended for production use. > - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-the-go-proxy). > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +WARNING: +The Go package registry for GitLab is under development and isn't ready for production use due to +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/3043) details the remaining +work and timelines to make it production ready. + With the Go proxy for GitLab, every project in GitLab can be fetched with the [Go proxy protocol](https://proxy.golang.org/). diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index 2d984d76b97..c88fba83dc7 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -8,6 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18997) in GitLab 14.1. +WARNING: +The Helm chart registry for GitLab is under development and isn't ready for production use due to +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6366) details the remaining +work and timelines to make it production ready. + Publish Helm packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -31,6 +36,10 @@ To authenticate to the Helm repository, you need either: ## Publish a package +WARNING: +The `helm-push` command is broken in Helm 3.7. For more information, see the [open issue](https://github.com/chartmuseum/helm-push/issues/109) +in the Chart Museum project. + NOTE: You can publish Helm charts with duplicate names or versions. If duplicates exist, GitLab always returns the chart with the latest version. @@ -105,7 +114,7 @@ helm install my-release project-1/mychart - `<project_id>`: the project ID (like `42`). - `<channel>`: the name of the channel (like `stable`). -If the repo has previously been added, you may need to run: +If the repository has previously been added, you may need to run: ```shell helm repo update @@ -123,3 +132,15 @@ Check the [Sidekiq log](../../../administration/logs.md#sidekiqlog) for any related errors. If you see `Validation failed: Version is invalid`, it means that the version in your `Chart.yaml` file does not follow [Helm Chart versioning specifications](https://helm.sh/docs/topics/charts/#charts-and-versioning). To fix the error, use the correct version syntax and upload the chart again. + +### `helm push` results in an error + +The `helm push` plugin is not yet supported in Helm 3.7. If you try to push a chart using +`helm push`, it produces the following error: + +```plaintext +Error: this feature has been marked as experimental and is not enabled by default. Please set HELM_EXPERIMENTAL_OCI=1 in your environment to use this feature +``` + +To continue to use the plugin, you can push an image using [curl](#use-cicd-to-publish-a-helm-package) +or downgrade your version of Helm. diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index 9158b5cc674..a8f1b1998ae 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -10,20 +10,31 @@ The GitLab [Package Registry](package_registry/index.md) acts as a private or pu for a variety of common package managers. You can publish and share packages, which can be easily consumed as a dependency in downstream projects. +WARNING: +Not all package manager formats are ready for production use. To view each format's status, see the +table's **Status** column. + The Package Registry supports the following formats: -| Package type | GitLab version | -| ------------ | -------------- | -| [Composer](composer_repository/index.md) | 13.2+ | -| [Conan](conan_repository/index.md) | 12.6+ | -| [Go](go_proxy/index.md) | 13.1+ | -| [Helm](helm_repository/index.md) | 14.1+ | -| [Maven](maven_repository/index.md) | 11.3+ | -| [npm](npm_registry/index.md) | 11.7+ | -| [NuGet](nuget_repository/index.md) | 12.8+ | -| [PyPI](pypi_repository/index.md) | 12.10+ | -| [Generic packages](generic_packages/index.md) | 13.5+ | -| [Ruby gems](rubygems_registry/index.md) | 13.10+ | +| Package type | GitLab version | Status | +| ------------ | -------------- |------- | +| [Maven](maven_repository/index.md) | 11.3+ | Stable | +| [npm](npm_registry/index.md) | 11.7+ | Stable | +| [NuGet](nuget_repository/index.md) | 12.8+ | Stable | +| [PyPI](pypi_repository/index.md) | 12.10+ | Stable | +| [Generic packages](generic_packages/index.md) | 13.5+ | Stable | +| [Composer](composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | +| [Conan](conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | +| [Helm](helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | +| [Debian](debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | +| [Go](go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | +| [Ruby gems](rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | + +Status: + +- Alpha: behind a feature flag and not officially supported. +- Beta: several known issues that may prevent expected use. +- Stable: ready for production use. You can also use the [API](../../api/packages.md) to administer the Package Registry. @@ -40,12 +51,12 @@ guides you through the process. | CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | | Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | | CRAN | [#36892](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | -| Debian | [Draft: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50438) | | Opkg | [#36894](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | | P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | | Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | -| RPM | [#5932](https://gitlab.com/gitlab-org/gitlab/-/issues/5932) | +| RPM | [#5932](https://gitlab.com/groups/gitlab-org/-/epics/5128) | | SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | +| Swift | [#12233](https://gitlab.com/gitlab-org/gitlab/-/issues/12233) | | Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | <!-- vale gitlab.Spelling = YES --> diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index fe7e6a0ea46..b07ae72e273 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -370,13 +370,26 @@ in a JavaScript project. You can install a package from the scope of a project o If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. -1. Set the URL for scoped packages by running: +1. Set the URL for scoped packages. + + For [instance-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: ```shell npm config set @foo:registry https://gitlab.example.com/api/v4/packages/npm/ ``` - Replace `@foo` with your scope. + - Replace `@foo` with your scope. + - Replace `gitlab.example.com` with your domain name. + + For [project-level endpoints](#use-the-gitlab-endpoint-for-npm-packages) run: + + ```shell + npm config set @foo:registry https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/ + ``` + + - Replace `@foo` with your scope. + - Replace `gitlab.example.com` with your domain name. + - Replace `<your_project_id>` with your project ID, found on the project's home page. 1. Ensure [authentication](#authenticate-to-the-package-registry) is configured. @@ -601,8 +614,3 @@ The GitLab npm repository supports the following commands for the npm CLI (`npm` - `npm view`: Show package metadata. - `yarn add`: Install an npm package. - `yarn update`: Update your dependencies. - -## Dependency Proxy - -The NPM Dependency Proxy for NPM packages isn't available. For more information, see -[this epic](https://gitlab.com/groups/gitlab-org/-/epics/3608). diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 40e8b74c2b6..2af6dc60078 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -77,6 +77,8 @@ To use the GitLab endpoint for NuGet Packages, choose an option: Some features such as [publishing](#publish-a-nuget-package) a package are only available on the project-level endpoint. +When asking for versions of a given NuGet package name, the GitLab Package Registry returns a maximum of 300 most recent versions. + WARNING: Because of how NuGet handles credentials, the Package Registry rejects anonymous requests on the group-level endpoint. To work around this limitation, set up [authentication](#add-the-package-registry-as-a-source-for-nuget-packages). @@ -352,12 +354,12 @@ the existing package is overwritten. ## Install packages -To install a NuGet package from the Package Registry, you must first -[add a project-level or group-level endpoint](#add-the-package-registry-as-a-source-for-nuget-packages). - If multiple packages have the same name and version, when you install a package, the most recently-published package is retrieved. +To install a NuGet package from the Package Registry, you must first +[add a project-level or group-level endpoint](#add-the-package-registry-as-a-source-for-nuget-packages). + ### Install a package with the NuGet CLI WARNING: diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index 2a0fea6e0ef..e31bd8bd0bf 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -9,8 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/803) in [GitLab Free](https://about.gitlab.com/pricing/) 13.10. WARNING: -The Ruby gems registry for GitLab is under development and isn't ready for production use due to -limited functionality. +The Ruby gems package registry for GitLab is under development and isn't ready for production use due to +limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/3200) details the remaining +work and timelines to make it production ready. You can publish Ruby gems in your project's Package Registry, then install the packages when you need to use them as a dependency. Although you can push gems to the registry, you cannot install diff --git a/doc/user/permissions.md b/doc/user/permissions.md index f240a9fd407..10147e7f69c 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -33,8 +33,6 @@ usernames. A GitLab administrator can configure the GitLab instance to ## Project members permissions -> The Master role was renamed to Maintainer in GitLab 11.0. - The Owner role is only available at the group or personal namespace level (and for instance administrators) and is inherited by its projects. While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner, or an instance administrator, who receives all permissions. For more information, see [projects members documentation](project/members/index.md). @@ -59,23 +57,23 @@ The following table lists project permissions available for each role: | [Application security](application_security/index.md):<br>View [threats list](application_security/threat_monitoring/index.md#threat-monitoring) **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) **(FREE SAAS)** | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) **(ULTIMATE)** | | | | | ✓ | -| [CI/CD](../ci/README.md):<br>Download and browse job artifacts | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>View a job log | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>View list of jobs | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>View [environments](../ci/environments/index.md) | | ✓ | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Cancel and retry jobs | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Create new [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Run CI/CD pipeline against a protected branch | | | ✓ (*5*) | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Stop [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>View a job with [debug logging](../ci/variables/index.md#debug-logging) | | | ✓ | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Manage CI/CD variables | | | | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Manage job triggers | | | | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Manage runners | | | | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Run Web IDE's Interactive Web Terminals **(ULTIMATE ONLY)** | | | | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Use [environment terminals](../ci/environments/index.md#web-terminals) | | | | ✓ | ✓ | -| [CI/CD](../ci/README.md):<br>Delete pipelines | | | | | ✓ | -| [Clusters](project/clusters/index.md):<br>View pod logs | | | ✓ | ✓ | ✓ | -| [Clusters](project/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Download and browse job artifacts | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>View a job log | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>View list of jobs | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>View [environments](../ci/environments/index.md) | | ✓ | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Cancel and retry jobs | | | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Create new [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Run CI/CD pipeline against a protected branch | | | ✓ (*5*) | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Stop [environments](../ci/environments/index.md) | | | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>View a job with [debug logging](../ci/variables/index.md#debug-logging) | | | ✓ | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Manage CI/CD variables | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Manage job triggers | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Manage runners | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Run Web IDE's Interactive Web Terminals **(ULTIMATE ONLY)** | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Use [environment terminals](../ci/environments/index.md#web-terminals) | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Delete pipelines | | | | | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | +| [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete cleanup policies | | | ✓ | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Remove a container registry image | | | ✓ | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Update container registry | | | ✓ | ✓ | ✓ | @@ -132,7 +130,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>View [wiki](project/wiki/index.md) pages | ✓ | ✓ | ✓ | ✓ | ✓ | | [Projects](project/index.md):<br>Create [snippets](snippets.md) | | ✓ | ✓ | ✓ | ✓ | | [Projects](project/index.md):<br>Manage labels | | ✓ | ✓ | ✓ | ✓ | -| [Projects](project/index.md):<br>View project statistics | | ✓ | ✓ | ✓ | ✓ | +| [Projects](project/index.md):<br>View [project traffic statistics](../api/project_statistics.md) | | ✓ | ✓ | ✓ | ✓ | | [Projects](project/index.md):<br>Create, edit, delete [milestones](project/milestones/index.md). | | | ✓ | ✓ | ✓ | | [Projects](project/index.md):<br>Create, edit, delete [releases](project/releases/index.md) | | | ✓ (*13*) | ✓ (*13*) | ✓ (*13*) | | [Projects](project/index.md):<br>Create, edit [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | @@ -269,9 +267,6 @@ Find the visibility permissions for the Container Registry, as described in the ## Group members permissions -NOTE: -In GitLab 11.0, the Master role was renamed to Maintainer. - Any user can remove themselves from a group, unless they are the last Owner of the group. @@ -434,7 +429,7 @@ mentioned in the [permissions table above](#project-members-permissions) (they are unable to browse the project's repository, for example). NOTE: -To prevent a guest user from creating projects, as an admin, you can edit the +To prevent a guest user from creating projects, as an administrator, you can edit the user's profile to mark the user as [external](#external-users). Beware though that even if a user is external, if they already have Reporter or higher permissions in any project or group, they are **not** counted as a @@ -442,8 +437,6 @@ free guest user. ## Auditor users **(PREMIUM SELF)** ->[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/998) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.17. - Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. @@ -454,7 +447,7 @@ with the permissions described on the documentation on [auditor users permission ## Users with minimal access **(PREMIUM)** ->[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40942) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40942) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. Owners can add members with a "minimal access" role to a parent group. Such users don't automatically have access to projects and subgroups underneath. To support such access, owners must explicitly add these "minimal access" users to the specific subgroups/projects. @@ -480,9 +473,6 @@ which visibility level you select on project settings. ## GitLab CI/CD permissions -NOTE: -In GitLab 11.0, the Master role was renamed to Maintainer. - GitLab CI/CD permissions rely on the role the user has in GitLab. There are four roles: @@ -513,13 +503,10 @@ instance and project. ### Job permissions -NOTE: -In GitLab 11.0, the Master role was renamed to Maintainer. - This table shows granted privileges for jobs triggered by specific types of users: -| Action | Guest, Reporter | Developer |Maintainer| Admin | +| Action | Guest, Reporter | Developer |Maintainer| Administrator | |---------------------------------------------|-----------------|-------------|----------|---------| | Run CI job | | ✓ | ✓ | ✓ | | Clone source and LFS from current project | | ✓ | ✓ | ✓ | @@ -534,8 +521,8 @@ users: | Push container images to other projects | | | | | | Push source and LFS | | | | | -1. Only if the user is not an external one -1. Only if the user is a member of the project +1. Only if the triggering user is not an external one +1. Only if the triggering user is a member of the project ## Running pipelines on protected branches @@ -555,7 +542,7 @@ for more information. ## LDAP users permissions -In GitLab 8.15 and later, LDAP user permissions can now be manually overridden by an admin user. +LDAP user permissions can be manually overridden by an administrator. Read through the documentation on [LDAP users permissions](group/index.md#manage-group-memberships-via-ldap) to learn more. ## Project aliases diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 3cc56cc47e6..ab0cae976d2 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can create users: -- Manually through the sign in page or Admin Area. +- Manually through the sign in page or Administrator Area. - Automatically through user authentication integrations. ## Create users on sign in page @@ -24,17 +24,17 @@ their own accounts by either: ## Create users in Admin Area -As an Admin user, you can manually create users: +As an Administrator user, you can manually create users: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users** (`/admin/users`). 1. Select **New user**. -You can also [create users through the API](../../../api/users.md) as an admin. +You can also [create users through the API](../../../api/users.md) as an administrator. - + - + ## Create users through authentication integrations diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 41b4e508c38..c555f5ca8cc 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -19,7 +19,7 @@ Deleting a user deletes all projects in that user namespace. As a user, to delete your own account: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. 1. Select **Delete account**. diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 44537707db6..6fe4b457fac 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -1,5 +1,4 @@ --- -type: howto stage: Manage group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -35,7 +34,8 @@ still access your account if you lose your U2F / WebAuthn device. ## Enabling 2FA -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35102) in GitLab 14.3, account email confirmation required. +> - Account email confirmation requirement [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35102) in GitLab 14.3. [Deployed behind the `ensure_verified_primary_email_for_2fa` flag](../../../administration/feature_flags.md), enabled by default. +> - Account email confirmation requirement generally available and [feature flag `ensure_verified_primary_email_for_2fa` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340151) in GitLab 14.4. There are multiple ways to enable two-factor authentication (2FA): @@ -44,11 +44,6 @@ There are multiple ways to enable two-factor authentication (2FA): In GitLab 14.3 and later, your account email must be confirmed to enable two-factor authentication. -FLAG: -On self-managed GitLab, account email confirmation requirement is enabled. To disable this -restriction, ask an administrator to -[disable the `ensure_verified_primary_email_for_2fa` flag](../../../administration/feature_flags.md). - ### One-time password To enable 2FA: @@ -239,8 +234,6 @@ Feature.disable(:forti_token_cloud, User.find(<user ID>)) ### U2F device -> Introduced in [GitLab 8.9](https://about.gitlab.com/blog/2016/06/22/gitlab-adds-support-for-u2f/). - GitLab officially only supports [YubiKey](https://www.yubico.com/products/) U2F devices, but users have successfully used [SoloKeys](https://solokeys.com/) or [Google Titan Security Key](https://cloud.google.com/titan-security-key). diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 25f349d9272..1797307a00c 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -9,14 +9,14 @@ type: howto > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17867) in GitLab 10.8. -GitLab lists all devices that have logged into your account. This allows you to +GitLab lists all devices that have logged into your account. You can review the sessions, and revoke any you don't recognize. ## Listing all active sessions To list all active sessions: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Active Sessions**. @@ -33,7 +33,7 @@ exceeds 100, the oldest ones are deleted. To revoke an active session: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Active Sessions**. 1. Select **Revoke** next to a session. The current session cannot be revoked, as this would sign you out of GitLab. diff --git a/doc/user/profile/img/notification_global_settings_v13_12.png b/doc/user/profile/img/notification_global_settings_v13_12.png Binary files differdeleted file mode 100644 index 0998bb89778..00000000000 --- a/doc/user/profile/img/notification_global_settings_v13_12.png +++ /dev/null diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 24006e6f875..7f16c4e244e 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -15,25 +15,25 @@ Your profile also includes settings, which you use to customize your GitLab expe To access your profile: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select your name or username. ## Access your user settings To access your user settings: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. ## Change your password To change your password: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Password**. -1. In the **Current password** field, enter your current password. -1. In the **New password** and **Password confirmation** field, enter your new password. +1. In the **Current password** text box, enter your current password. +1. In the **New password** and **Password confirmation** text box, enter your new password. 1. Select **Save password**. If you don't know your current password, select the **I forgot my password** link. @@ -53,7 +53,7 @@ Prerequisites: To change your username: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. 1. In the **Change username** section, enter a new username as the path. @@ -63,10 +63,10 @@ To change your username: To add new email to your account: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Emails**. -1. In the **Email** box, enter the new email. +1. In the **Email** text box, enter the new email. 1. Select **Add email address**. 1. Verify your email address with the verification email received. @@ -76,7 +76,7 @@ You can make your user profile visible to only you and GitLab administrators. To make your profile private: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. Select the **Private profile** checkbox. 1. Select **Update profile settings**. @@ -107,7 +107,7 @@ They can help other users connect with you on other platforms. To add links to other accounts: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. In the **Main settings** section, add your information from: - Skype @@ -121,7 +121,7 @@ In the user contribution calendar graph and recent activity list, you can see yo To show private contributions: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. In the **Main settings** section, select the **Include private contributions on my profile** checkbox. 1. Select **Update profile settings**. @@ -135,9 +135,9 @@ your name in your profile. To specify your pronouns: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. -1. In the **Pronouns** field, enter your pronouns. +1. In the **Pronouns** text box, enter your pronouns. 1. Select **Update profile settings**. ## Add your name pronunciation @@ -149,9 +149,9 @@ your name. To add your name pronunciation: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. -1. In the **Pronunciation** field, enter how your name is pronounced. +1. In the **Pronunciation** text box, enter how your name is pronounced. 1. Select **Update profile settings**. ## Set your current status @@ -165,11 +165,11 @@ Your status is publicly visible even if your [profile is private](#make-your-use To set your current status: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Set status** or, if you have already set a status, **Edit status**. 1. Set the desired emoji and status message. Status messages must be plain text and 100 characters or less. They can also contain emoji codes like, `I'm on vacation :palm_tree:`. -1. Select a value from the **Clear status after** dropdown. +1. Select a value from the **Clear status after** dropdown list. 1. Select **Set status**. Alternatively, you can select **Remove status** to remove your user status entirely. You can also set your current status by [using the API](../../api/users.md#user-status). @@ -188,12 +188,12 @@ To indicate to others that you are busy, you can set an indicator. To set the busy status indicator, either: - Set it directly: - 1. In the top-right corner, select your avatar. + 1. On the top bar, in the top-right corner, select your avatar. 1. Select **Set status** or, if you have already set a status, **Edit status**. 1. Select the **Busy** checkbox. - Set it on your profile: - 1. In the top-right corner, select your avatar. + 1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. In the **Current status** section, select the **Busy** checkbox. @@ -221,7 +221,7 @@ To set the busy status indicator, either: To set your time zone: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. 1. In the **Time settings** section, select your time zone from the dropdown list. @@ -236,7 +236,7 @@ To change your commit email: 1. In the top-right corner, select your avatar. 1. Select **Edit profile**. -1. In the **Commit email** list, select an email address. +1. In the **Commit email** dropdown list, select an email address. 1. Select **Update profile settings**. ### Use an automatically-generated private commit email @@ -246,9 +246,9 @@ so you can keep your email information private. To use a private commit email: -1. In the top-right corner, select your avatar. +1. On the top bar, in the top-right corner, select your avatar. 1. Select **Edit profile**. -1. In the **Commit email** list, select the **Use a private email** option. +1. In the **Commit email** dropdown list, select **Use a private email**. 1. Select **Update profile settings**. Every Git-related action uses the private commit email. diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index aaa311a4097..6de09f5538f 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -13,56 +13,62 @@ You can receive updates about activity in issues, merge requests, epics, and des For the tool that GitLab administrators can use to send messages to users, read [Email from GitLab](../../tools/email.md). -## Receive notifications +## Who receives notifications -You receive notifications for one of the following reasons: +When notifications are enabled for an issue, merge request, or epic, GitLab notifies you of actions +that happen there. -- You participate in an issue, merge request, epic, or design. In this context, _participate_ means - comment or edit. -- You've [enabled notifications in an issue, merge request, or epic](#notifications-on-issues-merge-requests-and-epics). -- You configured notifications at the [project](#project-notifications) and/or [group](#group-notifications) level. +You might receive notifications for one of the following reasons: -While notifications are enabled, you receive notification of actions occurring in that issue, merge request, or epic. +- You participate in an issue, merge request, epic, or design. You become a participant when you comment + or edit, or someone mentions you. +- You've [enabled notifications in an issue, merge request, or epic](#notifications-on-issues-merge-requests-and-epics). +- You've configured notifications for the [project](#change-level-of-project-notifications) or [group](#group-notifications). NOTE: Administrators can block notifications, preventing them from being sent. -## Tune your notifications +## Edit notification settings Getting many notifications can be overwhelming. You can tune the notifications you receive. For example, you might want to be notified about all activity in a specific project. For other projects, you only want to be notified when you are mentioned by name. -You can tune the notifications you receive by changing your notification settings: - -- [Global notification settings](#global-notification-settings) -- [Notification scope](#notification-scope) -- [Notification levels](#notification-levels) - -## Edit notification settings - These notification settings apply only to you. They do not affect the notifications received by -anyone else in the same project or group. +anyone else. To edit your notification settings: 1. In the top-right corner, select your avatar. 1. Select **Preferences**. 1. On the left sidebar, select **Notifications**. -1. Edit the desired notification settings. Edited settings are automatically saved and enabled. +1. Edit the desired global, group, or project notification settings. + Edited settings are automatically saved. -## Notification scope +### Notification scope -You can tune the scope of your notifications by selecting different notification levels for each project and group. +You can tune the scope of your notifications by selecting different notification levels for each +project and group. Notification scope is applied from the broadest to most specific levels: -- **Global (default)**: Your global, or _default_, notification level applies if you +- Your **global**, or _default_, notification level applies if you have not selected a notification level for the project or group in which the activity occurred. -- **Group**: For each group, you can select a notification level. Your group setting - overrides your default setting. -- **Project**: For each project, you can select a notification level. Your project - setting overrides the group setting. +- Your **group** setting overrides your default setting. +- Your **project** setting overrides the group setting. + +### Notification levels + +For each project and group you can select one of the following levels: + +| Level | Description | +| ----------- | ----------------------------------------------------------- | +| Global | Your global settings apply. | +| Watch | Receive notifications for any activity. | +| Participate | Receive notifications for threads you have participated in. | +| On mention | Receive notifications when you are [mentioned](../project/issues/issue_data_and_actions.md#mentions) in a comment. | +| Disabled | Receive no notifications. | +| Custom | Receive notifications for selected events. | ### Global notification settings @@ -72,19 +78,17 @@ different values for a project or a group. - **Notification email**: the email address your notifications are sent to. Defaults to your primary email address. - **Receive product marketing emails**: select this checkbox to receive - [periodic emails](#product-marketing-emails) about GitLab features. + [periodic emails](#opt-out-of-product-marketing-emails) about GitLab features. - **Global notification level**: the default [notification level](#notification-levels) which applies to all your notifications. - **Receive notifications about your own activity**: select this checkbox to receive notifications about your own activity. Not selected by default. - - ### Group notifications You can select a notification level and email address for each group. -#### Group notification level +#### Change level of group notifications To select a notification level for a group, use either of these methods: @@ -100,11 +104,12 @@ Or: 1. Select the notification dropdown, next to the bell icon (**{notifications}**). 1. Select the desired [notification level](#notification-levels). -#### Group notification email address +#### Change email address used for group notifications -> Introduced in GitLab 12.0 +> Introduced in GitLab 12.0. -You can select an email address to receive notifications for each group you belong to. This could be useful, for example, if you work freelance, and want to keep email about clients' projects separate. +You can select an email address to receive notifications for each group you belong to. +You can use group notifications, for example, if you work freelance, and want to keep email about clients' projects separate. 1. In the top-right corner, select your avatar. 1. Select **Preferences**. @@ -112,9 +117,9 @@ You can select an email address to receive notifications for each group you belo 1. Locate the project in the **Groups** section. 1. Select the desired email address. -### Project notifications +### Change level of project notifications -You can select a notification level for each project to help you closely monitor activity in select projects. +To help you stay up to date, you can select a notification level for each project. To select a notification level for a project, use either of these methods: @@ -131,28 +136,20 @@ Or: 1. Select the desired [notification level](#notification-levels). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -To learn how to be notified when a new release is available, see [Notification for releases](https://www.youtube.com/watch?v=qyeNkGgqmH4). +To learn how to be notified when a new release is available, watch [Notification for releases](https://www.youtube.com/watch?v=qyeNkGgqmH4). -### Notification levels - -For each project and group you can select one of the following levels: - -| Level | Description | -| ----------- | ----------------------------------------------------------- | -| Global | Your global settings apply. | -| Watch | Receive notifications for any activity. | -| On mention | Receive notifications when [mentioned](../project/issues/issue_data_and_actions.md#mentions) in a comment. | -| Participate | Receive notifications for threads you have participated in. | -| Disabled | Turns off notifications. | -| Custom | Receive notifications for custom selected events. | - -### Product marketing emails +### Opt out of product marketing emails You can receive emails that teach you about various GitLab features. -This is enabled by default. +These emails are enabled by default. -To opt out, [edit your notification settings](#edit-notification-settings) and clear the -**Receive product marketing emails** checkbox. +To opt out: + +1. In the top-right corner, select your avatar. +1. Select **Preferences**. +1. On the left sidebar, select **Notifications**. +1. Clear the **Receive product marketing emails** checkbox. + Edited settings are automatically saved and enabled. Disabling these emails does not disable all emails. Learn how to [opt out of all emails from GitLab](#opt-out-of-all-gitlab-emails). @@ -175,108 +172,115 @@ Users are notified of the following events: | Event | Sent to | Settings level | |------------------------------|---------------------|------------------------------| -| New release | Project members | Custom notification | -| Project moved | Project members (1) | (1) not disabled | +| New release | Project members | Custom notification. | +| Project moved | Project members | Any other than disabled. | | Email changed | User | Security email, always sent. | -| Group access level changed | User | Sent when user group access level is changed | +| Group access level changed | User | Sent when user group access level is changed. | | New email added | User | Security email, always sent. | | New SAML/SCIM user provisioned | User | Sent when a user is provisioned through SAML/SCIM. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276018) in GitLab 13.8 | | New SSH key added | User | Security email, always sent. | -| New user created | User | Sent on user creation, except for OmniAuth (LDAP)| -| Password changed | User | Security email, always sent when user changes their own password | -| Password changed by administrator | User | Security email, always sent when an administrator changes the password of another user | -| Personal access tokens expiring soon <!-- Do not delete or lint this instance of future tense --> | User | Security email, always sent. | +| New user created | User | Sent on user creation, except for OmniAuth (LDAP). | +| Password changed | User | Security email, always sent when user changes their own password. | +| Password changed by administrator | User | Security email, always sent when an administrator changes the password of another user. | +| Personal access tokens expiring soon | User | Security email, always sent. | | Personal access tokens have expired | User | Security email, always sent. | -| Project access level changed | User | Sent when user project access level is changed | -| SSH key has expired | User | Security email, always sent. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.12 | +| Project access level changed | User | Sent when user project access level is changed. | +| SSH key has expired | User | Security email, always sent. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322637) in GitLab 13.12._ | | Two-factor authentication disabled | User | Security email, always sent. | -| User added to group | User | Sent when user is added to group | -| User added to project | User | Sent when user is added to project | +| User added to group | User | Sent when user is added to group. | +| User added to project | User | Sent when user is added to project. | ## Notifications on issues, merge requests, and epics -To enable notifications on a specific issue, merge request, or epic, you must turn on the -**Notifications** toggle in the right sidebar. +You also receive notifications for events happening on +issues, merge requests, and epics. -- To subscribe, **turn on** if you are not a participant in the discussion, but want to receive - notifications on each update. -- To unsubscribe, **turn off** if you are receiving notifications for updates but no longer want to - receive them, unsubscribe from it. - -Turning this toggle off only unsubscribes you from updates related to this issue, merge request, or epic. -Learn how to [opt out of all emails from GitLab](#opt-out-of-all-gitlab-emails). - -Turning notifications on in an epic doesn't automatically subscribe you to the issues linked -to the epic. +### Who receives notifications on issues, merge requests, and epics -For most events, the notification is sent to: +In issues, merge requests, and epics, for most events, the notification is sent to: - Participants: - - The author and assignee of the issue/merge request. + - The author and assignee. - Authors of comments. - Anyone [mentioned](../project/issues/issue_data_and_actions.md#mentions) by username in the title - or description of the issue, merge request or epic. - - Anyone with notification level "Participating" or higher that is mentioned by their username in - any of the comments on the issue, merge request, or epic. + or description. + - Anyone mentioned by username in a comment if their notification level is "Participating" or higher. - Watchers: users with notification level "Watch". -- Subscribers: anyone who manually subscribed to the issue, merge request, or epic. -- Custom: Users with notification level "custom" who turned on notifications for any of the events in the following table. +- Subscribers: anyone who manually subscribed to notifications. +- Custom: users with notification level "Custom" who turned on notifications for a fitting type of events. NOTE: To minimize the number of notifications that do not require any action, in -[GitLab versions 12.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/616), eligible -approvers are no longer notified for all the activities in their projects. To receive them they have +[GitLab 12.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/616), eligible +approvers are no longer notified for all the activities in their projects. To turn on such notifications, they have to change their user notification settings to **Watch** instead. +### Edit notification settings for issues, merge requests, and epics + +To enable notifications on a specific issue, merge request, or epic, you must turn on the +**Notifications** toggle in the right sidebar. + +- To subscribe, **turn on** if you are not a participant in the discussion, but want to receive + notifications on each update. + + When you turn notifications on in an epic, you aren't automatically subscribed to the issues linked + to the epic. + +- To unsubscribe, **turn off** if you are receiving notifications for updates but no longer want to + receive them. + + Turning this toggle off only unsubscribes you from updates related to this issue, merge request, or epic. + Learn how to [opt out of all emails from GitLab](#opt-out-of-all-gitlab-emails). + +### Notification events on issues, merge requests, and epics + The following table presents the events that generate notifications for issues, merge requests, and epics: | Event | Sent to | |------------------------|---------| -| Change milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | -| Change milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | +| Change milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected. | +| Change milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected. | | Close epic | | | Close issue | | | Close merge request | | -| Due issue | Participants and Custom notification level with this event selected | -| Failed pipeline | The author of the pipeline | -| Fixed pipeline | The author of the pipeline. Enabled by default. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24309) in GitLab 13.1 | +| Due issue | Participants and Custom notification level with this event selected. | +| Failed pipeline | The author of the pipeline. | +| Fixed pipeline | The author of the pipeline. Enabled by default. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24309) in GitLab 13.1._ | | Merge merge request | | -| Merge when pipeline succeeds | Author, Participants, Watchers, Subscribers, and Custom notification level with this event selected. Custom notification level is ignored for Author, Watchers and Subscribers. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211961) in GitLab 13.4 | -| Merge request [marked as ready](../project/merge_requests/drafts.md) | Watchers and participants. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15332) in GitLab 13.10 | -| New comment | Participants, Watchers, Subscribers, and Custom notification level with this event selected, plus anyone mentioned by `@username` in the comment, with notification level "Mention" or higher | +| Merge when pipeline succeeds | Author, Participants, Watchers, Subscribers, and Custom notification level with this event selected. Custom notification level is ignored for Author, Watchers and Subscribers. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211961) in GitLab 13.4._ | +| Merge request [marked as ready](../project/merge_requests/drafts.md) | Watchers and participants. _[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15332) in GitLab 13.10._ | +| New comment | Participants, Watchers, Subscribers, and Custom notification level with this event selected. Also anyone mentioned by username in the comment, with notification level "Mention" or higher. | | New epic | | | New issue | | | New merge request | | -| Push to merge request | Participants and Custom notification level with this event selected | -| Reassign issue | Participants, Watchers, Subscribers, and Custom notification level with this event selected, plus the old assignee | -| Reassign merge request | Participants, Watchers, Subscribers, and Custom notification level with this event selected, plus the old assignee | -| Remove milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | -| Remove milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | +| Push to merge request | Participants and Custom notification level with this event selected. | +| Reassign issue | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old assignee. | +| Reassign merge request | Participants, Watchers, Subscribers, Custom notification level with this event selected, and the old assignee. | +| Remove milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected. | +| Remove milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected. | | Reopen epic | | | Reopen issue | | | Reopen merge request | | -| Successful pipeline | The author of the pipeline, if they have the custom notification setting for successful pipelines set. If the pipeline failed previously, a `Fixed pipeline` message is sent for the first successful pipeline after the failure, then a `Successful pipeline` message for any further successful pipelines. | +| Successful pipeline | The author of the pipeline, with Custom notification level for successful pipelines. If the pipeline failed previously, a "Fixed pipeline" message is sent for the first successful pipeline after the failure, and then a "Successful pipeline" message for any further successful pipelines. | If the title or description of an issue or merge request is -changed, notifications are sent to any **new** mentions by `@username` as +changed, notifications are sent to any **new** mentions by username as if they had been mentioned in the original text. -By default, you don't receive notifications for issues, merge requests, or epics created -by yourself. You only receive notifications when somebody else comments or adds changes to the ones -that you've created or mentions you, or when an issue is due soon. -To always receive notifications on your own issues and so on, you must turn on -[notifications about your own activity](#global-notification-settings). - If an open merge request becomes unmergeable due to conflict, its author is notified about the cause. If a user has also set the merge request to automatically merge when pipeline succeeds, then that user is also notified. +By default, you don't receive notifications for issues, merge requests, or epics created by yourself. +To always receive notifications on your own issues, merge requests, and so on, turn on +[notifications about your own activity](#global-notification-settings). + ## Notifications on designs > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217095) in GitLab 13.6. -Email notifications are sent to the participants when comments are made on a design. +Email notifications are sent to the participants when someone comments on a design. The participants are: @@ -288,7 +292,9 @@ The participants are: If you no longer wish to receive any email notifications: -1. [Go to the Notifications settings page.](#edit-notification-settings) +1. In the top-right corner, select your avatar. +1. Select **Preferences**. +1. On the left sidebar, select **Notifications**. 1. Clear the **Receive product marketing emails** checkbox. 1. Set your **Global notification level** to **Disabled**. 1. Clear the **Receive notifications about your own activity** checkbox. @@ -299,40 +305,43 @@ On self-managed installations, even after doing this, your instance administrato [can still email you](../../tools/email.md). To unsubscribe, select the unsubscribe link in one of these emails. -## Filter email +## Email headers you can use to filter email + +Notification email messages include GitLab-specific headers. To better manage your notifications, +you can filter the notification emails based on the content of these headers. -Notification email messages include GitLab-specific headers. You can filter the notification emails based on the content of these headers to better manage your notifications. For example, you could filter all emails for a specific project where you are being assigned either a merge request or issue. +For example, you could filter all emails from a specific project where you are being assigned a +a merge request or an issue. The following table lists all GitLab-specific email headers: | Header | Description | |------------------------------------|-------------------------------------------------------------------------| +| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. You can use it for email organization with filters. | +| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | +| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | | `X-GitLab-Group-Id` **(PREMIUM)** | The group's ID. Only present on notification emails for epics. | | `X-GitLab-Group-Path` **(PREMIUM)** | The group's path. Only present on notification emails for epics. | -| `X-GitLab-Project` | The name of the project the notification belongs to. | +| [`X-GitLab-NotificationReason`](#x-gitlab-notificationreason) | The reason for the notification. This can be `mentioned`, `assigned`, or `own_activity`. | +| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | | `X-GitLab-Project-Id` | The project's ID. | | `X-GitLab-Project-Path` | The project's path. | -| `X-GitLab-(Resource)-ID` | The ID of the resource the notification is for. The resource, for example, can be `Issue`, `MergeRequest`, `Commit`, or another such resource. | -| `X-GitLab-Discussion-ID` | The ID of the thread the comment belongs to, in notification emails for comments. | -| `X-GitLab-Pipeline-Id` | The ID of the pipeline the notification is for, in notification emails for pipelines. | +| `X-GitLab-Project` | The name of the project the notification belongs to. | | `X-GitLab-Reply-Key` | A unique token to support reply by email. | -| `X-GitLab-NotificationReason` | The reason for the notification. This can be `mentioned`, `assigned`, or `own_activity`. | -| `List-Id` | The path of the project in an RFC 2919 mailing list identifier. This is useful for email organization with filters, for example. | ### X-GitLab-NotificationReason -The `X-GitLab-NotificationReason` header contains the reason for the notification. The value is one of the following, in order of priority: +The `X-GitLab-NotificationReason` header contains the reason for the notification. +The value is one of the following, in order of priority: - `own_activity` - `assigned` - `mentioned` -The reason for the notification is also included in the footer of the notification email. For example an email with the -reason `assigned` has this sentence in the footer: +The reason for the notification is also included in the footer of the notification email. +For example, an email with the reason `assigned` has this sentence in the footer: -- `You are receiving this email because you have been assigned an item on <configured GitLab hostname>.` - -Notification of other events is being considered for inclusion in the `X-GitLab-NotificationReason` header. For details, see this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/20689). +> You are receiving this email because you have been assigned an item on \<configured GitLab hostname>. For example, an alert notification email can have one of [the alert's](../../operations/incident_management/alerts.md) statuses: @@ -341,3 +350,6 @@ For example, an alert notification email can have one of - `alert_acknowledged` - `alert_resolved` - `alert_ignored` + +Expanding the list of events included in the `X-GitLab-NotificationReason` header is tracked in +[issue 20689](https://gitlab.com/gitlab-org/gitlab/-/issues/20689). diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 64a375c6a1d..d54edc7e6d3 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -24,14 +24,16 @@ If you find that you have to add the same badges to several projects, you may wa To add a new badge to a project: -1. Navigate to your project's **Settings > General > Badges**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Badges**. 1. Under "Link", enter the URL that the badges should point to and under "Badge image URL" the URL of the image that should be displayed. -1. Submit the badge by clicking the **Add badge** button. +1. Select **Add badge**. After adding a badge to a project, you can see it in the list below the form. -You can edit it by clicking on the pen icon next to it or to delete it by -clicking on the trash icon. +You can edit the badge by selecting **Edit** (**{pencil}**) next to it or delete it by +selecting **Delete** (**{remove}**). Badges associated with a group can only be edited or deleted on the [group level](#group-badges). @@ -42,13 +44,15 @@ A common project badge presents the GitLab CI pipeline status. To add this badge to a project: -1. Navigate to your project's **Settings > General > Badges**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Badges**. 1. Under **Name**, enter _Pipeline Status_. 1. Under **Link**, enter the following URL: `https://gitlab.com/%{project_path}/-/commits/%{default_branch}` 1. Under **Badge image URL**, enter the following URL: `https://gitlab.com/%{project_path}/badges/%{default_branch}/pipeline.svg` -1. Submit the badge by clicking the **Add badge** button. +1. Select **Add badge**. ## Group badges @@ -60,14 +64,16 @@ project, consider adding them on the [project level](#project-badges) or use To add a new badge to a group: -1. Navigate to your group's **Settings > General > Badges**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand **Badges**. 1. Under "Link", enter the URL that the badges should point to and under "Badge image URL" the URL of the image that should be displayed. -1. Submit the badge by clicking the **Add badge** button. +1. Select **Add badge**. After adding a badge to a group, you can see it in the list below the form. -You can edit the badge by clicking on the pen icon next to it or to delete it -by clicking on the trash icon. +You can edit the badge by selecting **Edit** (**{pencil}**) next to it or delete it by +selecting **Delete** (**{remove}**). Badges directly associated with a project can be configured on the [project level](#project-badges). @@ -108,7 +114,8 @@ https://gitlab.example.com/<project_path>/-/raw/<default_branch>/my-image.svg To add a new badge to a group or project with a custom image: -1. Go to your group or project and select **Settings > General**. +1. On the top bar, select **Menu** and find your group or project. +1. On the left sidebar, select **Settings > General**. 1. Expand **Badges**. 1. Under **Name**, enter the name for the badge. 1. Under **Link**, enter the URL that the badge should point to. diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md index b4723294438..6d1fb0f6755 100644 --- a/doc/user/project/canary_deployments.md +++ b/doc/user/project/canary_deployments.md @@ -90,7 +90,7 @@ canary deployment is promoted to production. Here's an example setup flow from scratch: 1. Prepare an [Auto DevOps-enabled](../../topics/autodevops/index.md) project. -1. Set up a [Kubernetes Cluster](../../user/project/clusters/index.md) in your project. +1. Set up a [Kubernetes Cluster](../../user/infrastructure/clusters/index.md) in your project. 1. Install [NGINX Ingress](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) in your cluster. 1. Set up [the base domain](../../user/project/clusters/gitlab_managed_clusters.md#base-domain) based on the Ingress Endpoint assigned above. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index f7dd24fcfad..0db0f14b633 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -4,23 +4,37 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# EKS clusters (DEPRECATED) **(FREE)** +# Connect EKS clusters through cluster certificates **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22392) in GitLab 12.5. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22392) in GitLab 12.5. WARNING: -Use [Infrastructure as Code](../../infrastructure/index.md) to create new clusters. The method described in this document is deprecated as of GitLab 14.0. +Use [Infrastrucure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac) +to create new clusters. Through GitLab, you can create new clusters and add existing clusters hosted on Amazon Elastic Kubernetes Service (EKS). -## Add an existing EKS cluster +## Connect an existing EKS cluster -If you already have an EKS cluster and want to integrate it with GitLab, -see how to [add an existing cluster](add_existing_cluster.md). +If you already have an EKS cluster and want to connect it to GitLab, +use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). -## Create a new certificate-based EKS cluster +Alternatively, you can [connect them with cluster certificates](add_existing_cluster.md), +although this method is not recommended for [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). + +## Create a new EKS cluster + +To create a new cluster from GitLab, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). + +Alternatively, you can [create new EKS clusters using cluster certificates](#how-to-create-a-new-cluster-on-eks-through-cluster-certificates-deprecated). +Although still available on the GitLab UI, this method was deprecated +in GitLab 14.0 and is scheduled for removal in GitLab 15.0. +It also has [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). + +### How to create a new cluster on EKS through cluster certificates (DEPRECATED) + +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. Prerequisites: @@ -41,9 +55,10 @@ Further steps: 1. [Create a default Storage Class](#create-a-default-storage-class). 1. [Deploy the app to EKS](#deploy-the-app-to-eks). -### Create a new EKS cluster in GitLab +#### Create a new EKS cluster in GitLab -To create a new EKS cluster: +To create new a EKS cluster for your project, group, or instance, through +cluster certificates: 1. Go to your: - Project's **Infrastructure > Kubernetes clusters** page, for a project-level cluster. diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 505c493de4e..3347ef9a437 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -188,7 +188,7 @@ To add a Kubernetes cluster to your project, group, or instance: ``` 1. **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. - See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. + See the [Managed clusters section](gitlab_managed_clusters.md) for more information. 1. **Project namespace** (optional) - You don't have to fill this in. By leaving it blank, GitLab creates one for you. Also: - Each project should have a unique namespace. diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index 78d4bce737d..0d35e89a560 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -4,48 +4,55 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GKE clusters (DEPRECATED) **(FREE)** - -> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/6049) in GitLab 14.0. +# Connect GKE clusters through cluster certificates **(FREE)** WARNING: -Use [Infrastructure as Code](../../infrastructure/index.md) to create new clusters. The method described in this document is deprecated as of GitLab 14.0. +Use [Infrastrucure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md) +to create a cluster hosted on Google Kubernetes Engine (GKE). -Through GitLab, you can create new clusters and add existing clusters hosted on Amazon Elastic -Kubernetes Service (EKS). +Through GitLab, you can create new and connect existing clusters +hosted on Google Kubernetes Engine (GKE). -GitLab supports adding new and existing GKE clusters. +## Connect an existing GKE cluster -## GKE requirements +If you already have a GKE cluster and want to connect it to GitLab, +use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). -Before creating your first cluster on Google GKE with GitLab integration, make sure the following -requirements are met: +Alternatively, you can [connect them with cluster certificates](add_existing_cluster.md), +altough this method is not recommended for [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). -- A [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account) - set up with access. -- The Kubernetes Engine API and related service are enabled. It should work immediately but may - take up to 10 minutes after you create a project. For more information see the - ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). +## Create a new GKE cluster from GitLab + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25925) in GitLab 12.4, all the GKE clusters provisioned by GitLab are [VPC-native](https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips). + +To create a new GKE cluster from GitLab, use [Infrastructure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md). -## Add an existing GKE cluster +Alternatively, you can [create new GKE clusters using cluster certificates](#create-a-new-cluster-on-gke-through-cluster-certificates-deprecated). +Although still available in the GitLab UI, this method was deprecated +in GitLab 14.0 and is scheduled for removal in GitLab 15.0. +It also has [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). -If you already have a GKE cluster and want to integrate it with GitLab, -see how to [add an existing cluster](add_existing_cluster.md). +## Create a new cluster on GKE through cluster certificates (DEPRECATED) -## Create new GKE cluster +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/6049) in GitLab 14.0. -Starting from [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/25925), all the GKE clusters -provisioned by GitLab are [VPC-native](https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips). +Prerequisites: + +- A [Google Cloud billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account) + set up with access. +- Kubernetes Engine API and related services enabled. It should work immediately but may + take up to 10 minutes after you create a project. For more information see the + ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). Note the following: - The [Google authentication integration](../../../integration/google.md) must be enabled in GitLab at the instance level. If that's not the case, ask your GitLab administrator to enable it. On GitLab.com, this is enabled. -- Starting from [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55902), all GKE clusters +- In [GitLab 12.1](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55902) and later, all GKE clusters created by GitLab are RBAC-enabled. Take a look at the [RBAC section](cluster_access.md#rbac-cluster-resources) for more information. -- Starting from [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18341), the +- In [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18341) and later, the cluster's pod address IP range is set to `/16` instead of the regular `/14`. `/16` is a CIDR notation. - GitLab requires basic authentication enabled and a client certificate issued for the cluster to @@ -54,9 +61,8 @@ Note the following: explicitly requests GKE to create clusters with basic authentication enabled and a client certificate. -### Creating the cluster on GKE - -To create and add a new Kubernetes cluster to your project, group, or instance: +To create new Kubernetes clusters to your project, group, or instance, through +cluster certificates: 1. Navigate to your: - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level diff --git a/doc/user/project/clusters/cluster_access.md b/doc/user/project/clusters/cluster_access.md index 7bf202f6963..452f5727620 100644 --- a/doc/user/project/clusters/cluster_access.md +++ b/doc/user/project/clusters/cluster_access.md @@ -32,7 +32,7 @@ The resources created by GitLab differ depending on the type of cluster. Note the following about access controls: - Environment-specific resources are only created if your cluster is - [managed by GitLab](index.md#gitlab-managed-clusters). + [managed by GitLab](gitlab_managed_clusters.md). - If your cluster was created before GitLab 12.2, it uses a single namespace for all project environments. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 791dc90cad5..ac59f874244 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -4,77 +4,22 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Kubernetes clusters **(FREE)** +# Project-level Kubernetes clusters **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1 for projects. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in -> GitLab 11.6 for [groups](../../group/clusters/index.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in -> GitLab 11.11 for [instances](../../instance/clusters/index.md). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. -We offer extensive integrations to help you connect and manage your Kubernetes clusters from GitLab. +[Project-level Kubernetes clusters](../../infrastructure/clusters/connect/index.md#cluster-levels) +allow you to connect a Kubernetes cluster to a project in GitLab. -Read through this document to get started. +You can also [connect multiple clusters](multiple_kubernetes_clusters.md) +to a single project. -## Benefit from the GitLab-Kubernetes integration +After connecting a cluster to GitLab, you can benefit from the large number of +[GitLab features available for Kubernetes clusters](../../infrastructure/clusters/index.md) to manage and deploy to your cluster. -Using the GitLab-Kubernetes integration, you can benefit of GitLab -features such as: +## View your project-level clusters -- Create [CI/CD Pipelines](../../../ci/pipelines/index.md) to build, test, and deploy to your cluster. -- Use [Auto DevOps](#auto-devops) to automate the CI/CD process. -- Use [role-based or attribute-based access controls](cluster_access.md). -- Run serverless workloads on [Kubernetes with Knative](serverless/index.md). -- Connect GitLab to in-cluster applications using [cluster integrations](../../clusters/integrations.md). -- Use [deploy boards](../deploy_boards.md) to see the health and status of each CI [environment](../../../ci/environments/index.md) running on your Kubernetes cluster. -- Use [Canary deployments](../canary_deployments.md) to update only a portion of your fleet with the latest version of your application. -- View your [Kubernetes podlogs](kubernetes_pod_logs.md) directly in GitLab. -- Connect to your cluster through GitLab [web terminals](deploy_to_cluster.md#web-terminals-for-kubernetes-clusters). +To view project-level Kubernetes clusters: -## Supported cluster versions - -See the [Kubernetes clusters versions supported by GitLab](../../infrastructure/clusters/connect/index.md#supported-cluster-versions). - -## Connect your cluster to GitLab - -Learn how to [create new and connect existing clusters to GitLab](../../infrastructure/clusters/connect/index.md). - -## Cluster integrations - -See the available [cluster integrations](../../clusters/integrations.md) -to integrate third-party applications with your clusters through GitLab. - -## Cluster management project - -Attach a [Cluster management project](../../clusters/management_project.md) -to your cluster to manage shared resources requiring `cluster-admin` privileges for -installation, such as an Ingress controller. - -## GitLab-managed clusters - -See how to allow [GitLab to manage your cluster for you](gitlab_managed_clusters.md). - -## Auto DevOps - -You can use [Auto DevOps](../../../topics/autodevops/index.md) to automatically -detect, build, test, deploy, and monitor your applications. - -## Deploying to a Kubernetes cluster - -See how to [deploy to your Kubernetes cluster](deploy_to_cluster.md) from GitLab. - -## Monitoring your Kubernetes cluster - -Automatically detect and monitor Kubernetes metrics. Automatic monitoring of -[NGINX Ingress](../integrations/prometheus_library/nginx.md) is also supported. - -[Read more about Kubernetes monitoring](../integrations/prometheus_library/kubernetes.md) - -### Visualizing cluster health - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) to GitLab Free in 13.2. - -When [the Prometheus cluster integration is enabled](../../clusters/integrations.md#prometheus-cluster-integration), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. - - +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index 1940cf229b8..283e6c0b81c 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -57,7 +57,7 @@ to install Cilium in your Kubernetes cluster. ``` 1. Merge or push these changes to the default branch of your cluster management project, -and [GitLab CI/CD](../../../../../ci/README.md) will automatically install Cilium. +and [GitLab CI/CD](../../../../../ci/index.md) will automatically install Cilium. WARNING: Installation and removal of the Cilium requires a **manual** diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index 7357fc850e5..9d623518f72 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -63,8 +63,9 @@ information. Follow this step-by-step guide to configure an executable runbook in GitLab using the components outlined above and the pre-loaded demo runbook. -1. Create an [OAuth Application for JupyterHub](../../../../integration/oauth_provider.md#gitlab-as-an-oauth-20-authentication-service-provider). -1. When [installing JupyterHub with Helm](https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html), use the following values +1. Create an [OAuth application for JupyterHub](../../../../integration/oauth_provider.md). +1. When [installing JupyterHub with Helm](https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html), + use the following values: ```yaml #----------------------------------------------------------------------------- diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index fb32579f40e..f6598f8846b 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -282,7 +282,7 @@ Explanation of the fields used above: |-----------|-------------| | `name` | Indicates which provider is used to execute the `serverless.yml` file. In this case, the TriggerMesh middleware. | | `envs` | Includes the environment variables to be passed as part of function execution for **all** functions in the file, where `FOO` is the variable name and `BAR` are the variable contents. You may replace this with your own variables. | -| `secrets` | Includes the contents of the Kubernetes secret as environment variables accessible to be passed as part of function execution for **all** functions in the file. The secrets are expected in INI format. | +| `secrets` | Includes the contents of the Kubernetes secret as environment variables accessible to be passed as part of function execution for **all** functions in the file. The secrets are expected in `INI` format. | ### `functions` @@ -296,7 +296,7 @@ subsequent lines contain the function attributes. | `runtime` (optional)| The runtime to be used to execute the function. This can be a runtime alias (see [Runtime aliases](#runtime-aliases)), or it can be a full URL to a custom runtime repository. When the runtime is not specified, we assume that `Dockerfile` is present in the function directory specified by `source`. | | `description` | A short description of the function. | | `envs` | Sets an environment variable for the specific function only. | -| `secrets` | Includes the contents of the Kubernetes secret as environment variables accessible to be passed as part of function execution for the specific function only. The secrets are expected in INI format. | +| `secrets` | Includes the contents of the Kubernetes secret as environment variables accessible to be passed as part of function execution for the specific function only. The secrets are expected in `INI` format. | ### Deployment diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index 05f026cca18..6e2635b89f0 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -90,7 +90,7 @@ To display the deploy boards for a specific [environment](../../ci/environments/ 1. [Configure GitLab Runner](../../ci/runners/index.md) with the [`docker`](https://docs.gitlab.com/runner/executors/docker.html) or [`kubernetes`](https://docs.gitlab.com/runner/executors/kubernetes.html) executor. -1. Configure the [Kubernetes integration](clusters/index.md) in your project for the +1. Configure the [Kubernetes integration](../infrastructure/clusters/index.md) in your project for the cluster. The Kubernetes namespace is of particular note as you need it for your deployment scripts (exposed by the `KUBE_NAMESPACE` deployment variable). 1. Ensure Kubernetes annotations of `app.gitlab.com/env: $CI_ENVIRONMENT_SLUG` diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index e9a38f91677..61dccf1cb1b 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -10,9 +10,10 @@ type: howto, reference Deploy keys allow read-only or read-write access to your repositories by importing an SSH public key into your GitLab instance. -This is useful, for example, for cloning repositories to your Continuous -Integration (CI) server. By using deploy keys, you don't have to set up a -fake user account. +Deploy keys streamline interactions between your GitLab repository and another +machine. For example, setting up a deploy key allows secure cloning of your +repositories to a Continuous Integration (CI) server without setting up a fake +user account. There are two types of deploy keys: @@ -63,11 +64,12 @@ help you access a repository, but there are some notables differences between th - Deploy keys are shareable between projects that are not related or don't even belong to the same group. Deploy tokens belong to either a project or [a group](../deploy_tokens/index.md#group-deploy-token). -- A deploy key is an SSH key you need to generate yourself on your machine. A deploy - token is generated by your GitLab instance, and is provided to users only once - (at creation time). -- A deploy key is valid as long as it's registered and enabled. Deploy tokens can - be time-sensitive, as you can control their validity by setting an expiration date to them. +- A deploy key is an SSH key you generate on the **remote machine**. A deploy + token, on the other hand, is generated by your **GitLab instance**, and is + provided to users only once (at creation time). +- A deploy key is valid as long as it's registered and enabled. Deploy tokens + can be time-sensitive, as you can control their validity by setting an + expiration date to them. - You can't log in to a registry with deploy keys, or perform read / write operations on it, but this [is possible with deploy tokens](../deploy_tokens/index.md#gitlab-deploy-token). - You need an SSH key pair to use deploy keys, but not deploy tokens. @@ -115,9 +117,9 @@ project, and if you then update the access level for this key from `read-only` t ### Public deploy keys -Public deploy keys allow `read-only` or `read-write` -access to any repository in your GitLab instance. This is useful for integrating -repositories to secure, shared services, such as CI/CD. +Public deploy keys allow `read-only` or `read-write` access to any repository in +your GitLab instance. This allows for the integration of your repositories to +secure, shared services, such as CI/CD. Instance administrators can add public deploy keys: @@ -125,32 +127,37 @@ Instance administrators can add public deploy keys: 1. On the left sidebar, select **Deploy Keys**. 1. Select **New deploy key**. - Make sure your new key has a meaningful title, as it is the primary way for project - maintainers and owners to identify the correct public deploy key to add. For example, - if the key gives access to a SaaS CI/CD instance, use the name of that service - in the key name if that is all the key is used for. +Make sure your new key has a meaningful title. This title is the primary +way for project maintainers and owners to identify the correct public deploy key +to add to a repository or project. For example, the key you set up might be +intended to give access to a SaaS CI/CD instance. In this case use the name of +that service in the key title if that is all the key is used for.  -After adding a key, it's available to any shared systems. Project maintainers -or higher can [authorize a public deploy key](#project-deploy-keys) to start using it with the project. +After adding a key, it's available to any shared system. Users with a maintainer role or +higher can [authorize a public deploy key](#project-deploy-keys) to start using +it with the project. NOTE: -The **Publicly accessible deploy keys** tab within Project's CI/CD settings only appears -if there is at least one Public deploy key configured. +The **Publicly accessible deploy keys** tab in a Project's CI/CD +settings only appears if there is at least one Public deploy key configured. -Public deploy keys can provide greater security compared to project deploy keys, as -the administrator of the target integrated system is the only one who needs to know the key value, -or configure it. +Public deploy keys can provide greater security compared to project deploy keys. +This is because the administrator of the target integrated system is the only +entity who needs to know or configure the key value. -When creating a Public deploy key, determine whether or not it can be defined for -very narrow usage, such as just a specific service, or if it needs to be defined for -broader usage, such as full `read-write` access for all services. +When creating a Public deploy key, consider what scope and permissions are +required for it across the entire GitLab instance. For very narrow usage, such +as a single specific service, a `read-only` deploy key tied to this service is +best. If the service entails broader usage across the instance, a +deploy key with full `read-write` access is more appropriate. WARNING: -Adding a public deploy key does not immediately expose any repository to it. Public -deploy keys enable access from other systems, but access is not given to any project -until a project maintainer chooses to make use of it. +Adding a public deploy key **does not** immediately expose any repository +to the remote machine. Access to a project is only given when a project +maintainer chooses to make use of a deploy key in the project's +configuration. ## How to disable deploy keys @@ -162,22 +169,29 @@ can remove or disable a deploy key for a project repository: 1. Select the **{remove}** or **{cancel}** button. NOTE: -If anything relies on the removed deploy key, it will stop working once removed. +Any service that relies on a deploy key stops working after that key is removed. -If the key is **publicly accessible**, it will be removed from the project, but still available under **Publicly accessible deploy keys**. +If the key is **publicly accessible**, it is removed from the project, but can +still be found under **Publicly accessible deploy keys**. -If the key is **privately accessible** and only in use by this project, it will deleted. +If the key is **privately accessible** and only in use by this project, it is +deleted entirely from GitLab on removal. -If the key is **privately accessible** and in use by other projects, it will be removed from the project, but still available under **Privately accessible deploy keys**. +If the key is **privately accessible** and also in use by other projects, it is +removed from the project, but still available under **Privately accessible +deploy keys**. ## Troubleshooting ### Deploy key cannot push to a protected branch -If the owner of this deploy key doesn't have access to a [protected -branch](../protected_branches.md), then this deploy key doesn't have access to -the branch either. In addition to this, choosing the **No one** value in -[the "Allowed to push" section](../protected_branches.md#configure-a-protected-branch) -means that no users **and** no services using deploy keys can push to that selected branch. +There are a few scenarios where a deploy key will fail to push to a [protected +branch](../protected_branches.md). -Refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30769) for more information. +- The owner associated to a deploy key does not have access to the protected branch. +- The owner associated to a deploy key does not have [membership](../members/index.md) to the project of the protected branch. +- **No one** is selected in [the "Allowed to push" section](../protected_branches.md#configure-a-protected-branch) of the protected branch. + +All deploy keys are associated to an account. Since the permissions for an account can change, this might lead to scenarios where a deploy key that was working is suddenly unable to push to a protected branch. + +We recommend you create a service account, and associate a deploy key to the service account, for projects using deploy keys. diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 1798aa0c1c6..483de3b21bd 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -2,12 +2,10 @@ stage: Release group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Deploy tokens -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17894) in GitLab 10.7. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199370) from **Settings > Repository** in GitLab 12.9. > - [Added `write_registry` scope](https://gitlab.com/gitlab-org/gitlab/-/issues/22743) in GitLab 12.10. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29280) from **Settings > CI/CD** in GitLab 12.10.1. @@ -59,8 +57,8 @@ following table along with GitLab version it was introduced in: | Scope | Description | Introduced in GitLab Version | |--------------------------|-------------|------------------------------| -| `read_repository` | Allows read-access to the repository through `git clone` | 10.7 | -| `read_registry` | Allows read-access to [container registry](../../packages/container_registry/index.md) images if a project is private and authorization is required. | 10.7 | +| `read_repository` | Allows read-access to the repository through `git clone` | -- | +| `read_registry` | Allows read-access to [container registry](../../packages/container_registry/index.md) images if a project is private and authorization is required. | -- | | `write_registry` | Allows write-access (push) to [container registry](../../packages/container_registry/index.md). | 12.10 | | `read_package_registry` | Allows read access to the package registry. | 13.0 | | `write_package_registry` | Allows write access to the package registry. | 13.0 | @@ -185,8 +183,6 @@ To pull images from the Dependency Proxy, you must: ### GitLab deploy token -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18414) in GitLab 10.8. - There's a special case when it comes to deploy tokens. If a user creates one named `gitlab-deploy-token`, the username and token of the deploy token is automatically exposed to the CI/CD jobs as CI/CD variables: `CI_DEPLOY_USER` @@ -203,3 +199,18 @@ NOTE: The special handling for the `gitlab-deploy-token` deploy token is not implemented for group deploy tokens. To make the group-level deploy token available for CI/CD jobs, the `CI_DEPLOY_USER` and `CI_DEPLOY_PASSWORD` variables should be set under **Settings** to the name and token of the group deploy token respectively. + +## Troubleshooting + +### Group deploy tokens and LFS + +A bug +[prevents Group Deploy Tokens from cloning LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/235398). +If you receive `404 Not Found` errors and this error, +use a Project Deploy Token to work around the bug: + +```plaintext +api error: Repository or object not found: +https://<URL-with-token>.git/info/lfs/objects/batch +Check that it exists and that you have proper access to it +``` diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 5ffde38b348..db8c6f24063 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -49,7 +49,7 @@ This process allows you to lock single files or file extensions and it is done through the command line. It doesn't require GitLab paid subscriptions. Git LFS is well known for tracking files to reduce the storage of -Git repositories, but it can also be user for [locking files](https://github.com/git-lfs/git-lfs/wiki/File-Locking). +Git repositories, but it can also be used for [locking files](https://github.com/git-lfs/git-lfs/wiki/File-Locking). This is the method used for Exclusive File Locks. ### Install Git LFS diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 7ccdb632c19..2715804b37a 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -126,6 +126,6 @@ Feature.disable(:bitbucket_server_user_mapping_by_username) If the GUI-based import tool does not work, you can try to: - Use the [GitLab Import API](../../../api/import.md#import-repository-from-bitbucket-server) Bitbucket server endpoint. -- Set up [Repository Mirroring](../repository/repository_mirroring.md), which provides verbose error output. +- Set up [Repository Mirroring](../repository/mirror/index.md), which provides verbose error output. See the [troubleshooting](bitbucket.md#troubleshooting) section for [Bitbucket](bitbucket.md). diff --git a/doc/user/project/import/gitea.md b/doc/user/project/import/gitea.md index 9364ac4f954..3bbc70b4337 100644 --- a/doc/user/project/import/gitea.md +++ b/doc/user/project/import/gitea.md @@ -1,5 +1,4 @@ --- -type: reference, howto stage: Manage group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments @@ -10,30 +9,30 @@ info: To determine the technical writer assigned to the Stage/Group associated w Import your projects from Gitea to GitLab with minimal effort. NOTE: -This requires Gitea `v1.0.0` or newer. +This requires Gitea `v1.0.0` or later. The Gitea importer can import: -- Repository description (GitLab 8.15+) -- Git repository data (GitLab 8.15+) -- Issues (GitLab 8.15+) -- Pull requests (GitLab 8.15+) -- Milestones (GitLab 8.15+) -- Labels (GitLab 8.15+) +- Repository description +- Git repository data +- Issues +- Pull requests +- Milestones +- Labels When importing, repository public access is retained. If a repository is private in Gitea, it's created as private in GitLab as well. ## How it works -Since Gitea is currently not an OAuth provider, author/assignee cannot be mapped -to users in your GitLab instance. This means that the project creator (most of -the times the current user that started the import process) is set as the author, -but a reference on the issue about the original Gitea author is kept. +Because Gitea isn't an OAuth provider, author/assignee can't be mapped to users +in your GitLab instance. This means the project creator (usually the user that +started the import process) is set as the author. A reference, however, is kept +on the issue about the original Gitea author. -The importer creates any new namespaces (groups) if they don't exist or in -the case the namespace is taken, the repository is imported under the user's -namespace that started the import process. +The importer creates any new namespaces (groups) if they don't exist. If the +namespace is taken, the repository is imported under the user's namespace +that started the import process. ## Import your Gitea repositories @@ -41,7 +40,7 @@ The importer page is visible when you create a new project.  -Click the **Gitea** link and the import authorization process starts. +Select the **Gitea** link to start the import authorization process.  @@ -52,13 +51,13 @@ GitLab access your repositories: 1. Go to `https://your-gitea-instance/user/settings/applications` (replace `your-gitea-instance` with the host of your Gitea instance). -1. Click **Generate New Token**. +1. Select **Generate New Token**. 1. Enter a token description. -1. Click **Generate Token**. +1. Select **Generate Token**. 1. Copy the token hash. 1. Go back to GitLab and provide the token to the Gitea importer. -1. Hit the **List Your Gitea Repositories** button and wait while GitLab reads - your repositories' information. Once done, you are taken to the importer +1. Select **List Your Gitea Repositories** and wait while GitLab reads + your repositories' information. After it's done, GitLab displays the importer page to select the repositories to import. ### Select which repositories to import @@ -66,19 +65,19 @@ GitLab access your repositories: After you've authorized access to your Gitea repositories, you are redirected to the Gitea importer page. -From there, you can see the import statuses of your Gitea repositories. +From there, you can view the import statuses of your Gitea repositories: -- Those that are being imported show a _started_ status, -- those already successfully imported are green with a _done_ status, -- whereas those that are not yet imported have an **Import** button on the +- Those that are being imported show a _started_ status. +- Those already successfully imported are green with a _done_ status. +- Those that aren't yet imported have an **Import** button on the right side of the table. You also can: -- Import all your Gitea projects in one go by hitting **Import all projects** in - the upper left corner. -- Filter projects by name. If filter is applied, hitting **Import all projects** - only imports matched projects. +- Import all of your Gitea projects in one go by selecting **Import all projects** + in the upper left corner. +- Filter projects by name. If filter is applied, selecting **Import all projects** + imports only matched projects.  diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 4443ae902fb..eff733b0b3d 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -162,7 +162,7 @@ your GitHub repositories are listed. ## Mirror a repository and share pipeline status -Depending on your GitLab tier, [repository mirroring](../repository/repository_mirroring.md) can be set up to keep +Depending on your GitLab tier, [repository mirroring](../repository/mirror/index.md) can be set up to keep your imported repository in sync with its GitHub copy. Additionally, you can configure GitLab to send pipeline status updates back GitHub with the diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 65e1eafa477..887eb546148 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -47,7 +47,7 @@ information, see [the import notes](../settings/import_export.md#important-notes NOTE: When migrating to GitLab.com, you must create users manually unless [SCIM](../../../user/group/saml_sso/scim_setup.md) will be used. Creating users with the API is limited to self-managed instances as it requires -administrator access. +the Administrator role. To migrate all data from self-managed to GitLab.com, you can leverage the [API](../../../api/index.md). Migrate the assets in this order: diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 8c81af418a0..f3173736e9b 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Manage +group: Workspace info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- @@ -31,7 +31,7 @@ Projects include the following [features](https://about.gitlab.com/features/): from changing history or pushing code without review. - [Protected tags](protected_tags.md): Control who has permission to create tags and prevent accidental updates or deletions. - - [Repository mirroring](repository/repository_mirroring.md) + - [Repository mirroring](repository/mirror/index.md) - [Signing commits](repository/gpg_signed_commits/index.md): Use GNU Privacy Guard (GPG) to sign your commits. - [Deploy tokens](deploy_tokens/index.md): Manage access to the repository and Container Registry. - [Web IDE](web_ide/index.md) @@ -81,7 +81,7 @@ Projects include the following [features](https://about.gitlab.com/features/): browse, and download job artifacts. - [Pipeline settings](../../ci/pipelines/settings.md): Set up Git strategy (how jobs fetch your repository), timeout (the maximum amount of time a job can run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline visibility, and more. - - [Kubernetes cluster integration](clusters/index.md): Connect your GitLab project + - [Kubernetes cluster integration](../infrastructure/clusters/index.md): Connect your GitLab project with a Kubernetes cluster. - [Feature Flags](../../operations/feature_flags.md): Ship different features by dynamically toggling functionality. **(PREMIUM)** diff --git a/doc/user/project/integrations/asana.md b/doc/user/project/integrations/asana.md index e1e926da19b..963fca34827 100644 --- a/doc/user/project/integrations/asana.md +++ b/doc/user/project/integrations/asana.md @@ -4,9 +4,9 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Asana service **(FREE)** +# Asana integration **(FREE)** -This service adds commit messages as comments to Asana tasks. +This integration adds commit messages as comments to Asana tasks. Once enabled, commit messages are checked for Asana task URLs (for example, `https://app.asana.com/0/123456/987654`) or task IDs starting with `#` (for example, `#987654`). Every task ID found gets the commit comment added to it. @@ -23,7 +23,7 @@ You can use either of these words: - `closed` - `closing` -See also the [Asana service API documentation](../../../api/services.md#asana). +See also the [Asana integration API documentation](../../../api/integrations.md#asana). ## Setup diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 4908d21e764..3177aaefb75 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -18,7 +18,7 @@ and is automatically configured on [GitHub import](../../../integration/github.m ## Configuration -This integration requires a [GitHub API token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) +This integration requires a [GitHub API token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) with `repo:status` access granted. Complete these steps on GitHub: diff --git a/doc/user/project/integrations/img/slack_setup.png b/doc/user/project/integrations/img/slack_setup.png Binary files differindex 8acae659fb4..d8aedf84be5 100644 --- a/doc/user/project/integrations/img/slack_setup.png +++ b/doc/user/project/integrations/img/slack_setup.png diff --git a/doc/user/project/integrations/img/unify_circuit_configuration.png b/doc/user/project/integrations/img/unify_circuit_configuration.png Binary files differdeleted file mode 100644 index adba065347f..00000000000 --- a/doc/user/project/integrations/img/unify_circuit_configuration.png +++ /dev/null diff --git a/doc/user/project/integrations/img/webhook_logs.png b/doc/user/project/integrations/img/webhook_logs.png Binary files differindex 24bb593c7d0..05d068fd119 100644 --- a/doc/user/project/integrations/img/webhook_logs.png +++ b/doc/user/project/integrations/img/webhook_logs.png diff --git a/doc/user/project/integrations/img/webhook_testing.png b/doc/user/project/integrations/img/webhook_testing.png Binary files differindex acfebf473b9..27836556acc 100644 --- a/doc/user/project/integrations/img/webhook_testing.png +++ b/doc/user/project/integrations/img/webhook_testing.png diff --git a/doc/user/project/integrations/img/zentao_product_id.png b/doc/user/project/integrations/img/zentao_product_id.png Binary files differdeleted file mode 100644 index a91b4c3f82d..00000000000 --- a/doc/user/project/integrations/img/zentao_product_id.png +++ /dev/null diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 8027cc1c61e..1ff558b569b 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -16,7 +16,7 @@ separately configured [Mattermost Notifications Service](mattermost.md). ## Prerequisites -Mattermost [3.4 or later](https://mattermost.com/blog/category/releases/) is required. +Mattermost [3.4 or later](https://mattermost.com/blog/category/platform/releases/) is required. GitLab provides different methods of configuring Mattermost slash commands, depending on your configuration: diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index a6f739c6408..2c154467115 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -32,7 +32,7 @@ Click on the service links to see further configuration instructions and details | [Bugzilla](bugzilla.md) | Use Bugzilla as the issue tracker. | **{dotted-circle}** No | | Buildkite | Run CI/CD pipelines with Buildkite. | **{check-circle}** Yes | | Campfire | Connect to chat. | **{dotted-circle}** No | -| [Confluence Workspace](../../../api/services.md#confluence-service) | Replace the link to the internal wiki with a link to a Confluence Cloud Workspace. | **{dotted-circle}** No | +| [Confluence Workspace](../../../api/integrations.md#confluence-integration) | Replace the link to the internal wiki with a link to a Confluence Cloud Workspace. | **{dotted-circle}** No | | [Custom issue tracker](custom_issue_tracker.md) | Use a custom issue tracker. | **{dotted-circle}** No | | [Datadog](../../../integration/datadog.md) | Trace your GitLab pipelines with Datadog. | **{check-circle}** Yes | | [Discord Notifications](discord_notifications.md) | Send notifications about project events to a Discord channel. | **{dotted-circle}** No | @@ -40,7 +40,7 @@ Click on the service links to see further configuration instructions and details | [Emails on push](emails_on_push.md) | Send commits and diff of each push by email. | **{dotted-circle}** No | | [EWM](ewm.md) | Use IBM Engineering Workflow Management as the issue tracker. | **{dotted-circle}** No | | [External wiki](../wiki/index.md#link-an-external-wiki) | Link an external wiki. | **{dotted-circle}** No | -| [Flowdock](../../../api/services.md#flowdock) | Send notifications from GitLab to Flowdock flows. | **{dotted-circle}** No | +| [Flowdock](../../../api/integrations.md#flowdock) | Send notifications from GitLab to Flowdock flows. | **{dotted-circle}** No | | [GitHub](github.md) | Obtain statuses for commits and pull requests. | **{dotted-circle}** No | | [Google Chat](hangouts_chat.md) | Send notifications from your GitLab project to a room in Google Chat.| **{dotted-circle}** No | | [irker (IRC gateway)](irker.md) | Send IRC messages. | **{dotted-circle}** No | @@ -59,10 +59,9 @@ Click on the service links to see further configuration instructions and details | [Slack application](gitlab_slack_application.md) | Use Slack's official GitLab application. | **{dotted-circle}** No | | [Slack notifications](slack.md) | Send notifications about project events to Slack. | **{dotted-circle}** No | | [Slack slash commands](slack_slash_commands.md) | Enable slash commands in workspace. | **{dotted-circle}** No | -| [Unify Circuit](unify_circuit.md) | Receive events notifications. | **{dotted-circle}** No | +| [Unify Circuit](unify_circuit.md) | Send notifications about project events to Unify Circuit. | **{dotted-circle}** No | | [Webex Teams](webex_teams.md) | Receive events notifications. | **{dotted-circle}** No | | [YouTrack](youtrack.md) | Use YouTrack as the issue tracker. | **{dotted-circle}** No | -| [ZenTao](zentao.md) | Use ZenTao as the issue tracker. | **{dotted-circle}** No | ## Push hooks limit @@ -84,22 +83,7 @@ Read more about [Project integration management](../../admin_area/settings/proje ## Troubleshooting integrations -Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. GitLab stores details of service hook requests made within the last 2 days. To view details of the requests, go to that integration's configuration page. - -The **Recent Deliveries** section lists the details of each request made within the last 2 days: - -- HTTP status code (green for 200-299 codes, red for the others, `internal error` for failed deliveries) -- Triggered event -- URL to which the request was sent -- Elapsed time of the request -- Relative time in which the request was made - -To view more information about the request's execution, click the respective **View details** link. -On the details page, you can see the request headers and body sent and received by GitLab. - -To repeat a delivery using the same data, click **Resend Request**. - - +Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. Learn more about [troubleshooting service hooks](webhooks.md#troubleshoot-webhooks). ### Uninitialized repositories diff --git a/doc/user/project/integrations/pivotal_tracker.md b/doc/user/project/integrations/pivotal_tracker.md index d464007dd5e..93a3490e4b6 100644 --- a/doc/user/project/integrations/pivotal_tracker.md +++ b/doc/user/project/integrations/pivotal_tracker.md @@ -29,7 +29,7 @@ Read more about the [Source Commits endpoint](https://www.pivotaltracker.com/help/api/rest/v5#Source_Commits) in the Pivotal Tracker API documentation. -See also the [Pivotal Tracker service API documentation](../../../api/services.md#pivotal-tracker). +See also the [Pivotal Tracker integration API documentation](../../../api/integrations.md#pivotal-tracker). ## Set up Pivotal Tracker diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index acae0793e19..680f787c83c 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -75,7 +75,7 @@ service account can be found at Google's documentation for 1. (Optional) In **Google IAP Service Account JSON**, provide the contents of the Service Account credentials file that is authorized to access the Prometheus resource. The JSON key `token_credential_uri` is discarded to prevent - [Server-side Request Forgery (SSRF)](https://www.hackerone.com/blog-How-To-Server-Side-Request-Forgery-SSRF). + [Server-side Request Forgery (SSRF)](https://www.hackerone.com/application-security/how-server-side-request-forgery-ssrf). 1. Click **Save changes**.  diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index ea0119f2e94..429df7f7e27 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -12,7 +12,7 @@ GitLab has support for automatically detecting and monitoring Kubernetes metrics ## Requirements -The [Prometheus](../prometheus.md) and [Kubernetes](../../clusters/index.md) +The [Prometheus](../prometheus.md) and [Kubernetes](../../../infrastructure/clusters/index.md) integration services must be enabled. ## Metrics supported diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index d1fe58390fe..6478011b730 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -38,6 +38,8 @@ Managing these settings depends on how NGINX Ingress has been deployed. If you h ## Specifying the Environment label -In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do this, GitLab searches for metrics with appropriate labels. In this case, the `ingress` label must `<CI_ENVIRONMENT_SLUG>`. +To isolate and display only relevant metrics for a given environment, GitLab needs a method to +detect which labels are associated. To do this, GitLab searches for metrics with appropriate labels. +In this case, the `ingress` label must include the value `<CI_ENVIRONMENT_SLUG>`. If you have used [Auto Deploy](../../../../topics/autodevops/stages.md#auto-deploy) to deploy your app, this format is used automatically and metrics are detected with no action on your part. diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index 066a2f83753..cddb72a83b2 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Slack slash commands **(FREE SELF)** -> Introduced in GitLab 8.15. - If you want to control and view GitLab content while you're working in Slack, you can use Slack slash commands. To use Slack slash commands, you must configure both Slack and GitLab. diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index 2e166e87ff5..814c64d8140 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -6,29 +6,22 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Unify Circuit service **(FREE)** -The Unify Circuit service sends notifications from GitLab to the conversation for which the webhook was created. +The Unify Circuit service sends notifications from GitLab to a Circuit conversation. -## On Unify Circuit +## Set up Unify Circuit service -1. Open the conversation in which you want to see the notifications. -1. From the conversation menu, select **Configure Webhooks**. -1. Click **ADD WEBHOOK** and fill in the name of the bot to post the messages. Optionally - define an avatar. -1. Click **SAVE** and copy the **Webhook URL** of your webhook. +In Unify Circuit, [add a webhook](https://www.circuit.com/unifyportalfaqdetail?articleId=164448) and +copy its URL. -For more information, see the [Unify Circuit documentation for configuring incoming webhooks](https://www.circuit.com/unifyportalfaqdetail?articleId=164448). +In GitLab: -## On GitLab - -When you have the **Webhook URL** for your Unify Circuit conversation webhook, you can set up the GitLab service. - -1. Navigate to the [Integrations page](overview.md#accessing-integrations) in your project's settings, i.e. **Project > Settings > Integrations**. -1. Select the **Unify Circuit** integration to configure it. -1. Ensure that the **Active** toggle is enabled. -1. Check the checkboxes corresponding to the GitLab events you want to receive in Unify Circuit. +1. Go to the [Integrations page](overview.md#accessing-integrations) in your project's settings. +1. Select **Unify Circuit**. +1. Turn on the **Active** toggle. +1. Select the checkboxes corresponding to the GitLab events you want to receive in Unify Circuit. 1. Paste the **Webhook URL** that you copied from the Unify Circuit configuration step. -1. Configure the remaining options and click `Save changes`. - -Your Unify Circuit conversation now starts receiving GitLab event notifications as configured. +1. Select the **Notify only broken pipelines** checkbox to notify only on failures. +1. In the **Branches to be notified** dropdown, select which types of branches to send notifications for. +1. Select `Save changes` or optionally select **Test settings**. - +Your Unify Circuit conversation now starts receiving GitLab event notifications. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md new file mode 100644 index 00000000000..9b07e6322bc --- /dev/null +++ b/doc/user/project/integrations/webhook_events.md @@ -0,0 +1,1669 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Webhook events **(FREE)** + +You can configure a [webhook](webhooks.md) in your project that triggers when +an event occurs. The following events are supported. + +## Push events + +Triggered when you push to the repository except when pushing tags. + +NOTE: +When more than 20 commits are pushed at once, the `commits` webhook +attribute only contains the newest 20 for performance reasons. Loading +detailed commit data is expensive. Note that despite only 20 commits being +present in the `commits` attribute, the `total_commits_count` attribute contains the actual total. + +NOTE: +If a branch creation push event is generated without new commits being introduced, the +`commits` attribute in the payload is empty. + +Also, if a single push includes changes for more than three (by default, depending on +[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) +branches, this hook isn't executed. + +**Request header**: + +```plaintext +X-Gitlab-Event: Push Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "push", + "event_name": "push", + "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", + "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "ref": "refs/heads/master", + "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "user_id": 4, + "user_name": "John Smith", + "user_username": "jsmith", + "user_email": "john@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 15, + "project":{ + "id": 15, + "name":"Diaspora", + "description":"", + "web_url":"http://example.com/mike/diaspora", + "avatar_url":null, + "git_ssh_url":"git@example.com:mike/diaspora.git", + "git_http_url":"http://example.com/mike/diaspora.git", + "namespace":"Mike", + "visibility_level":0, + "path_with_namespace":"mike/diaspora", + "default_branch":"master", + "homepage":"http://example.com/mike/diaspora", + "url":"git@example.com:mike/diaspora.git", + "ssh_url":"git@example.com:mike/diaspora.git", + "http_url":"http://example.com/mike/diaspora.git" + }, + "repository":{ + "name": "Diaspora", + "url": "git@example.com:mike/diaspora.git", + "description": "", + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 + }, + "commits": [ + { + "id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", + "message": "Update Catalan translation to e38cb41.\n\nSee https://gitlab.com/gitlab-org/gitlab for more information", + "title": "Update Catalan translation to e38cb41.", + "timestamp": "2011-12-12T14:27:31+02:00", + "url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", + "author": { + "name": "Jordi Mallach", + "email": "jordi@softcatala.org" + }, + "added": ["CHANGELOG"], + "modified": ["app/controller/application.rb"], + "removed": [] + }, + { + "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "message": "fixed readme", + "title": "fixed readme", + "timestamp": "2012-01-03T23:36:29+02:00", + "url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "author": { + "name": "GitLab dev user", + "email": "gitlabdev@dv6700.(none)" + }, + "added": ["CHANGELOG"], + "modified": ["app/controller/application.rb"], + "removed": [] + } + ], + "total_commits_count": 4 +} +``` + +## Tag events + +Triggered when you create (or delete) tags to the repository. + +NOTE: +If a single push includes changes for more than three (by default, depending on +[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) +tags, this hook is not executed. + +**Request header**: + +```plaintext +X-Gitlab-Event: Tag Push Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "tag_push", + "event_name": "tag_push", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "ref": "refs/tags/v1.0.0", + "checkout_sha": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "user_id": 1, + "user_name": "John Smith", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project":{ + "id": 1, + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "repository":{ + "name": "Example", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + +## Issue events + +Triggered when a new issue is created or an existing issue was updated/closed/reopened. + +**Request header**: + +```plaintext +X-Gitlab-Event: Issue Hook +``` + +**Available `object_attributes.action`:** + +- `open` +- `close` +- `reopen` +- `update` + +**Payload example:** + +```json +{ + "object_kind": "issue", + "event_type": "issue", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project": { + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "ci_config_path": null, + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "object_attributes": { + "id": 301, + "title": "New API: create/update/delete file", + "assignee_ids": [51], + "assignee_id": 51, + "author_id": 51, + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "updated_by_id": 1, + "last_edited_at": null, + "last_edited_by_id": null, + "relative_position": 0, + "description": "Create new API for manipulations with repository", + "milestone_id": null, + "state_id": 1, + "confidential": false, + "discussion_locked": true, + "due_date": null, + "moved_to_id": null, + "duplicated_to_id": null, + "time_estimate": 0, + "total_time_spent": 0, + "time_change": 0, + "human_total_time_spent": null, + "human_time_estimate": null, + "human_time_change": null, + "weight": null, + "iid": 23, + "url": "http://example.com/diaspora/issues/23", + "state": "opened", + "action": "open", + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" + }, + "assignees": [{ + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }], + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + }, + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "changes": { + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current": "2017-09-15 16:52:00 UTC" + }, + "labels": { + "previous": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "current": [{ + "id": 205, + "title": "Platform", + "color": "#123123", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "Platform related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + } + } +} +``` + +NOTE: +`assignee` and `assignee_id` keys are deprecated and now show the first assignee only. + +## Comment events + +Triggered when a new comment is made on commits, merge requests, issues, and code snippets. +The note data is stored in `object_attributes` (for example, `note` or `noteable_type`). The +payload also includes information about the target of the comment. For example, +a comment on an issue includes the specific issue information under the `issue` key. +Valid target types: + +- `commit` +- `merge_request` +- `issue` +- `snippet` + +### Comment on commit + +**Request header**: + +```plaintext +X-Gitlab-Event: Note Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "note", + "event_type": "note", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project_id": 5, + "project":{ + "id": 5, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "repository":{ + "name": "Gitlab Test", + "url": "http://example.com/gitlab-org/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlab-org/gitlab-test" + }, + "object_attributes": { + "id": 1243, + "note": "This is a commit comment. How does this work?", + "noteable_type": "Commit", + "author_id": 1, + "created_at": "2015-05-17 18:08:09 UTC", + "updated_at": "2015-05-17 18:08:09 UTC", + "project_id": 5, + "attachment":null, + "line_code": "bec9703f7a456cd2b4ab5fb3220ae016e3e394e3_0_1", + "commit_id": "cfe32cf61b73a0d5e9f13e774abde7ff789b1660", + "noteable_id": null, + "system": false, + "st_diff": { + "diff": "--- /dev/null\n+++ b/six\n@@ -0,0 +1 @@\n+Subproject commit 409f37c4f05865e4fb208c771485f211a22c4c2d\n", + "new_path": "six", + "old_path": "six", + "a_mode": "0", + "b_mode": "160000", + "new_file": true, + "renamed_file": false, + "deleted_file": false + }, + "url": "http://example.com/gitlab-org/gitlab-test/commit/cfe32cf61b73a0d5e9f13e774abde7ff789b1660#note_1243" + }, + "commit": { + "id": "cfe32cf61b73a0d5e9f13e774abde7ff789b1660", + "message": "Add submodule\n\nSigned-off-by: Example User \u003cuser@example.com.com\u003e\n", + "timestamp": "2014-02-27T10:06:20+02:00", + "url": "http://example.com/gitlab-org/gitlab-test/commit/cfe32cf61b73a0d5e9f13e774abde7ff789b1660", + "author": { + "name": "Example User", + "email": "user@example.com" + } + } +} +``` + +### Comment on merge request + +**Request header**: + +```plaintext +X-Gitlab-Event: Note Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "note", + "event_type": "note", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project_id": 5, + "project":{ + "id": 5, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlab-org/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", + "namespace":"Gitlab Org", + "visibility_level":10, + "path_with_namespace":"gitlab-org/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlab-org/gitlab-test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "http_url":"http://example.com/gitlab-org/gitlab-test.git" + }, + "repository":{ + "name": "Gitlab Test", + "url": "http://localhost/gitlab-org/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlab-org/gitlab-test" + }, + "object_attributes": { + "id": 1244, + "note": "This MR needs work.", + "noteable_type": "MergeRequest", + "author_id": 1, + "created_at": "2015-05-17 18:21:36 UTC", + "updated_at": "2015-05-17 18:21:36 UTC", + "project_id": 5, + "attachment": null, + "line_code": null, + "commit_id": "", + "noteable_id": 7, + "system": false, + "st_diff": null, + "url": "http://example.com/gitlab-org/gitlab-test/merge_requests/1#note_1244" + }, + "merge_request": { + "id": 7, + "target_branch": "markdown", + "source_branch": "master", + "source_project_id": 5, + "author_id": 8, + "assignee_id": 28, + "title": "Tempora et eos debitis quae laborum et.", + "created_at": "2015-03-01 20:12:53 UTC", + "updated_at": "2015-03-21 18:27:27 UTC", + "milestone_id": 11, + "state": "opened", + "merge_status": "cannot_be_merged", + "target_project_id": 5, + "iid": 1, + "description": "Et voluptas corrupti assumenda temporibus. Architecto cum animi eveniet amet asperiores. Vitae numquam voluptate est natus sit et ad id.", + "position": 0, + "source":{ + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlab-org/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", + "namespace":"Gitlab Org", + "visibility_level":10, + "path_with_namespace":"gitlab-org/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlab-org/gitlab-test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "http_url":"http://example.com/gitlab-org/gitlab-test.git" + }, + "target": { + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlab-org/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", + "namespace":"Gitlab Org", + "visibility_level":10, + "path_with_namespace":"gitlab-org/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlab-org/gitlab-test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "http_url":"http://example.com/gitlab-org/gitlab-test.git" + }, + "last_commit": { + "id": "562e173be03b8ff2efb05345d12df18815438a4b", + "message": "Merge branch 'another-branch' into 'master'\n\nCheck in this test\n", + "timestamp": "2015-04-08T21: 00:25-07:00", + "url": "http://example.com/gitlab-org/gitlab-test/commit/562e173be03b8ff2efb05345d12df18815438a4b", + "author": { + "name": "John Smith", + "email": "john@example.com" + } + }, + "work_in_progress": false, + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + } + } +} +``` + +### Comment on issue + +**Request header**: + +```plaintext +X-Gitlab-Event: Note Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "note", + "event_type": "note", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project_id": 5, + "project":{ + "id": 5, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlab-org/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", + "namespace":"Gitlab Org", + "visibility_level":10, + "path_with_namespace":"gitlab-org/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlab-org/gitlab-test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "http_url":"http://example.com/gitlab-org/gitlab-test.git" + }, + "repository":{ + "name":"diaspora", + "url":"git@example.com:mike/diaspora.git", + "description":"", + "homepage":"http://example.com/mike/diaspora" + }, + "object_attributes": { + "id": 1241, + "note": "Hello world", + "noteable_type": "Issue", + "author_id": 1, + "created_at": "2015-05-17 17:06:40 UTC", + "updated_at": "2015-05-17 17:06:40 UTC", + "project_id": 5, + "attachment": null, + "line_code": null, + "commit_id": "", + "noteable_id": 92, + "system": false, + "st_diff": null, + "url": "http://example.com/gitlab-org/gitlab-test/issues/17#note_1241" + }, + "issue": { + "id": 92, + "title": "test", + "assignee_ids": [], + "assignee_id": null, + "author_id": 1, + "project_id": 5, + "created_at": "2015-04-12 14:53:17 UTC", + "updated_at": "2015-04-26 08:28:42 UTC", + "position": 0, + "branch_name": null, + "description": "test", + "milestone_id": null, + "state": "closed", + "iid": 17, + "labels": [ + { + "id": 25, + "title": "Afterpod", + "color": "#3e8068", + "project_id": null, + "created_at": "2019-06-05T14:32:20.211Z", + "updated_at": "2019-06-05T14:32:20.211Z", + "template": false, + "description": null, + "type": "GroupLabel", + "group_id": 4 + }, + { + "id": 86, + "title": "Element", + "color": "#231afe", + "project_id": 4, + "created_at": "2019-06-05T14:32:20.637Z", + "updated_at": "2019-06-05T14:32:20.637Z", + "template": false, + "description": null, + "type": "ProjectLabel", + "group_id": null + } + ] + } +} +``` + +NOTE: +`assignee_id` field is deprecated and now shows the first assignee only. + +NOTE: +`event_type` is set to `confidential_note` for confidential issues. + +### Comment on code snippet + +**Request header**: + +```plaintext +X-Gitlab-Event: Note Hook +``` + +**Payload example:** + +```json +{ + "object_kind": "note", + "event_type": "note", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project_id": 5, + "project":{ + "id": 5, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlab-org/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", + "namespace":"Gitlab Org", + "visibility_level":10, + "path_with_namespace":"gitlab-org/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlab-org/gitlab-test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", + "http_url":"http://example.com/gitlab-org/gitlab-test.git" + }, + "repository":{ + "name":"Gitlab Test", + "url":"http://example.com/gitlab-org/gitlab-test.git", + "description":"Aut reprehenderit ut est.", + "homepage":"http://example.com/gitlab-org/gitlab-test" + }, + "object_attributes": { + "id": 1245, + "note": "Is this snippet doing what it's supposed to be doing?", + "noteable_type": "Snippet", + "author_id": 1, + "created_at": "2015-05-17 18:35:50 UTC", + "updated_at": "2015-05-17 18:35:50 UTC", + "project_id": 5, + "attachment": null, + "line_code": null, + "commit_id": "", + "noteable_id": 53, + "system": false, + "st_diff": null, + "url": "http://example.com/gitlab-org/gitlab-test/snippets/53#note_1245" + }, + "snippet": { + "id": 53, + "title": "test", + "content": "puts 'Hello world'", + "author_id": 1, + "project_id": 5, + "created_at": "2015-04-09 02:40:38 UTC", + "updated_at": "2015-04-09 02:40:38 UTC", + "file_name": "test.rb", + "expires_at": null, + "type": "ProjectSnippet", + "visibility_level": 0 + } +} +``` + +## Merge request events + +Triggered when a new merge request is created, an existing merge request was updated/merged/closed or a commit is added in the source branch. + +**Request header**: + +```plaintext +X-Gitlab-Event: Merge Request Hook +``` + +**Available `object_attributes.action`:** + +- `open` +- `close` +- `reopen` +- `update` +- `approved` +- `unapproved` +- `merge` + +**Payload example:** + +```json +{ + "object_kind": "merge_request", + "event_type": "merge_request", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project": { + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" + }, + "object_attributes": { + "id": 99, + "target_branch": "master", + "source_branch": "ms-viewport", + "source_project_id": 14, + "author_id": 51, + "assignee_id": 6, + "title": "MS-Viewport", + "created_at": "2013-12-03T17:23:34Z", + "updated_at": "2013-12-03T17:23:34Z", + "milestone_id": null, + "state": "opened", + "merge_status": "unchecked", + "target_project_id": 14, + "iid": 1, + "description": "", + "source": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "target": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "last_commit": { + "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "message": "fixed readme", + "timestamp": "2012-01-03T23:36:29+02:00", + "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "author": { + "name": "GitLab dev user", + "email": "gitlabdev@dv6700.(none)" + } + }, + "work_in_progress": false, + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open", + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + } + }, + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "changes": { + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, + "labels": { + "previous": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "current": [{ + "id": 205, + "title": "Platform", + "color": "#123123", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "Platform related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + } + } +} +``` + +## Wiki Page events + +Triggered when a wiki page is created, updated or deleted. + +**Request Header**: + +```plaintext +X-Gitlab-Event: Wiki Page Hook +``` + +**Payload example**: + +```json +{ + "object_kind": "wiki_page", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "project": { + "id": 1, + "name": "awesome-project", + "description": "This is awesome", + "web_url": "http://example.com/root/awesome-project", + "avatar_url": null, + "git_ssh_url": "git@example.com:root/awesome-project.git", + "git_http_url": "http://example.com/root/awesome-project.git", + "namespace": "root", + "visibility_level": 0, + "path_with_namespace": "root/awesome-project", + "default_branch": "master", + "homepage": "http://example.com/root/awesome-project", + "url": "git@example.com:root/awesome-project.git", + "ssh_url": "git@example.com:root/awesome-project.git", + "http_url": "http://example.com/root/awesome-project.git" + }, + "wiki": { + "web_url": "http://example.com/root/awesome-project/-/wikis/home", + "git_ssh_url": "git@example.com:root/awesome-project.wiki.git", + "git_http_url": "http://example.com/root/awesome-project.wiki.git", + "path_with_namespace": "root/awesome-project.wiki", + "default_branch": "master" + }, + "object_attributes": { + "title": "Awesome", + "content": "awesome content goes here", + "format": "markdown", + "message": "adding an awesome page to the wiki", + "slug": "awesome", + "url": "http://example.com/root/awesome-project/-/wikis/awesome", + "action": "create" + } +} +``` + +## Pipeline events + +In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53159) +and later, the pipeline webhook returns only the latest jobs. + +Triggered on status change of Pipeline. + +**Request Header**: + +```plaintext +X-Gitlab-Event: Pipeline Hook +``` + +**Payload example**: + +```json +{ + "object_kind": "pipeline", + "object_attributes":{ + "id": 31, + "ref": "master", + "tag": false, + "sha": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", + "before_sha": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", + "source": "merge_request_event", + "status": "success", + "stages":[ + "build", + "test", + "deploy" + ], + "created_at": "2016-08-12 15:23:28 UTC", + "finished_at": "2016-08-12 15:26:29 UTC", + "duration": 63, + "variables": [ + { + "key": "NESTOR_PROD_ENVIRONMENT", + "value": "us-west-1" + } + ] + }, + "merge_request": { + "id": 1, + "iid": 1, + "title": "Test", + "source_branch": "test", + "source_project_id": 1, + "target_branch": "master", + "target_project_id": 1, + "state": "opened", + "merge_status": "can_be_merged", + "url": "http://192.168.64.1:3005/gitlab-org/gitlab-test/merge_requests/1" + }, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "user_email@gitlab.com" + }, + "project":{ + "id": 1, + "name": "Gitlab Test", + "description": "Atque in sunt eos similique dolores voluptatem.", + "web_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test", + "avatar_url": null, + "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", + "git_http_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test.git", + "namespace": "Gitlab Org", + "visibility_level": 20, + "path_with_namespace": "gitlab-org/gitlab-test", + "default_branch": "master" + }, + "commit":{ + "id": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", + "message": "test\n", + "timestamp": "2016-08-12T17:23:21+02:00", + "url": "http://example.com/gitlab-org/gitlab-test/commit/bcbb5ec396a2c0f828686f14fac9b80b780504f2", + "author":{ + "name": "User", + "email": "user@gitlab.com" + } + }, + "builds":[ + { + "id": 380, + "stage": "deploy", + "name": "production", + "status": "skipped", + "created_at": "2016-08-12 15:23:28 UTC", + "started_at": null, + "finished_at": null, + "when": "manual", + "manual": true, + "allow_failure": false, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "runner": null, + "artifacts_file":{ + "filename": null, + "size": null + }, + "environment": { + "name": "production", + "action": "start", + "deployment_tier": "production" + } + }, + { + "id": 377, + "stage": "test", + "name": "test-image", + "status": "success", + "created_at": "2016-08-12 15:23:28 UTC", + "started_at": "2016-08-12 15:26:12 UTC", + "finished_at": null, + "when": "on_success", + "manual": false, + "allow_failure": false, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "runner": { + "id": 380987, + "description": "shared-runners-manager-6.gitlab.com", + "active": true, + "runner_type": "instance_type", + "is_shared": true, + "tags": [ + "linux", + "docker", + "shared-runner" + ] + }, + "artifacts_file":{ + "filename": null, + "size": null + }, + "environment": null + }, + { + "id": 378, + "stage": "test", + "name": "test-build", + "status": "success", + "created_at": "2016-08-12 15:23:28 UTC", + "started_at": "2016-08-12 15:26:12 UTC", + "finished_at": "2016-08-12 15:26:29 UTC", + "when": "on_success", + "manual": false, + "allow_failure": false, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "runner": { + "id":380987, + "description":"shared-runners-manager-6.gitlab.com", + "active":true, + "runner_type": "instance_type", + "is_shared": true, + "tags": [ + "linux", + "docker" + ] + }, + "artifacts_file":{ + "filename": null, + "size": null + }, + "environment": null + }, + { + "id": 376, + "stage": "build", + "name": "build-image", + "status": "success", + "created_at": "2016-08-12 15:23:28 UTC", + "started_at": "2016-08-12 15:24:56 UTC", + "finished_at": "2016-08-12 15:25:26 UTC", + "when": "on_success", + "manual": false, + "allow_failure": false, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "runner": { + "id": 380987, + "description": "shared-runners-manager-6.gitlab.com", + "active": true, + "runner_type": "instance_type", + "is_shared": true, + "tags": [ + "linux", + "docker" + ] + }, + "artifacts_file":{ + "filename": null, + "size": null + }, + "environment": null + }, + { + "id": 379, + "stage": "deploy", + "name": "staging", + "status": "created", + "created_at": "2016-08-12 15:23:28 UTC", + "started_at": null, + "finished_at": null, + "when": "on_success", + "manual": false, + "allow_failure": false, + "user":{ + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + "email": "admin@example.com" + }, + "runner": null, + "artifacts_file":{ + "filename": null, + "size": null + }, + "environment": { + "name": "staging", + "action": "start", + "deployment_tier": "staging" + } + } + ] +} +``` + +## Job events + +Triggered on status change of a job. + +**Request Header**: + +```plaintext +X-Gitlab-Event: Job Hook +``` + +**Payload example**: + +```json +{ + "object_kind": "build", + "ref": "gitlab-script-trigger", + "tag": false, + "before_sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", + "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", + "build_id": 1977, + "build_name": "test", + "build_stage": "test", + "build_status": "created", + "build_created_at": "2021-02-23T02:41:37.886Z", + "build_started_at": null, + "build_finished_at": null, + "build_duration": null, + "build_allow_failure": false, + "build_failure_reason": "script_failure", + "pipeline_id": 2366, + "project_id": 380, + "project_name": "gitlab-org/gitlab-test", + "user": { + "id": 3, + "name": "User", + "email": "user@gitlab.com", + "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", + }, + "commit": { + "id": 2366, + "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", + "message": "test\n", + "author_name": "User", + "author_email": "user@gitlab.com", + "status": "created", + "duration": null, + "started_at": null, + "finished_at": null + }, + "repository": { + "name": "gitlab_test", + "description": "Atque in sunt eos similique dolores voluptatem.", + "homepage": "http://192.168.64.1:3005/gitlab-org/gitlab-test", + "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", + "git_http_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test.git", + "visibility_level": 20 + }, + "runner": { + "active": true, + "runner_type": "project_type", + "is_shared": false, + "id": 380987, + "description": "shared-runners-manager-6.gitlab.com", + "tags": [ + "linux", + "docker" + ] + }, + "environment": null +} +``` + +Note that `commit.id` is the ID of the pipeline, not the ID of the commit. + +## Deployment events + +Triggered when a deployment: + +- Starts ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41214) in GitLab 13.5.) +- Succeeds +- Fails +- Is cancelled + +**Request Header**: + +```plaintext +X-Gitlab-Event: Deployment Hook +``` + +**Payload example**: + +```json +{ + "object_kind": "deployment", + "status": "success", + "status_changed_at":"2021-04-28 21:50:00 +0200", + "deployment_id": 15, + "deployable_id": 796, + "deployable_url": "http://10.126.0.2:3000/root/test-deployment-webhooks/-/jobs/796", + "environment": "staging", + "project": { + "id": 30, + "name": "test-deployment-webhooks", + "description": "", + "web_url": "http://10.126.0.2:3000/root/test-deployment-webhooks", + "avatar_url": null, + "git_ssh_url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", + "git_http_url": "http://10.126.0.2:3000/root/test-deployment-webhooks.git", + "namespace": "Administrator", + "visibility_level": 0, + "path_with_namespace": "root/test-deployment-webhooks", + "default_branch": "master", + "ci_config_path": "", + "homepage": "http://10.126.0.2:3000/root/test-deployment-webhooks", + "url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", + "ssh_url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", + "http_url": "http://10.126.0.2:3000/root/test-deployment-webhooks.git" + }, + "short_sha": "279484c0", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "email": "admin@example.com" + }, + "user_url": "http://10.126.0.2:3000/root", + "commit_url": "http://10.126.0.2:3000/root/test-deployment-webhooks/-/commit/279484c09fbe69ededfced8c1bb6e6d24616b468", + "commit_title": "Add new file" +} +``` + +Note that `deployable_id` is the ID of the CI job. + +## Group member events **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260347) in GitLab 13.7. + +Member events are triggered when: + +- A user is added as a group member +- The access level of a user has changed +- The expiration date for user access has been updated +- A user has been removed from the group + +### Add member to group + +**Request Header**: + +```plaintext +X-Gitlab-Event: Member Hook +``` + +**Payload example**: + +```json +{ + "created_at": "2020-12-11T04:57:22Z", + "updated_at": "2020-12-11T04:57:22Z", + "group_name": "webhook-test", + "group_path": "webhook-test", + "group_id": 100, + "user_username": "test_user", + "user_name": "Test User", + "user_email": "testuser@webhooktest.com", + "user_id": 64, + "group_access": "Guest", + "group_plan": null, + "expires_at": "2020-12-14T00:00:00Z", + "event_name": "user_add_to_group" +} +``` + +### Update member access level or expiration date + +**Request Header**: + +```plaintext +X-Gitlab-Event: Member Hook +``` + +**Payload example**: + +```json +{ + "created_at": "2020-12-11T04:57:22Z", + "updated_at": "2020-12-12T08:48:19Z", + "group_name": "webhook-test", + "group_path": "webhook-test", + "group_id": 100, + "user_username": "test_user", + "user_name": "Test User", + "user_email": "testuser@webhooktest.com", + "user_id": 64, + "group_access": "Developer", + "group_plan": null, + "expires_at": "2020-12-20T00:00:00Z", + "event_name": "user_update_for_group" +} +``` + +### Remove member from group + +**Request Header**: + +```plaintext +X-Gitlab-Event: Member Hook +``` + +**Payload example**: + +```json +{ + "created_at": "2020-12-11T04:57:22Z", + "updated_at": "2020-12-12T08:52:34Z", + "group_name": "webhook-test", + "group_path": "webhook-test", + "group_id": 100, + "user_username": "test_user", + "user_name": "Test User", + "user_email": "testuser@webhooktest.com", + "user_id": 64, + "group_access": "Guest", + "group_plan": null, + "expires_at": "2020-12-14T00:00:00Z", + "event_name": "user_remove_from_group" +} +``` + +## Subgroup events **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260419) in GitLab 13.9. + +Subgroup events are triggered when: + +- A [subgroup is created in a group](#subgroup-created-in-a-group) +- A [subgroup is removed from a group](#subgroup-removed-from-a-group) + +### Subgroup created in a group + +**Request Header**: + +```plaintext +X-Gitlab-Event: Subgroup Hook +``` + +**Payload example**: + +```json +{ + + "created_at": "2021-01-20T09:40:12Z", + "updated_at": "2021-01-20T09:40:12Z", + "event_name": "subgroup_create", + "name": "subgroup1", + "path": "subgroup1", + "full_path": "group1/subgroup1", + "group_id": 10, + "parent_group_id": 7, + "parent_name": "group1", + "parent_path": "group1", + "parent_full_path": "group1" + +} +``` + +### Subgroup removed from a group + +**Request Header**: + +```plaintext +X-Gitlab-Event: Subgroup Hook +``` + +**Payload example**: + +```json +{ + + "created_at": "2021-01-20T09:40:12Z", + "updated_at": "2021-01-20T09:40:12Z", + "event_name": "subgroup_destroy", + "name": "subgroup1", + "path": "subgroup1", + "full_path": "group1/subgroup1", + "group_id": 10, + "parent_group_id": 7, + "parent_name": "group1", + "parent_path": "group1", + "parent_full_path": "group1" + +} +``` + +NOTE: +Webhooks for when a [subgroup is removed from a group](#subgroup-removed-from-a-group) are not triggered when a [subgroup is transferred to a new parent group](../../group/index.md#transfer-a-group) + +## Feature Flag events + +Triggered when a feature flag is turned on or off. + +**Request Header**: + +```plaintext +X-Gitlab-Event: Feature Flag Hook +``` + +**Payload example**: + +```json +{ + "object_kind": "feature_flag", + "project": { + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "ci_config_path": null, + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "email": "admin@example.com" + }, + "user_url": "http://example.com/root", + "object_attributes": { + "id": 6, + "name": "test-feature-flag", + "description": "test-feature-flag-description", + "active": true + } +} +``` + +## Release events + +Triggered when a release is created or updated. + +**Request Header**: + +```plaintext +X-Gitlab-Event: Release Hook +``` + +**Available `object_attributes.action`:** + +- `create` +- `update` + +**Payload example**: + +```json +{ + "id": 1, + "created_at": "2020-11-02 12:55:12 UTC", + "description": "v1.0 has been released", + "name": "v1.1", + "released_at": "2020-11-02 12:55:12 UTC", + "tag": "v1.1", + "object_kind": "release", + "project": { + "id": 2, + "name": "release-webhook-example", + "description": "", + "web_url": "https://example.com/gitlab-org/release-webhook-example", + "avatar_url": null, + "git_ssh_url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", + "git_http_url": "https://example.com/gitlab-org/release-webhook-example.git", + "namespace": "Gitlab", + "visibility_level": 0, + "path_with_namespace": "gitlab-org/release-webhook-example", + "default_branch": "master", + "ci_config_path": null, + "homepage": "https://example.com/gitlab-org/release-webhook-example", + "url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", + "ssh_url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", + "http_url": "https://example.com/gitlab-org/release-webhook-example.git" + }, + "url": "https://example.com/gitlab-org/release-webhook-example/-/releases/v1.1", + "action": "create", + "assets": { + "count": 5, + "links": [ + { + "id": 1, + "external": true, + "link_type": "other", + "name": "Changelog", + "url": "https://example.net/changelog" + } + ], + "sources": [ + { + "format": "zip", + "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.zip" + }, + { + "format": "tar.gz", + "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar.gz" + }, + { + "format": "tar.bz2", + "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar.bz2" + }, + { + "format": "tar", + "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar" + } + ] + }, + "commit": { + "id": "ee0a3fb31ac16e11b9dbb596ad16d4af654d08f8", + "message": "Release v1.1", + "title": "Release v1.1", + "timestamp": "2020-10-31T14:58:32+11:00", + "url": "https://example.com/gitlab-org/release-webhook-example/-/commit/ee0a3fb31ac16e11b9dbb596ad16d4af654d08f8", + "author": { + "name": "Example User", + "email": "user@example.com" + } + } +} +``` diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 44225ac2921..0891d48c038 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -6,1802 +6,226 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Webhooks **(FREE)** -Project webhooks allow you to trigger a percent-encoded URL if, for example, new code is pushed or -a new issue is created. You can configure webhooks to listen for specific events -like pushes, issues or merge requests. GitLab sends a POST request with data -to the webhook URL. - -You usually need to set up your own [webhook receiver](#example-webhook-receiver) -to receive information from GitLab and send it to another app, according to your requirements. -We already have a [built-in receiver](slack.md) -for sending [Slack](https://api.slack.com/incoming-webhooks) notifications _per project_. - -## Overview - -[Webhooks](https://en.wikipedia.org/wiki/Webhook) are "_user-defined HTTP -callbacks_". They are usually triggered by some -event, such as pushing code to a repository or a comment being posted to a blog. -When that event occurs, the source app makes an HTTP request to the URI -configured for the webhook. The action taken may be anything. -Common uses are to trigger builds with continuous integration systems or to -notify bug tracking systems. - -Webhooks can be used to update an external issue tracker, trigger CI jobs, -update a backup mirror, or even deploy to your production server. - -Webhooks are available: - -- Per project, at a project's **Settings > Webhooks** menu. **(FREE)** -- Additionally per group, at a group's **Settings > Webhooks** menu. **(PREMIUM)** - -GitLab.com enforces various [webhook limits](../../../user/gitlab_com/index.md#webhooks), including: - -- The maximum number of webhooks and their size, both per project, and per group. -- The number of webhook calls per minute. - -## Possible uses for webhooks - -- You can set up a webhook in GitLab to send a notification to +[Webhooks](https://en.wikipedia.org/wiki/Webhook) are custom HTTP callbacks +that you define. They are usually triggered by an +event, such as pushing code to a repository or posting a comment on a blog. +When the event occurs, the source app makes an HTTP request to the URI +configured for the webhook. The action to take may be anything. For example, +you can use webhooks to: + +- Trigger continuous integration (CI) jobs, update external issue trackers, + update a backup mirror, or deploy to your production server. +- Send a notification to [Slack](https://api.slack.com/incoming-webhooks) every time a job fails. -- You can [integrate with Twilio to be notified via SMS](https://www.datadoghq.com/blog/send-alerts-sms-customizable-webhooks-twilio/) - every time an issue is created for a specific project or group within GitLab -- You can use them to [automatically assign labels to merge requests](https://about.gitlab.com/blog/2016/08/19/applying-gitlab-labels-automatically/). - -## Webhook endpoint tips - -If you are writing your own endpoint (web server) to receive -GitLab webhooks, keep in mind the following: - -- Your endpoint should send its HTTP response as fast as possible. If the response takes longer than - the configured timeout, GitLab decides the hook failed and retries it. For information on - customizing this timeout, see - [Webhook fails or multiple webhook requests are triggered](#webhook-fails-or-multiple-webhook-requests-are-triggered). -- Your endpoint should ALWAYS return a valid HTTP response. If you do - not do this then GitLab thinks the hook failed and retries it. - Most HTTP libraries take care of this for you automatically but if - you are writing a low-level hook this is important to remember. -- GitLab ignores the HTTP status code returned by your endpoint. - -## Secret token - -If you specify a secret token, it is sent with the hook request in the -`X-Gitlab-Token` HTTP header. Your webhook endpoint can check that to verify -that the request is legitimate. - -## SSL verification - -By default, the SSL certificate of the webhook endpoint is verified based on -an internal list of Certificate Authorities. This means the certificate cannot -be self-signed. - -You can turn this off in the webhook settings in your GitLab projects. - -## Branch filtering - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20338) in GitLab 11.3. - -Push events can be filtered by branch using a branch name or wildcard pattern -to limit which push events are sent to your webhook endpoint. By default the -field is blank causing all push events to be sent to your webhook endpoint. - -## Events - -Below are described the supported events. - -### Push events - -Triggered when you push to the repository except when pushing tags. - -NOTE: -When more than 20 commits are pushed at once, the `commits` webhook -attribute only contains the newest 20 for performance reasons. Loading -detailed commit data is expensive. Note that despite only 20 commits being -present in the `commits` attribute, the `total_commits_count` attribute contains the actual total. - -NOTE: -If a branch creation push event is generated without new commits being introduced, the -`commits` attribute in the payload is empty. - -Also, if a single push includes changes for more than three (by default, depending on -[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) -branches, this hook isn't executed. - -**Request header**: - -```plaintext -X-Gitlab-Event: Push Hook -``` - -**Payload example:** - -```json -{ - "object_kind": "push", - "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", - "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "ref": "refs/heads/master", - "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "user_id": 4, - "user_name": "John Smith", - "user_username": "jsmith", - "user_email": "john@example.com", - "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", - "project_id": 15, - "project":{ - "id": 15, - "name":"Diaspora", - "description":"", - "web_url":"http://example.com/mike/diaspora", - "avatar_url":null, - "git_ssh_url":"git@example.com:mike/diaspora.git", - "git_http_url":"http://example.com/mike/diaspora.git", - "namespace":"Mike", - "visibility_level":0, - "path_with_namespace":"mike/diaspora", - "default_branch":"master", - "homepage":"http://example.com/mike/diaspora", - "url":"git@example.com:mike/diaspora.git", - "ssh_url":"git@example.com:mike/diaspora.git", - "http_url":"http://example.com/mike/diaspora.git" - }, - "repository":{ - "name": "Diaspora", - "url": "git@example.com:mike/diaspora.git", - "description": "", - "homepage": "http://example.com/mike/diaspora", - "git_http_url":"http://example.com/mike/diaspora.git", - "git_ssh_url":"git@example.com:mike/diaspora.git", - "visibility_level":0 - }, - "commits": [ - { - "id": "b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", - "message": "Update Catalan translation to e38cb41.\n\nSee https://gitlab.com/gitlab-org/gitlab for more information", - "title": "Update Catalan translation to e38cb41.", - "timestamp": "2011-12-12T14:27:31+02:00", - "url": "http://example.com/mike/diaspora/commit/b6568db1bc1dcd7f8b4d5a946b0b91f9dacd7327", - "author": { - "name": "Jordi Mallach", - "email": "jordi@softcatala.org" - }, - "added": ["CHANGELOG"], - "modified": ["app/controller/application.rb"], - "removed": [] - }, - { - "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "message": "fixed readme", - "title": "fixed readme", - "timestamp": "2012-01-03T23:36:29+02:00", - "url": "http://example.com/mike/diaspora/commit/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "author": { - "name": "GitLab dev user", - "email": "gitlabdev@dv6700.(none)" - }, - "added": ["CHANGELOG"], - "modified": ["app/controller/application.rb"], - "removed": [] - } - ], - "total_commits_count": 4 -} -``` +- [Integrate with Twilio to be notified via SMS](https://www.datadoghq.com/blog/send-alerts-sms-customizable-webhooks-twilio/) + every time an issue is created for a specific project or group in GitLab. +- [Automatically assign labels to merge requests](https://about.gitlab.com/blog/2016/08/19/applying-gitlab-labels-automatically/). -### Tag events +You can configure your GitLab project or [group](#group-webhooks) to trigger +a percent-encoded webhook URL when an event occurs. For example, when new code +is pushed or a new issue is created. +The webhook listens for specific [events](#events) and +GitLab sends a POST request with data to the webhook URL. -Triggered when you create (or delete) tags to the repository. - -NOTE: -If a single push includes changes for more than three (by default, depending on -[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) -tags, this hook is not executed. +Usually, you set up your own [webhook receiver](#create-an-example-webhook-receiver) +to receive information from GitLab and send it to another app, according to your requirements. +We have a [built-in receiver](slack.md) +for sending [Slack](https://api.slack.com/incoming-webhooks) notifications per project. -**Request header**: +GitLab.com enforces [webhook limits](../../../user/gitlab_com/index.md#webhooks), +including: -```plaintext -X-Gitlab-Event: Tag Push Hook -``` +- The maximum number of webhooks and their size, both per project and per group. +- The number of webhook calls per minute. -**Payload example:** - -```json -{ - "object_kind": "tag_push", - "before": "0000000000000000000000000000000000000000", - "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", - "ref": "refs/tags/v1.0.0", - "checkout_sha": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", - "user_id": 1, - "user_name": "John Smith", - "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", - "project_id": 1, - "project":{ - "id": 1, - "name":"Example", - "description":"", - "web_url":"http://example.com/jsmith/example", - "avatar_url":null, - "git_ssh_url":"git@example.com:jsmith/example.git", - "git_http_url":"http://example.com/jsmith/example.git", - "namespace":"Jsmith", - "visibility_level":0, - "path_with_namespace":"jsmith/example", - "default_branch":"master", - "homepage":"http://example.com/jsmith/example", - "url":"git@example.com:jsmith/example.git", - "ssh_url":"git@example.com:jsmith/example.git", - "http_url":"http://example.com/jsmith/example.git" - }, - "repository":{ - "name": "Example", - "url": "ssh://git@example.com/jsmith/example.git", - "description": "", - "homepage": "http://example.com/jsmith/example", - "git_http_url":"http://example.com/jsmith/example.git", - "git_ssh_url":"git@example.com:jsmith/example.git", - "visibility_level":0 - }, - "commits": [], - "total_commits_count": 0 -} -``` +## Group webhooks **(PREMIUM)** -### Issue events +You can configure a webhook for a group to ensure all projects in the group +receive the same webhook settings. -Triggered when a new issue is created or an existing issue was updated/closed/reopened. +## Configure a webhook -**Request header**: +You can configure a webhook for a group or a project. -```plaintext -X-Gitlab-Event: Issue Hook -``` +1. In your project or group, on the left sidebar, select **Settings > Webhooks**. +1. In **URL**, enter the URL of the webhook endpoint. + The URL must be percentage-encoded, if necessary. +1. In **Secret token**, enter the [secret token](#validate-payloads-by-using-a-secret-token) to validate payloads. +1. In the **Trigger** section, select the [events](webhook_events.md) to trigger the webhook. +1. Optional. Clear the **Enable SSL verification** checkbox to disable [SSL verification](#verify-an-ssl-certificate). +1. Select **Add webhook**. -**Available `object_attributes.action`:** - -- `open` -- `close` -- `reopen` -- `update` - -**Payload example:** - -```json -{ - "object_kind": "issue", - "event_type": "issue", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project": { - "id": 1, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlabhq/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", - "namespace":"GitlabHQ", - "visibility_level":20, - "path_with_namespace":"gitlabhq/gitlab-test", - "default_branch":"master", - "ci_config_path": null, - "homepage":"http://example.com/gitlabhq/gitlab-test", - "url":"http://example.com/gitlabhq/gitlab-test.git", - "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "http_url":"http://example.com/gitlabhq/gitlab-test.git" - }, - "object_attributes": { - "id": 301, - "title": "New API: create/update/delete file", - "assignee_ids": [51], - "assignee_id": 51, - "author_id": 51, - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "updated_by_id": 1, - "last_edited_at": null, - "last_edited_by_id": null, - "relative_position": 0, - "description": "Create new API for manipulations with repository", - "milestone_id": null, - "state_id": 1, - "confidential": false, - "discussion_locked": true, - "due_date": null, - "moved_to_id": null, - "duplicated_to_id": null, - "time_estimate": 0, - "total_time_spent": 0, - "time_change": 0, - "human_total_time_spent": null, - "human_time_estimate": null, - "human_time_change": null, - "weight": null, - "iid": 23, - "url": "http://example.com/diaspora/issues/23", - "state": "opened", - "action": "open", - "labels": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }] - }, - "repository": { - "name": "Gitlab Test", - "url": "http://example.com/gitlabhq/gitlab-test.git", - "description": "Aut reprehenderit ut est.", - "homepage": "http://example.com/gitlabhq/gitlab-test" - }, - "assignees": [{ - "name": "User1", - "username": "user1", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - }], - "assignee": { - "name": "User1", - "username": "user1", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - }, - "labels": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "changes": { - "updated_by_id": { - "previous": null, - "current": 1 - }, - "updated_at": { - "previous": "2017-09-15 16:50:55 UTC", - "current": "2017-09-15 16:52:00 UTC" - }, - "labels": { - "previous": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "current": [{ - "id": 205, - "title": "Platform", - "color": "#123123", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "Platform related issues", - "type": "ProjectLabel", - "group_id": 41 - }] - } - } -} -``` +## Test a webhook -NOTE: -`assignee` and `assignee_id` keys are deprecated and now show the first assignee only. +You can trigger a webhook manually, to ensure it's working properly. -### Comment events +For example, to test `push events`, your project should have at least one commit. The webhook uses this commit in the webhook. -Triggered when a new comment is made on commits, merge requests, issues, and code snippets. -The note data is stored in `object_attributes` (for example, `note` or `noteable_type`). The -payload also includes information about the target of the comment. For example, -a comment on an issue includes the specific issue information under the `issue` key. -Valid target types: +To test a webhook: -- `commit` -- `merge_request` -- `issue` -- `snippet` +1. In your project, on the left sidebar, select **Settings > Webhooks**. +1. Scroll down to the list of configured webhooks. +1. From the **Test** dropdown list, select the type of event to test. -#### Comment on commit + -**Request header**: +## Create an example webhook receiver -```plaintext -X-Gitlab-Event: Note Hook -``` +To test how GitLab webhooks work, you can use +an echo script running in a console session. For the following script to +work you must have Ruby installed. -**Payload example:** - -```json -{ - "object_kind": "note", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project_id": 5, - "project":{ - "id": 5, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlabhq/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", - "namespace":"GitlabHQ", - "visibility_level":20, - "path_with_namespace":"gitlabhq/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlabhq/gitlab-test", - "url":"http://example.com/gitlabhq/gitlab-test.git", - "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "http_url":"http://example.com/gitlabhq/gitlab-test.git" - }, - "repository":{ - "name": "Gitlab Test", - "url": "http://example.com/gitlab-org/gitlab-test.git", - "description": "Aut reprehenderit ut est.", - "homepage": "http://example.com/gitlab-org/gitlab-test" - }, - "object_attributes": { - "id": 1243, - "note": "This is a commit comment. How does this work?", - "noteable_type": "Commit", - "author_id": 1, - "created_at": "2015-05-17 18:08:09 UTC", - "updated_at": "2015-05-17 18:08:09 UTC", - "project_id": 5, - "attachment":null, - "line_code": "bec9703f7a456cd2b4ab5fb3220ae016e3e394e3_0_1", - "commit_id": "cfe32cf61b73a0d5e9f13e774abde7ff789b1660", - "noteable_id": null, - "system": false, - "st_diff": { - "diff": "--- /dev/null\n+++ b/six\n@@ -0,0 +1 @@\n+Subproject commit 409f37c4f05865e4fb208c771485f211a22c4c2d\n", - "new_path": "six", - "old_path": "six", - "a_mode": "0", - "b_mode": "160000", - "new_file": true, - "renamed_file": false, - "deleted_file": false - }, - "url": "http://example.com/gitlab-org/gitlab-test/commit/cfe32cf61b73a0d5e9f13e774abde7ff789b1660#note_1243" - }, - "commit": { - "id": "cfe32cf61b73a0d5e9f13e774abde7ff789b1660", - "message": "Add submodule\n\nSigned-off-by: Example User \u003cuser@example.com.com\u003e\n", - "timestamp": "2014-02-27T10:06:20+02:00", - "url": "http://example.com/gitlab-org/gitlab-test/commit/cfe32cf61b73a0d5e9f13e774abde7ff789b1660", - "author": { - "name": "Example User", - "email": "user@example.com" - } - } -} -``` +1. Save the following file as `print_http_body.rb`: -#### Comment on merge request + ```ruby + require 'webrick' -**Request header**: + server = WEBrick::HTTPServer.new(:Port => ARGV.first) + server.mount_proc '/' do |req, res| + puts req.body + end -```plaintext -X-Gitlab-Event: Note Hook -``` + trap 'INT' do + server.shutdown + end + server.start + ``` -**Payload example:** - -```json -{ - "object_kind": "note", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project_id": 5, - "project":{ - "id": 5, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlab-org/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", - "namespace":"Gitlab Org", - "visibility_level":10, - "path_with_namespace":"gitlab-org/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlab-org/gitlab-test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "http_url":"http://example.com/gitlab-org/gitlab-test.git" - }, - "repository":{ - "name": "Gitlab Test", - "url": "http://localhost/gitlab-org/gitlab-test.git", - "description": "Aut reprehenderit ut est.", - "homepage": "http://example.com/gitlab-org/gitlab-test" - }, - "object_attributes": { - "id": 1244, - "note": "This MR needs work.", - "noteable_type": "MergeRequest", - "author_id": 1, - "created_at": "2015-05-17 18:21:36 UTC", - "updated_at": "2015-05-17 18:21:36 UTC", - "project_id": 5, - "attachment": null, - "line_code": null, - "commit_id": "", - "noteable_id": 7, - "system": false, - "st_diff": null, - "url": "http://example.com/gitlab-org/gitlab-test/merge_requests/1#note_1244" - }, - "merge_request": { - "id": 7, - "target_branch": "markdown", - "source_branch": "master", - "source_project_id": 5, - "author_id": 8, - "assignee_id": 28, - "title": "Tempora et eos debitis quae laborum et.", - "created_at": "2015-03-01 20:12:53 UTC", - "updated_at": "2015-03-21 18:27:27 UTC", - "milestone_id": 11, - "state": "opened", - "merge_status": "cannot_be_merged", - "target_project_id": 5, - "iid": 1, - "description": "Et voluptas corrupti assumenda temporibus. Architecto cum animi eveniet amet asperiores. Vitae numquam voluptate est natus sit et ad id.", - "position": 0, - "source":{ - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlab-org/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", - "namespace":"Gitlab Org", - "visibility_level":10, - "path_with_namespace":"gitlab-org/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlab-org/gitlab-test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "http_url":"http://example.com/gitlab-org/gitlab-test.git" - }, - "target": { - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlab-org/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", - "namespace":"Gitlab Org", - "visibility_level":10, - "path_with_namespace":"gitlab-org/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlab-org/gitlab-test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "http_url":"http://example.com/gitlab-org/gitlab-test.git" - }, - "last_commit": { - "id": "562e173be03b8ff2efb05345d12df18815438a4b", - "message": "Merge branch 'another-branch' into 'master'\n\nCheck in this test\n", - "timestamp": "2015-04-08T21: 00:25-07:00", - "url": "http://example.com/gitlab-org/gitlab-test/commit/562e173be03b8ff2efb05345d12df18815438a4b", - "author": { - "name": "John Smith", - "email": "john@example.com" - } - }, - "work_in_progress": false, - "assignee": { - "name": "User1", - "username": "user1", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - } - } -} -``` +1. Choose an unused port (for example, `8000`) and start the script: -#### Comment on issue + ```shell + ruby print_http_body.rb 8000 + ``` -**Request header**: +1. In GitLab, add your webhook receiver as `http://my.host:8000/`. -```plaintext -X-Gitlab-Event: Note Hook -``` +1. Select **Test**. You should see something like this in the console: -**Payload example:** - -```json -{ - "object_kind": "note", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project_id": 5, - "project":{ - "id": 5, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlab-org/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", - "namespace":"Gitlab Org", - "visibility_level":10, - "path_with_namespace":"gitlab-org/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlab-org/gitlab-test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "http_url":"http://example.com/gitlab-org/gitlab-test.git" - }, - "repository":{ - "name":"diaspora", - "url":"git@example.com:mike/diaspora.git", - "description":"", - "homepage":"http://example.com/mike/diaspora" - }, - "object_attributes": { - "id": 1241, - "note": "Hello world", - "noteable_type": "Issue", - "author_id": 1, - "created_at": "2015-05-17 17:06:40 UTC", - "updated_at": "2015-05-17 17:06:40 UTC", - "project_id": 5, - "attachment": null, - "line_code": null, - "commit_id": "", - "noteable_id": 92, - "system": false, - "st_diff": null, - "url": "http://example.com/gitlab-org/gitlab-test/issues/17#note_1241" - }, - "issue": { - "id": 92, - "title": "test", - "assignee_ids": [], - "assignee_id": null, - "author_id": 1, - "project_id": 5, - "created_at": "2015-04-12 14:53:17 UTC", - "updated_at": "2015-04-26 08:28:42 UTC", - "position": 0, - "branch_name": null, - "description": "test", - "milestone_id": null, - "state": "closed", - "iid": 17, - "labels": [ - { - "id": 25, - "title": "Afterpod", - "color": "#3e8068", - "project_id": null, - "created_at": "2019-06-05T14:32:20.211Z", - "updated_at": "2019-06-05T14:32:20.211Z", - "template": false, - "description": null, - "type": "GroupLabel", - "group_id": 4 - }, - { - "id": 86, - "title": "Element", - "color": "#231afe", - "project_id": 4, - "created_at": "2019-06-05T14:32:20.637Z", - "updated_at": "2019-06-05T14:32:20.637Z", - "template": false, - "description": null, - "type": "ProjectLabel", - "group_id": null - } - ] - } -} -``` + ```plaintext + {"before":"077a85dd266e6f3573ef7e9ef8ce3343ad659c4e","after":"95cd4a99e93bc4bbabacfa2cd10e6725b1403c60",<SNIP>} + example.com - - [14/May/2014:07:45:26 EDT] "POST / HTTP/1.1" 200 0 + - -> / + ``` NOTE: -`assignee_id` field is deprecated and now shows the first assignee only. - -#### Comment on code snippet - -**Request header**: - -```plaintext -X-Gitlab-Event: Note Hook -``` - -**Payload example:** - -```json -{ - "object_kind": "note", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project_id": 5, - "project":{ - "id": 5, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlab-org/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "git_http_url":"http://example.com/gitlab-org/gitlab-test.git", - "namespace":"Gitlab Org", - "visibility_level":10, - "path_with_namespace":"gitlab-org/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlab-org/gitlab-test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "ssh_url":"git@example.com:gitlab-org/gitlab-test.git", - "http_url":"http://example.com/gitlab-org/gitlab-test.git" - }, - "repository":{ - "name":"Gitlab Test", - "url":"http://example.com/gitlab-org/gitlab-test.git", - "description":"Aut reprehenderit ut est.", - "homepage":"http://example.com/gitlab-org/gitlab-test" - }, - "object_attributes": { - "id": 1245, - "note": "Is this snippet doing what it's supposed to be doing?", - "noteable_type": "Snippet", - "author_id": 1, - "created_at": "2015-05-17 18:35:50 UTC", - "updated_at": "2015-05-17 18:35:50 UTC", - "project_id": 5, - "attachment": null, - "line_code": null, - "commit_id": "", - "noteable_id": 53, - "system": false, - "st_diff": null, - "url": "http://example.com/gitlab-org/gitlab-test/snippets/53#note_1245" - }, - "snippet": { - "id": 53, - "title": "test", - "content": "puts 'Hello world'", - "author_id": 1, - "project_id": 5, - "created_at": "2015-04-09 02:40:38 UTC", - "updated_at": "2015-04-09 02:40:38 UTC", - "file_name": "test.rb", - "expires_at": null, - "type": "ProjectSnippet", - "visibility_level": 0 - } -} -``` - -### Merge request events - -Triggered when a new merge request is created, an existing merge request was updated/merged/closed or a commit is added in the source branch. - -**Request header**: - -```plaintext -X-Gitlab-Event: Merge Request Hook -``` - -**Available `object_attributes.action`:** - -- `open` -- `close` -- `reopen` -- `update` -- `approved` -- `unapproved` -- `merge` - -**Payload example:** - -```json -{ - "object_kind": "merge_request", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", - "email": "admin@example.com" - }, - "project": { - "id": 1, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlabhq/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", - "namespace":"GitlabHQ", - "visibility_level":20, - "path_with_namespace":"gitlabhq/gitlab-test", - "default_branch":"master", - "homepage":"http://example.com/gitlabhq/gitlab-test", - "url":"http://example.com/gitlabhq/gitlab-test.git", - "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "http_url":"http://example.com/gitlabhq/gitlab-test.git" - }, - "repository": { - "name": "Gitlab Test", - "url": "http://example.com/gitlabhq/gitlab-test.git", - "description": "Aut reprehenderit ut est.", - "homepage": "http://example.com/gitlabhq/gitlab-test" - }, - "object_attributes": { - "id": 99, - "target_branch": "master", - "source_branch": "ms-viewport", - "source_project_id": 14, - "author_id": 51, - "assignee_id": 6, - "title": "MS-Viewport", - "created_at": "2013-12-03T17:23:34Z", - "updated_at": "2013-12-03T17:23:34Z", - "milestone_id": null, - "state": "opened", - "merge_status": "unchecked", - "target_project_id": 14, - "iid": 1, - "description": "", - "source": { - "name":"Awesome Project", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/awesome_space/awesome_project", - "avatar_url":null, - "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", - "git_http_url":"http://example.com/awesome_space/awesome_project.git", - "namespace":"Awesome Space", - "visibility_level":20, - "path_with_namespace":"awesome_space/awesome_project", - "default_branch":"master", - "homepage":"http://example.com/awesome_space/awesome_project", - "url":"http://example.com/awesome_space/awesome_project.git", - "ssh_url":"git@example.com:awesome_space/awesome_project.git", - "http_url":"http://example.com/awesome_space/awesome_project.git" - }, - "target": { - "name":"Awesome Project", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/awesome_space/awesome_project", - "avatar_url":null, - "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", - "git_http_url":"http://example.com/awesome_space/awesome_project.git", - "namespace":"Awesome Space", - "visibility_level":20, - "path_with_namespace":"awesome_space/awesome_project", - "default_branch":"master", - "homepage":"http://example.com/awesome_space/awesome_project", - "url":"http://example.com/awesome_space/awesome_project.git", - "ssh_url":"git@example.com:awesome_space/awesome_project.git", - "http_url":"http://example.com/awesome_space/awesome_project.git" - }, - "last_commit": { - "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "message": "fixed readme", - "timestamp": "2012-01-03T23:36:29+02:00", - "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", - "author": { - "name": "GitLab dev user", - "email": "gitlabdev@dv6700.(none)" - } - }, - "work_in_progress": false, - "url": "http://example.com/diaspora/merge_requests/1", - "action": "open", - "assignee": { - "name": "User1", - "username": "user1", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" - } - }, - "labels": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "changes": { - "updated_by_id": { - "previous": null, - "current": 1 - }, - "updated_at": { - "previous": "2017-09-15 16:50:55 UTC", - "current":"2017-09-15 16:52:00 UTC" - }, - "labels": { - "previous": [{ - "id": 206, - "title": "API", - "color": "#ffffff", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "API related issues", - "type": "ProjectLabel", - "group_id": 41 - }], - "current": [{ - "id": 205, - "title": "Platform", - "color": "#123123", - "project_id": 14, - "created_at": "2013-12-03T17:15:43Z", - "updated_at": "2013-12-03T17:15:43Z", - "template": false, - "description": "Platform related issues", - "type": "ProjectLabel", - "group_id": 41 - }] - } - } -} -``` - -### Wiki Page events - -Triggered when a wiki page is created, updated or deleted. - -**Request Header**: - -```plaintext -X-Gitlab-Event: Wiki Page Hook -``` - -**Payload example**: - -```json -{ - "object_kind": "wiki_page", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "project": { - "id": 1, - "name": "awesome-project", - "description": "This is awesome", - "web_url": "http://example.com/root/awesome-project", - "avatar_url": null, - "git_ssh_url": "git@example.com:root/awesome-project.git", - "git_http_url": "http://example.com/root/awesome-project.git", - "namespace": "root", - "visibility_level": 0, - "path_with_namespace": "root/awesome-project", - "default_branch": "master", - "homepage": "http://example.com/root/awesome-project", - "url": "git@example.com:root/awesome-project.git", - "ssh_url": "git@example.com:root/awesome-project.git", - "http_url": "http://example.com/root/awesome-project.git" - }, - "wiki": { - "web_url": "http://example.com/root/awesome-project/-/wikis/home", - "git_ssh_url": "git@example.com:root/awesome-project.wiki.git", - "git_http_url": "http://example.com/root/awesome-project.wiki.git", - "path_with_namespace": "root/awesome-project.wiki", - "default_branch": "master" - }, - "object_attributes": { - "title": "Awesome", - "content": "awesome content goes here", - "format": "markdown", - "message": "adding an awesome page to the wiki", - "slug": "awesome", - "url": "http://example.com/root/awesome-project/-/wikis/awesome", - "action": "create" - } -} -``` - -### Pipeline events - -In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53159) -and later, the pipeline webhook returns only the latest jobs. - -Triggered on status change of Pipeline. - -**Request Header**: - -```plaintext -X-Gitlab-Event: Pipeline Hook -``` - -**Payload example**: - -```json -{ - "object_kind": "pipeline", - "object_attributes":{ - "id": 31, - "ref": "master", - "tag": false, - "sha": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", - "before_sha": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", - "source": "merge_request_event", - "status": "success", - "stages":[ - "build", - "test", - "deploy" - ], - "created_at": "2016-08-12 15:23:28 UTC", - "finished_at": "2016-08-12 15:26:29 UTC", - "duration": 63, - "variables": [ - { - "key": "NESTOR_PROD_ENVIRONMENT", - "value": "us-west-1" - } - ] - }, - "merge_request": { - "id": 1, - "iid": 1, - "title": "Test", - "source_branch": "test", - "source_project_id": 1, - "target_branch": "master", - "target_project_id": 1, - "state": "opened", - "merge_status": "can_be_merged", - "url": "http://192.168.64.1:3005/gitlab-org/gitlab-test/merge_requests/1" - }, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "user_email@gitlab.com" - }, - "project":{ - "id": 1, - "name": "Gitlab Test", - "description": "Atque in sunt eos similique dolores voluptatem.", - "web_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test", - "avatar_url": null, - "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", - "git_http_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test.git", - "namespace": "Gitlab Org", - "visibility_level": 20, - "path_with_namespace": "gitlab-org/gitlab-test", - "default_branch": "master" - }, - "commit":{ - "id": "bcbb5ec396a2c0f828686f14fac9b80b780504f2", - "message": "test\n", - "timestamp": "2016-08-12T17:23:21+02:00", - "url": "http://example.com/gitlab-org/gitlab-test/commit/bcbb5ec396a2c0f828686f14fac9b80b780504f2", - "author":{ - "name": "User", - "email": "user@gitlab.com" - } - }, - "builds":[ - { - "id": 380, - "stage": "deploy", - "name": "production", - "status": "skipped", - "created_at": "2016-08-12 15:23:28 UTC", - "started_at": null, - "finished_at": null, - "when": "manual", - "manual": true, - "allow_failure": false, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "runner": null, - "artifacts_file":{ - "filename": null, - "size": null - }, - "environment": { - "name": "production", - "action": "start", - "deployment_tier": "production" - } - }, - { - "id": 377, - "stage": "test", - "name": "test-image", - "status": "success", - "created_at": "2016-08-12 15:23:28 UTC", - "started_at": "2016-08-12 15:26:12 UTC", - "finished_at": null, - "when": "on_success", - "manual": false, - "allow_failure": false, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "runner": { - "id": 380987, - "description": "shared-runners-manager-6.gitlab.com", - "active": true, - "runner_type": "instance_type", - "is_shared": true, - "tags": [ - "linux", - "docker", - "shared-runner" - ] - }, - "artifacts_file":{ - "filename": null, - "size": null - }, - "environment": null - }, - { - "id": 378, - "stage": "test", - "name": "test-build", - "status": "success", - "created_at": "2016-08-12 15:23:28 UTC", - "started_at": "2016-08-12 15:26:12 UTC", - "finished_at": "2016-08-12 15:26:29 UTC", - "when": "on_success", - "manual": false, - "allow_failure": false, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "runner": { - "id":380987, - "description":"shared-runners-manager-6.gitlab.com", - "active":true, - "runner_type": "instance_type", - "is_shared": true, - "tags": [ - "linux", - "docker" - ] - }, - "artifacts_file":{ - "filename": null, - "size": null - }, - "environment": null - }, - { - "id": 376, - "stage": "build", - "name": "build-image", - "status": "success", - "created_at": "2016-08-12 15:23:28 UTC", - "started_at": "2016-08-12 15:24:56 UTC", - "finished_at": "2016-08-12 15:25:26 UTC", - "when": "on_success", - "manual": false, - "allow_failure": false, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "runner": { - "id": 380987, - "description": "shared-runners-manager-6.gitlab.com", - "active": true, - "runner_type": "instance_type", - "is_shared": true, - "tags": [ - "linux", - "docker" - ] - }, - "artifacts_file":{ - "filename": null, - "size": null - }, - "environment": null - }, - { - "id": 379, - "stage": "deploy", - "name": "staging", - "status": "created", - "created_at": "2016-08-12 15:23:28 UTC", - "started_at": null, - "finished_at": null, - "when": "on_success", - "manual": false, - "allow_failure": false, - "user":{ - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - "email": "admin@example.com" - }, - "runner": null, - "artifacts_file":{ - "filename": null, - "size": null - }, - "environment": { - "name": "staging", - "action": "start", - "deployment_tier": "staging" - } - } - ] -} -``` - -### Job events - -Triggered on status change of a job. - -**Request Header**: - -```plaintext -X-Gitlab-Event: Job Hook -``` - -**Payload example**: - -```json -{ - "object_kind": "build", - "ref": "gitlab-script-trigger", - "tag": false, - "before_sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", - "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", - "build_id": 1977, - "build_name": "test", - "build_stage": "test", - "build_status": "created", - "build_created_at": "2021-02-23T02:41:37.886Z", - "build_started_at": null, - "build_finished_at": null, - "build_duration": null, - "build_allow_failure": false, - "build_failure_reason": "script_failure", - "pipeline_id": 2366, - "project_id": 380, - "project_name": "gitlab-org/gitlab-test", - "user": { - "id": 3, - "name": "User", - "email": "user@gitlab.com", - "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon", - }, - "commit": { - "id": 2366, - "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", - "message": "test\n", - "author_name": "User", - "author_email": "user@gitlab.com", - "status": "created", - "duration": null, - "started_at": null, - "finished_at": null - }, - "repository": { - "name": "gitlab_test", - "description": "Atque in sunt eos similique dolores voluptatem.", - "homepage": "http://192.168.64.1:3005/gitlab-org/gitlab-test", - "git_ssh_url": "git@192.168.64.1:gitlab-org/gitlab-test.git", - "git_http_url": "http://192.168.64.1:3005/gitlab-org/gitlab-test.git", - "visibility_level": 20 - }, - "runner": { - "active": true, - "runner_type": "project_type", - "is_shared": false, - "id": 380987, - "description": "shared-runners-manager-6.gitlab.com", - "tags": [ - "linux", - "docker" - ] - }, - "environment": null -} -``` - -Note that `commit.id` is the ID of the pipeline, not the ID of the commit. - -### Deployment events - -Triggered when a deployment: - -- Starts ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41214) in GitLab 13.5.) -- Succeeds -- Fails -- Is cancelled - -**Request Header**: - -```plaintext -X-Gitlab-Event: Deployment Hook -``` - -**Payload example**: - -```json -{ - "object_kind": "deployment", - "status": "success", - "status_changed_at":"2021-04-28 21:50:00 +0200", - "deployment_id": 15, - "deployable_id": 796, - "deployable_url": "http://10.126.0.2:3000/root/test-deployment-webhooks/-/jobs/796", - "environment": "staging", - "project": { - "id": 30, - "name": "test-deployment-webhooks", - "description": "", - "web_url": "http://10.126.0.2:3000/root/test-deployment-webhooks", - "avatar_url": null, - "git_ssh_url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", - "git_http_url": "http://10.126.0.2:3000/root/test-deployment-webhooks.git", - "namespace": "Administrator", - "visibility_level": 0, - "path_with_namespace": "root/test-deployment-webhooks", - "default_branch": "master", - "ci_config_path": "", - "homepage": "http://10.126.0.2:3000/root/test-deployment-webhooks", - "url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", - "ssh_url": "ssh://vlad@10.126.0.2:2222/root/test-deployment-webhooks.git", - "http_url": "http://10.126.0.2:3000/root/test-deployment-webhooks.git" - }, - "short_sha": "279484c0", - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "email": "admin@example.com" - }, - "user_url": "http://10.126.0.2:3000/root", - "commit_url": "http://10.126.0.2:3000/root/test-deployment-webhooks/-/commit/279484c09fbe69ededfced8c1bb6e6d24616b468", - "commit_title": "Add new file" -} -``` - -Note that `deployable_id` is the ID of the CI job. - -### Group member events **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260347) in GitLab 13.7. - -Member events are triggered when: - -- A user is added as a group member -- The access level of a user has changed -- The expiration date for user access has been updated -- A user has been removed from the group - -#### Add member to group - -**Request Header**: - -```plaintext -X-Gitlab-Event: Member Hook -``` - -**Payload example**: - -```json -{ - "created_at": "2020-12-11T04:57:22Z", - "updated_at": "2020-12-11T04:57:22Z", - "group_name": "webhook-test", - "group_path": "webhook-test", - "group_id": 100, - "user_username": "test_user", - "user_name": "Test User", - "user_email": "testuser@webhooktest.com", - "user_id": 64, - "group_access": "Guest", - "group_plan": null, - "expires_at": "2020-12-14T00:00:00Z", - "event_name": "user_add_to_group" -} -``` - -#### Update member access level or expiration date - -**Request Header**: - -```plaintext -X-Gitlab-Event: Member Hook -``` - -**Payload example**: - -```json -{ - "created_at": "2020-12-11T04:57:22Z", - "updated_at": "2020-12-12T08:48:19Z", - "group_name": "webhook-test", - "group_path": "webhook-test", - "group_id": 100, - "user_username": "test_user", - "user_name": "Test User", - "user_email": "testuser@webhooktest.com", - "user_id": 64, - "group_access": "Developer", - "group_plan": null, - "expires_at": "2020-12-20T00:00:00Z", - "event_name": "user_update_for_group" -} -``` - -#### Remove member from group - -**Request Header**: - -```plaintext -X-Gitlab-Event: Member Hook -``` - -**Payload example**: - -```json -{ - "created_at": "2020-12-11T04:57:22Z", - "updated_at": "2020-12-12T08:52:34Z", - "group_name": "webhook-test", - "group_path": "webhook-test", - "group_id": 100, - "user_username": "test_user", - "user_name": "Test User", - "user_email": "testuser@webhooktest.com", - "user_id": 64, - "group_access": "Guest", - "group_plan": null, - "expires_at": "2020-12-14T00:00:00Z", - "event_name": "user_remove_from_group" -} -``` - -### Subgroup events **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260419) in GitLab 13.9. - -Subgroup events are triggered when: - -- A [subgroup is created in a group](#subgroup-created-in-a-group) -- A [subgroup is removed from a group](#subgroup-removed-from-a-group) - -#### Subgroup created in a group - -**Request Header**: - -```plaintext -X-Gitlab-Event: Subgroup Hook -``` - -**Payload example**: - -```json -{ - - "created_at": "2021-01-20T09:40:12Z", - "updated_at": "2021-01-20T09:40:12Z", - "event_name": "subgroup_create", - "name": "subgroup1", - "path": "subgroup1", - "full_path": "group1/subgroup1", - "group_id": 10, - "parent_group_id": 7, - "parent_name": "group1", - "parent_path": "group1", - "parent_full_path": "group1" - -} -``` - -#### Subgroup removed from a group - -**Request Header**: - -```plaintext -X-Gitlab-Event: Subgroup Hook -``` - -**Payload example**: - -```json -{ - - "created_at": "2021-01-20T09:40:12Z", - "updated_at": "2021-01-20T09:40:12Z", - "event_name": "subgroup_destroy", - "name": "subgroup1", - "path": "subgroup1", - "full_path": "group1/subgroup1", - "group_id": 10, - "parent_group_id": 7, - "parent_name": "group1", - "parent_path": "group1", - "parent_full_path": "group1" +You may need to [allow requests to the local network](../../../security/webhooks.md) for this +receiver to be added. -} -``` +## Validate payloads by using a secret token -NOTE: -Webhooks for when a [subgroup is removed from a group](#subgroup-removed-from-a-group) are not triggered when a [subgroup is transferred to a new parent group](../../group/index.md#transfer-a-group) +You can specify a secret token to validate received payloads. +The token is sent with the hook request in the +`X-Gitlab-Token` HTTP header. Your webhook endpoint can check the token to verify +that the request is legitimate. -### Feature Flag events +## Verify an SSL certificate -Triggered when a feature flag is turned on or off. +By default, the SSL certificate of the webhook endpoint is verified based on +an internal list of Certificate Authorities. This means the certificate cannot +be self-signed. -**Request Header**: +You can turn off SSL verification in the [webhook settings](#configure-a-webhook) +in your GitLab projects. -```plaintext -X-Gitlab-Event: Feature Flag Hook -``` +## Filter push events by branch -**Payload example**: - -```json -{ - "object_kind": "feature_flag", - "project": { - "id": 1, - "name":"Gitlab Test", - "description":"Aut reprehenderit ut est.", - "web_url":"http://example.com/gitlabhq/gitlab-test", - "avatar_url":null, - "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", - "namespace":"GitlabHQ", - "visibility_level":20, - "path_with_namespace":"gitlabhq/gitlab-test", - "default_branch":"master", - "ci_config_path": null, - "homepage":"http://example.com/gitlabhq/gitlab-test", - "url":"http://example.com/gitlabhq/gitlab-test.git", - "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", - "http_url":"http://example.com/gitlabhq/gitlab-test.git" - }, - "user": { - "id": 1, - "name": "Administrator", - "username": "root", - "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", - "email": "admin@example.com" - }, - "user_url": "http://example.com/root", - "object_attributes": { - "id": 6, - "name": "test-feature-flag", - "description": "test-feature-flag-description", - "active": true - } -} -``` +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/20338) in GitLab 11.3. -### Release events +Push events can be filtered by branch using a branch name or wildcard pattern +to limit which push events are sent to your webhook endpoint. By default, +all push events are sent to your webhook endpoint. You can configure branch filtering +in the [webhook settings](#configure-a-webhook) in your project. -Triggered when a release is created or updated. +## HTTP responses for your endpoint -**Request Header**: +If you are writing your own endpoint (web server) to receive +GitLab webhooks, keep in mind the following: -```plaintext -X-Gitlab-Event: Release Hook -``` +- Your endpoint should send its HTTP response as fast as possible. If the response + takes longer than the configured timeout, GitLab assumes the hook failed and retries it. + To customize the timeout, see + [Webhook fails or multiple webhook requests are triggered](#webhook-fails-or-multiple-webhook-requests-are-triggered). +- Your endpoint should ALWAYS return a valid HTTP response. If not, + GitLab assumes the hook failed and retries it. + Most HTTP libraries take care of the response for you automatically but if + you are writing a low-level hook, this is important to remember. +- GitLab ignores the HTTP status code returned by your endpoint. -**Available `object_attributes.action`:** - -- `create` -- `update` - -**Payload example**: - -```json -{ - "id": 1, - "created_at": "2020-11-02 12:55:12 UTC", - "description": "v1.0 has been released", - "name": "v1.1", - "released_at": "2020-11-02 12:55:12 UTC", - "tag": "v1.1", - "object_kind": "release", - "project": { - "id": 2, - "name": "release-webhook-example", - "description": "", - "web_url": "https://example.com/gitlab-org/release-webhook-example", - "avatar_url": null, - "git_ssh_url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", - "git_http_url": "https://example.com/gitlab-org/release-webhook-example.git", - "namespace": "Gitlab", - "visibility_level": 0, - "path_with_namespace": "gitlab-org/release-webhook-example", - "default_branch": "master", - "ci_config_path": null, - "homepage": "https://example.com/gitlab-org/release-webhook-example", - "url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", - "ssh_url": "ssh://git@example.com/gitlab-org/release-webhook-example.git", - "http_url": "https://example.com/gitlab-org/release-webhook-example.git" - }, - "url": "https://example.com/gitlab-org/release-webhook-example/-/releases/v1.1", - "action": "create", - "assets": { - "count": 5, - "links": [ - { - "id": 1, - "external": true, - "link_type": "other", - "name": "Changelog", - "url": "https://example.net/changelog" - } - ], - "sources": [ - { - "format": "zip", - "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.zip" - }, - { - "format": "tar.gz", - "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar.gz" - }, - { - "format": "tar.bz2", - "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar.bz2" - }, - { - "format": "tar", - "url": "https://example.com/gitlab-org/release-webhook-example/-/archive/v1.1/release-webhook-example-v1.1.tar" - } - ] - }, - "commit": { - "id": "ee0a3fb31ac16e11b9dbb596ad16d4af654d08f8", - "message": "Release v1.1", - "title": "Release v1.1", - "timestamp": "2020-10-31T14:58:32+11:00", - "url": "https://example.com/gitlab-org/release-webhook-example/-/commit/ee0a3fb31ac16e11b9dbb596ad16d4af654d08f8", - "author": { - "name": "Example User", - "email": "user@example.com" - } - } -} -``` +## How image URLs are displayed in the webhook body -## Image URL rewriting +> Introduced in GitLab 11.2. -From GitLab 11.2, simple image references are rewritten to use an absolute URL -in webhooks. So if an image, merge request, comment, or wiki page has this in -its description: +Relative image references are rewritten to use an absolute URL +in the body of a webhook. +For example, if an image, merge request, comment, or wiki page includes the +following image reference: ```markdown  ``` -It appears in the webhook body as follows assuming that: +If: - GitLab is installed at `gitlab.example.com`. - The project is at `example-group/example-project`. +The reference is rewritten in the webhook body as follows: + ```markdown  ``` -This doesn't rewrite URLs that already are pointing to HTTP, HTTPS, or -protocol-relative URLs. It also doesn't rewrite image URLs using advanced -Markdown features, like link labels. +Image URLs are not rewritten if: -## Testing webhooks +- They already point to HTTP, HTTPS, or + protocol-relative URLs. +- They use advanced Markdown features like link labels. -You can trigger the webhook manually. Sample data from the project is used. -For example, for triggering `Push Events` your project should have at least one commit. +## Events - +For more information about supported events for Webhooks, go to [Webhook events](webhook_events.md). ## Troubleshoot webhooks -GitLab stores each perform of the webhook. -You can find records for last 2 days in "Recent Deliveries" section on the edit page of each webhook. +GitLab records the history of each webhook request. +You can view requests made in the last 2 days in the **Recent events** table. - +To view the table: -In this section you can see: +1. In your project, on the left sidebar, select **Settings > Webhooks**. +1. Scroll down to the webhooks. +1. Select **Edit** for the webhook you want to view. -- HTTP status code (green for `200-299` codes, red for the others, `internal error` for failed deliveries). -- Triggered event. -- A time when the event was called. -- Elapsed time of the request. +The table includes the following details about each request: -If you need more information about execution, you can click `View details` link. -On this page, you can see data that GitLab sends (request headers and body) and data that it received (response headers and body). +- HTTP status code (green for `200`-`299` codes, red for the others, and `internal error` for failed deliveries) +- Triggered event +- Elapsed time of the request +- Relative time for when the request was made -From this page, you can repeat delivery with the same data by clicking `Resend Request` button. + NOTE: -This history is unavailable for Group-level webhooks. For more information, read +The **Recent events** table is unavailable for group-level webhooks. For more information, read [issue #325642](https://gitlab.com/gitlab-org/gitlab/-/issues/325642). +Each webhook event has a corresponding **Details** page. This page details the data that GitLab sent (request headers and body) and received (response headers and body). +To view the **Details** page, select **View details** for the webhook event. + +To repeat the delivery with the same data, select **Resend Request**. + NOTE: -If URL or secret token of the webhook were updated, data is delivered to the new address. +If you update the URL or secret token of the webhook, data is delivered to the new address. ### Webhook fails or multiple webhook requests are triggered -When GitLab sends a webhook, it expects a response in 10 seconds by default. If it does not receive -one, it retries the webhook. If the endpoint doesn't send its HTTP response within those 10 seconds, -GitLab may decide the hook failed and retry it. +When GitLab sends a webhook, it expects a response in 10 seconds by default. +If the endpoint doesn't send an HTTP response in those 10 seconds, +GitLab may assume the webhook failed and retry it. -If your webhooks are failing or you are receiving multiple requests, you can try changing the -default value. You can do this by uncommenting or adding the following setting to your -`/etc/gitlab/gitlab.rb` file: +If your webhooks are failing or you are receiving multiple requests, +you can try changing the default timeout value. +In your `/etc/gitlab/gitlab.rb` file, uncomment or add the following setting: ```ruby gitlab_rails['webhook_timeout'] = 10 @@ -1809,48 +233,11 @@ gitlab_rails['webhook_timeout'] = 10 ### Unable to get local issuer certificate -When SSL verification is enabled, this error indicates that GitLab isn't able to verify the SSL certificate of the webhook endpoint. -Typically, this is because the root certificate isn't issued by a trusted certification authority as +When SSL verification is enabled, you might get an error that GitLab cannot +verify the SSL certificate of the webhook endpoint. +Typically, this error occurs because the root certificate isn't +issued by a trusted certification authority as determined by [CAcert.org](http://www.cacert.org/). -Should that not be the case, consider using [SSL Checker](https://www.sslshopper.com/ssl-checker.html) to identify faults. -Missing intermediate certificates are a common point of verification failure. - -## Example webhook receiver - -If you want to see GitLab webhooks in action for testing purposes you can use -a simple echo script running in a console session. For the following script to -work you need to have Ruby installed. - -Save the following file as `print_http_body.rb`: - -```ruby -require 'webrick' - -server = WEBrick::HTTPServer.new(:Port => ARGV.first) -server.mount_proc '/' do |req, res| - puts req.body -end - -trap 'INT' do - server.shutdown -end -server.start -``` - -Pick an unused port (for example, `8000`) and start the script: `ruby print_http_body.rb -8000`. Then add your server as a webhook receiver in GitLab as -`http://my.host:8000/`. - -When you press 'Test' in GitLab, you should see something like this in the -console: - -```plaintext -{"before":"077a85dd266e6f3573ef7e9ef8ce3343ad659c4e","after":"95cd4a99e93bc4bbabacfa2cd10e6725b1403c60",<SNIP>} -example.com - - [14/May/2014:07:45:26 EDT] "POST / HTTP/1.1" 200 0 -- -> / -``` - -NOTE: -You may need to [allow requests to the local network](../../../security/webhooks.md) for this -receiver to be added. +If that is not the case, consider using [SSL Checker](https://www.sslshopper.com/ssl-checker.html) to identify faults. +Missing intermediate certificates are common causes of verification failure. diff --git a/doc/user/project/integrations/zentao.md b/doc/user/project/integrations/zentao.md deleted file mode 100644 index ab8a7829139..00000000000 --- a/doc/user/project/integrations/zentao.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -stage: Ecosystem -group: Integrations -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# ZenTao product integration **(PREMIUM)** - -[ZenTao](https://www.zentao.net/) is a web-based project management platform. - -## Configure ZenTao - -This integration requires a ZenTao API secret key. - -Complete these steps in ZenTao: - -1. Go to your **Admin** page and select **Develop > Application**. -1. Select **Add Application**. -1. Under **Name** and **Code**, enter a name and a code for the new secret key. -1. Under **Account**, select an existing account name. -1. Select **Save**. -1. Copy the generated key to use in GitLab. - -## Configure GitLab - -Complete these steps in GitLab: - -1. Go to your project and select **Settings > Integrations**. -1. Select **ZenTao**. -1. Turn on the **Active** toggle under **Enable Integration**. -1. Provide the ZenTao configuration information: - - **ZenTao Web URL**: The base URL of the ZenTao instance web interface you're linking to this GitLab project (for example, `example.zentao.net`). - - **ZenTao API URL** (optional): The base URL to the ZenTao instance API. Defaults to the Web URL value if not set. - - **ZenTao API token**: Use the key you generated when you [configured ZenTao](#configure-zentao). - - **ZenTao Product ID**: To display issues from a single ZenTao product in a given GitLab project. The Product ID can be found in the ZenTao product page under **Settings > Overview**. - -  - -1. To verify the ZenTao connection is working, select **Test settings**. -1. Select **Save changes**. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 4d1805e3d31..8a599608f71 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -203,17 +203,13 @@ When visiting a board, issues appear ordered in any list. You're able to change that order by dragging the issues. The changed order is saved, so that anybody who visits the same board later sees the reordering, with some exceptions. -The first time an issue appears in any board (that is, the first time a user -loads a board containing that issue), it is ordered in relation to other issues in that list. -The order is done according to [label priority](labels.md#label-priority). +When an issue is created, the system assigns a relative order value that is greater than the maximum value +of that issue's project or root group. This means the issue will be at the bottom of any issue list that +it appears in. -At this point, that issue is assigned a relative order value by the system, -with respect to the other issues in the list. Any time -you drag and reorder the issue, its relative order value changes accordingly. - -Also, any time that issue appears in any board, the ordering is done according to -the updated relative order value. It's only the first -time an issue appears that it takes from the priority order mentioned above. If a user in your GitLab instance +Any time you drag and reorder the issue, its relative order value changes accordingly. +Then, any time that issue appears in any board, the ordering is done according to +the updated relative order value. If a user in your GitLab instance drags issue `A` above issue `B`, the ordering is maintained when these two issues are subsequently loaded in any board in the same instance. This could be a different project board or a different group board, for example. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 7f2713b81e6..37c00bf0efa 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -1,21 +1,21 @@ --- stage: Plan group: Product Planning -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Design Management **(FREE)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. -> - Support for SVGs was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4. -> - Design Management was [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/660) in GitLab Premium 12.2. +> - Support for SVGs [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12771) in GitLab Premium 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. -Design Management allows you to upload design assets (wireframes, mockups, etc.) -to GitLab issues and keep them stored in one single place, accessed by the Design +Design Management allows you to upload design assets (including wireframes and mockups) +to GitLab issues and keep them stored in a single place, accessed by the Design Management's page within an issue, giving product designers, product managers, and engineers a -way to collaborate on designs over one single source of truth. +way to collaborate on designs over a single source of truth. -You can easily share mock-ups of designs with your team, or visual regressions can be easily +You can share mock-ups of designs with your team, or visual regressions can be viewed and addressed. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> @@ -36,10 +36,11 @@ to be enabled: and enable **Git Large File Storage**. Design Management also requires that projects are using -[hashed storage](../../../administration/raketasks/storage.md#migrate-to-hashed-storage). Since - GitLab 10.0, newly created projects use hashed storage by default. A GitLab administrator can verify the storage type of a -project by navigating to **Admin Area > Projects** and then selecting the project in question. -A project can be identified as hashed-stored if its *Gitaly relative path* contains `@hashed`. +[hashed storage](../../../administration/raketasks/storage.md#migrate-to-hashed-storage). +Newly created projects use hashed storage by default. A GitLab administrator +can verify the storage type of a project by going to **Admin Area > Projects** +and then selecting the project in question. A project can be identified as +hashed-stored if its *Gitaly relative path* contains `@hashed`. If the requirements are not met, the **Designs** tab displays a message to the user. @@ -74,8 +75,8 @@ and connect to GitLab through a personal access token. The details are explained ## The Design Management section -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223193) in GitLab 13.2, Designs are displayed directly on the issue description rather than on a separate tab. -> - New display's feature flag [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/223197) in GitLab 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223193) in GitLab 13.2. Designs are displayed directly in the issue description instead of a separate tab. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/223197) for new displays in GitLab 13.4. You can find to the **Design Management** section in the issue description: @@ -83,22 +84,26 @@ You can find to the **Design Management** section in the issue description: ## Adding designs +> - Drag and drop uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.9. +> - New version creation on upload [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.9. +> - Copy and paste uploads [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202634) in GitLab 12.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. + To upload Design images, drag files from your computer and drop them in the Design Management section, -or click **upload** to select images from your file browser: +or select **click to upload** to select images from your file browser:  -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.9, -you can drag and drop designs onto the dedicated drop zone to upload them. +You can drag and drop designs onto the dedicated drop zone to upload them.  -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202634) -in GitLab 12.10, you can also copy images from your file system and -paste them directly on the GitLab Design page as a new design. +You can also copy images from your file system and paste them directly on the +GitLab Design page as a new design. -On macOS you can also take a screenshot and immediately copy it to -the clipboard by simultaneously clicking <kbd>Control</kbd> + <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>3</kbd>, and then paste it as a design. +On macOS, you can take a screenshot and immediately copy it to the clipboard +by simultaneously pressing <kbd>Control</kbd> + <kbd>Command</kbd> + <kbd>Shift</kbd> + <kbd>3</kbd>, +and then paste it as a design. Copy-and-pasting has some limitations: @@ -108,24 +113,24 @@ Copy-and-pasting has some limitations: - Copy/pasting designs is not supported on Internet Explorer. Designs with the same filename as an existing uploaded design create a new version -of the design, and replaces the previous version. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.9, dropping a design on an existing uploaded design also creates a new version, -provided the filenames are the same. +of the design, and replaces the previous version. Dropping a design on an +existing uploaded design creates a new version if the filenames are the same. ### Skipped designs Designs with the same filename as an existing uploaded design _and_ whose content has not changed are skipped. -This means that no new version of the design is created. When designs are skipped, you are made aware via a warning +This means that no new version of the design is created. When designs are skipped, you are made aware by a warning message on the Issue. ## Viewing designs -Images on the Design Management page can be enlarged by clicking on them. -You can navigate through designs by clicking on the navigation buttons on the +Images on the Design Management page can be enlarged by selecting them. +You can navigate through designs by selecting the navigation buttons on the top-right corner or with <kbd>Left</kbd>/<kbd>Right</kbd> keyboard buttons. The number of discussions on a design — if any — is listed to the right -of the design filename. Clicking on this number enlarges the design -just like clicking anywhere else on the design. +of the design filename. Selecting this number enlarges the design, +similar to clicking or tapping anywhere else in the design. When a design is added or modified, an icon is displayed on the item to help summarize changes between versions. @@ -137,27 +142,29 @@ to help summarize changes between versions. ### Exploring designs by zooming -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13217) in GitLab Premium 12.7. +> - Drag to move a zoomed image [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. Designs can be explored in greater detail by zooming in and out of the image. Control the amount of zoom with the `+` and `-` buttons at the bottom of the image. While zoomed, you can still [start new discussions](#starting-discussions-on-designs) on the image, and see any existing ones. -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197324) in GitLab 12.10, while zoomed in, -you can click-and-drag on the image to move around it. +While zoomed in, you can drag the image to move around it.  ## Deleting designs -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11089) in GitLab Premium 12.4. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. There are two ways to delete designs: manually delete them individually, or select a few of them to delete at once, as shown below. -To delete a single design, click it to view it enlarged, -then click the trash icon on the top right corner and confirm -the deletion by clicking the **Delete** button on the modal window: +To delete a single design, select it to view it enlarged, +then select the trash icon on the top right corner and confirm +the deletion by selecting **Delete** in the window:  @@ -166,7 +173,7 @@ first select the designs you want to delete:  -Once selected, click the **Delete selected** button to confirm the deletion: +Select **Delete selected** to confirm the deletion:  @@ -183,14 +190,16 @@ You can change the order of designs by dragging them to a new position. ## Starting discussions on designs -When a design is uploaded, you can start a discussion by clicking on +> - Adjusting a pin's position [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in GitLab Premium 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212566) to GitLab Free in 13.0. + +When a design is uploaded, you can start a discussion by selecting the image on the exact location you would like the discussion to be focused on. A pin is added to the image, identifying the discussion's location.  -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34353) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8, -you can adjust a pin's position by dragging it around the image. This is useful +You can adjust a pin's position by dragging it around the image. This is useful for when your design layout has changed between revisions, or if you need to move an existing pin to add a new one in its place. @@ -198,7 +207,7 @@ Different discussions have different pin numbers:  -From GitLab 12.5 on, new discussions are outputted to the issue activity, +In GitLab 12.5 and later, new discussions are output to the issue activity, so that everyone involved can participate in the discussion. ## Resolve Design threads @@ -209,20 +218,20 @@ Discussion threads can be resolved on Designs. There are two ways to resolve/unresolve a Design thread: -1. You can mark a thread as resolved or unresolved by clicking the checkmark icon for **Resolve thread** in the top-right corner of the first comment of the discussion: +1. You can mark a thread as resolved or unresolved by selecting the checkmark icon for **Resolve thread** in the top-right corner of the first comment of the discussion: -  +  1. Design threads can also be resolved or unresolved in their threads by using a checkbox. - When replying to a comment, there is a checkbox that you can click in order to resolve or unresolve - the thread once published: + When replying to a comment, you can select or clear a checkbox to resolve or unresolve + the thread after publishing: -  +  Resolving a discussion thread also marks any pending to-do items related to notes inside the thread as done. This is applicable only for to-do items owned by the user triggering the action. -Note that your resolved comment pins disappear from the Design to free up space for new discussions. +Your resolved comment pins disappear from the Design to free up space for new discussions. However, if you need to revisit or find a resolved discussion, all of your resolved threads are available in the **Resolved Comment** area at the bottom of the right sidebar. @@ -231,19 +240,19 @@ available in the **Resolved Comment** area at the bottom of the right sidebar. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198439) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245074) in GitLab 13.5. -Add a to-do item for a design by clicking **Add a to do** on the design sidebar: +Add a to-do item for a design by selecting **Add a to do** on the design sidebar:  ## Referring to designs in Markdown -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217160) in **GitLab 13.1**. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258662) in **GitLab 13.5** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217160) in GitLab 13.1. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258662) in GitLab 13.5. We support referring to designs in [Markdown](../../markdown.md), which is available throughout the application, including in merge request and issue descriptions, in discussions and comments, and in wiki pages. -At present, full URL references are supported. For example, if we refer to a design +Full URL references are supported. For example, if we refer to a design somewhere with: ```markdown diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 5b8dd617ab9..94a5fdc3f0f 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -15,7 +15,7 @@ the issue can view the due date. When creating an issue, select the **Due date** field to make a calendar appear for choosing the date. To remove the date, select the date -text and delete it. The date is related to the server's timezone, not the timezone of +text and delete it. The date is related to the server's time zone, not the time zone of the user setting the due date.  @@ -45,7 +45,7 @@ Due dates also appear in your [to-do list](../../todos.md). The day before an open issue is due, an email is sent to all participants of the issue. Like the due date, the "day before the due date" is determined by the -server's timezone. +server's time zone. Issues with due dates can also be exported as an iCalendar feed. The URL of the feed can be added to calendar applications. The feed is accessible by selecting diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 7033b90b736..a2fa044be6b 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -212,7 +212,7 @@ that are not in status **closed** from one project to another. To access rails console run `sudo gitlab-rails console` on the GitLab server and run the below script. Please be sure to change `project`, `admin_user`, and `target_project` to your values. -We do also recommend [creating a backup](../../../raketasks/backup_restore.md#back-up-gitlab) before +We do also recommend [creating a backup](../../../raketasks/backup_restore.md) before attempting any changes in the console. ```ruby diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 43d6ab2070d..e9cbe012110 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -156,8 +156,6 @@ to the project: ## Scoped labels **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9175) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. - Scoped labels allow teams to use the label feature to annotate issues, merge requests and epics with mutually exclusive labels. This can enable more complicated workflows by preventing certain labels from being used together. @@ -180,6 +178,19 @@ For example: 1. GitLab automatically removes the `priority::low` label, as an issue should not have two priority labels at the same time. +### Filter by scoped labels + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12285) in GitLab 14.4. + +To filter issue, merge request, or epic lists for ones with labels that belong to a given scope, enter +`<scope>::*` in the searched label name. + +For example, filtering by the `platform::*` label returns issues that have `platform::iOS`, +`platform::Android`, or `platform::Linux` labels. + +NOTE: +This is not available on the [issues or merge requests dashboard pages](../search/index.md#issues-and-merge-requests). + ### Workflows with scoped labels Suppose you wanted a custom field in issues to track the operating system platform @@ -228,14 +239,14 @@ to label notifications for the project only, or the whole group. ## Label priority -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14189) in GitLab 8.9. -> - Priority sorting is based on the highest priority label only. [This discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/14523) considers changing this. - Labels can have relative priorities, which are used in the **Label priority** and **Priority** sort orders of issues and merge request list pages. Prioritization for both group and project labels happens at the project level, and cannot be done from the group label list. +NOTE: +Priority sorting is based on the highest priority label only. [This discussion](https://gitlab.com/gitlab-org/gitlab/-/issues/14523) considers changing this. + From the project label list page, star a label to indicate that it has a priority.  diff --git a/doc/user/project/members/img/project_members_filter_direct_v13_9.png b/doc/user/project/members/img/project_members_filter_direct_v13_9.png Binary files differdeleted file mode 100644 index 50115ee4052..00000000000 --- a/doc/user/project/members/img/project_members_filter_direct_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/project_members_filter_direct_v14_4.png b/doc/user/project/members/img/project_members_filter_direct_v14_4.png Binary files differnew file mode 100644 index 00000000000..79cee06bc30 --- /dev/null +++ b/doc/user/project/members/img/project_members_filter_direct_v14_4.png diff --git a/doc/user/project/members/img/project_members_filter_inherited_v13_9.png b/doc/user/project/members/img/project_members_filter_inherited_v13_9.png Binary files differdeleted file mode 100644 index 433003fe58b..00000000000 --- a/doc/user/project/members/img/project_members_filter_inherited_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/project_members_filter_inherited_v14_4.png b/doc/user/project/members/img/project_members_filter_inherited_v14_4.png Binary files differnew file mode 100644 index 00000000000..ce2a0ebf088 --- /dev/null +++ b/doc/user/project/members/img/project_members_filter_inherited_v14_4.png diff --git a/doc/user/project/members/img/project_members_search_v13_9.png b/doc/user/project/members/img/project_members_search_v13_9.png Binary files differdeleted file mode 100644 index 67280d11dca..00000000000 --- a/doc/user/project/members/img/project_members_search_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/project_members_search_v14_4.png b/doc/user/project/members/img/project_members_search_v14_4.png Binary files differnew file mode 100644 index 00000000000..8c52c5788d4 --- /dev/null +++ b/doc/user/project/members/img/project_members_search_v14_4.png diff --git a/doc/user/project/members/img/project_members_sort_v13_9.png b/doc/user/project/members/img/project_members_sort_v13_9.png Binary files differdeleted file mode 100644 index 47abe18ba49..00000000000 --- a/doc/user/project/members/img/project_members_sort_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/project_members_sort_v14_4.png b/doc/user/project/members/img/project_members_sort_v14_4.png Binary files differnew file mode 100644 index 00000000000..20834b9307e --- /dev/null +++ b/doc/user/project/members/img/project_members_sort_v14_4.png diff --git a/doc/user/project/members/img/project_members_v13_9.png b/doc/user/project/members/img/project_members_v13_9.png Binary files differdeleted file mode 100644 index 3b48c752c6a..00000000000 --- a/doc/user/project/members/img/project_members_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/project_members_v14_4.png b/doc/user/project/members/img/project_members_v14_4.png Binary files differnew file mode 100644 index 00000000000..0a235e91d28 --- /dev/null +++ b/doc/user/project/members/img/project_members_v14_4.png diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 8a70b74fcc1..f9788ef18ec 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -87,7 +87,7 @@ A success message is displayed and the new members are now displayed in the list When your project belongs to a group, group members inherit their role from the group. - + In this example: @@ -140,7 +140,7 @@ You can filter and sort members in a project. 1. In the **Filter members** box, select `Membership` `=` `Inherited`. 1. Press Enter. - + ### Display direct members @@ -148,19 +148,19 @@ You can filter and sort members in a project. 1. In the **Filter members** box, select `Membership` `=` `Direct`. 1. Press Enter. - + ### Search You can search for members by name, username, or email. - + ### Sort You can sort members by **Account**, **Access granted**, **Max role**, or **Last sign-in** in ascending or descending order. - + ## Request access to a project diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index e1149b85cd5..abca140411a 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index 7e168fb239a..b422982c0e7 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -154,7 +154,7 @@ become eligible approvers in the project. To enable this merge request approval 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Locate **Any eligible user** and select the number of approvals required: +1. Locate **Eligible users** and select the number of approvals required:  @@ -225,7 +225,7 @@ approval rule for certain branches: 1. Go to your project and select **Settings**. 1. Expand **Merge request (MR) approvals**. 1. Select a **Target branch**: - - To protect all branches, select **Any branch**. + - To protect all branches, select **All branches**. - To select a specific branch, select it from the list:  diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index ebd07f30f52..1c56e91ed6b 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, concepts --- -# Merge request approval settings **(FREE)** +# Merge request approval settings **(PREMIUM)** You can configure the settings for [merge request approvals](index.md) to ensure the approval rules meet your use case. You can also configure @@ -30,7 +30,7 @@ In this section of general settings, you can configure the following settings: | [Require user password to approve](#require-user-password-to-approve) | Force potential approvers to first authenticate with a password. | | [Remove all approvals when commits are added to the source branch](#remove-all-approvals-when-commits-are-added-to-the-source-branch) | When enabled, remove all existing approvals on a merge request when more changes are added to it. | -## Prevent approval by author **(PREMIUM)** +## Prevent approval by author > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3349) in GitLab 11.3. > - Moved to GitLab Premium in 13.9. @@ -52,7 +52,7 @@ this setting, unless you configure one of these options: at the instance level, you can't edit this setting at the project or individual merge request levels. -## Prevent approvals by users who add commits **(PREMIUM)** +## Prevent approvals by users who add commits > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10441) in GitLab 11.10. > - Moved to GitLab Premium in 13.9. @@ -126,13 +126,29 @@ merge request could introduce a vulnerability. To learn more, see [Security approvals in merge requests](../../../application_security/index.md#security-approvals-in-merge-requests). -## Code coverage check approvals **(PREMIUM)** +## Code coverage check approvals You can require specific approvals if a merge request would result in a decline in code test coverage. To learn more, see [Coverage check approval rule](../../../../ci/pipelines/settings.md#coverage-check-approval-rule). +## Merge request approval settings cascading + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.4. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md), disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md). On GitLab.com, this feature is not available. +You should not use this feature for production environments + +You can also enforce merge request approval settings: + +- At the [instance level](../../../admin_area/merge_requests_approvals.md), which apply to all groups on an instance and, therefore, all + projects. +- On a [top-level group](../../../group/index.md#group-approval-rules), which apply to all subgroups and projects. + +If the settings are inherited by a group or project, they cannot be overridden by the group or project that inherited them. + ## Related links - [Instance-level merge request approval settings](../../../admin_area/merge_requests_approvals.md) diff --git a/doc/user/project/merge_requests/confidential.md b/doc/user/project/merge_requests/confidential.md index 6df84dd1dd1..ff2e6acf123 100644 --- a/doc/user/project/merge_requests/confidential.md +++ b/doc/user/project/merge_requests/confidential.md @@ -11,11 +11,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w Merge requests in a public repository are also public, even when the merge request is created for a [confidential issue](../issues/confidential_issues.md). To avoid leaking confidential information when working on a confidential issue, -create your merge request from a private fork. +create your merge request from a private fork in the same namespace. Roles are inherited from parent groups. If you create your private fork in the -same group or subgroup as the original (public) repository, developers receive -the same permissions in your fork. This inheritance ensures: +same namespace (same group or subgroup) as the original (public) repository, +developers receive the same permissions in your fork. This inheritance ensures: - Developer users have the needed permissions to view confidential issues and resolve them. - You do not need grant individual users access to your fork. @@ -24,13 +24,17 @@ The [security practices for confidential merge requests](https://gitlab.com/gitl ## Create a confidential merge request -WARNING: -To create a confidential merge request, you must create a private fork. This fork -may expose confidential information, if you create your fork in another namespace -that may have other members. - Branches are public by default. To protect the confidentiality of your work, you -must create your changes in a private fork. +must create your branches and merge requests in the same namespace, but downstream +in a private fork. If you create your private fork in the same namespace as the +public repository, your fork inherits the permissions of the upstream public repository. +Users with the Developer role in the upstream public repository inherit those upstream +permissions in your downstream private fork without action by you. These users can +immediately push code to branches in your private fork to help fix the confidential issue. + +WARNING: +Your private fork may expose confidential information, if you create it in a different +namespace than the upstream repository. The two namespaces may not contain the same users. Prerequisites: @@ -56,16 +60,15 @@ To create a confidential merge request: branches must be available in your selected fork. 1. Select **Create**. -If you created a branch in your private fork, users with the Developer role in the -public repository can push code to that branch in your private fork to fix the -confidential issue. +This merge request targets your private fork, not the public upstream project. +Your branch, merge requests, and commits remain in your private fork. This prevents +prematurely revealing confidential information. -As your merge request targets your private fork, not the public upstream project, -your branch, merge request, and commits do not enter the public repository. This -prevents prematurely revealing confidential information. +Open a merge request +[from your fork to the upstream repository](../repository/forking_workflow.md#merging-upstream) when: -To make a confidential commit public, open a merge request from the private fork -to the public upstream project. +- You are satisfied the problem is resolved in your private fork. +- You are ready to make the confidential commits public. ## Related links diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index 72fcd7f36b0..cee4df1f61e 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -50,8 +50,9 @@ Learn the various ways to [create a merge request](creating_merge_requests.md). ## What you can do with merge requests When you start a new merge request, you can immediately include the following -options, or add them later by clicking the **Edit** button on the merge -request's page at the top-right side: +options. You can also add them later by either selecting **Edit** on the merge +request's page at the top-right side, or by using +[keyboard shortcuts for merge requests](../../shortcuts.md#issues-and-merge-requests): - [Assign](#assignee) the merge request to a colleague for review. With [multiple assignees](#multiple-assignees), you can assign it to more than one person at a time. - Set a [milestone](../milestones/index.md) to track time-sensitive changes. @@ -74,8 +75,10 @@ After you have created the merge request, you can also: - Add [code suggestions](reviews/suggestions.md) to change the content of merge requests directly into merge request threads, and easily apply them to the codebase directly from the UI. - Add a time estimation and the time spent with that merge request with [Time Tracking](../time_tracking.md#time-tracking). -Many of these can be set when pushing changes from the command line, -with [Git push options](../push_options.md). +Many of these options can be set: + +- From the merge request page, with [keyboard shortcuts](../../shortcuts.md#issues-and-merge-requests). +- When pushing changes from the command line, with [Git push options](../push_options.md). See also other [features associated to merge requests](reviews/index.md#associated-features). diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index b7e055ca749..2c062c2c592 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -40,14 +40,18 @@ important parts of the merge request: ## View merge requests -You can view merge requests for a specific project, or for all projects in a group: +To view a list of merge requests: -- **Specific project**: Go to your project and select **Merge requests**. +- **Merge requests for a project**: Go to your project and select **Merge requests**, or use + the <kbd>g</kbd> + <kbd>m</kbd> [keyboard shortcut](../../shortcuts.md) from a page in your project. - **All projects in a group**: Go to your group and select **Merge requests**. If your group contains subgroups, this view also displays merge requests from the subgroup projects. GitLab displays a count of open merge requests in the left sidebar, but [caches the value](reviews/index.md#cached-merge-request-count) for groups with a large number of open merge requests. +- **Merge requests assigned to you**: On any GitLab page, select **Merge requests** + in the top bar, or use the <kbd>Shift</kbd> + <kbd>m</kbd> + [global keyboard shortcut](../../shortcuts.md). GitLab displays open merge requests, with tabs to filter the list by open and closed status: @@ -153,3 +157,4 @@ For a web developer writing a webpage for your company's website: - [Review a merge request](reviews/index.md) - [Authorization for merge requests](authorization_for_merge_requests.md) - [Testing and reports](testing_and_reports_in_merge_requests.md) +- [GitLab keyboard shortcuts](../../shortcuts.md) diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index dbf3b0180e6..e6f84f1c357 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -158,7 +158,7 @@ Multiline comments display the comment's line numbers above the body of the comm Users with permission level of [Developer or higher](../../../permissions.md) can manage merge requests. -When bulk editing merge requests in a project, you can edit the following attributes: +When bulk-editing merge requests in a project, you can edit the following attributes: - Status (open/closed) - Assignee @@ -211,6 +211,8 @@ These features are associated with merge requests: GitLab can provide the option to resolve certain merge request conflicts in the GitLab UI. - [Revert changes](../revert_changes.md): Revert changes from any commit from a merge request. +- [Keyboard shortcuts](../../../shortcuts.md#issues-and-merge-requests): + Access and modify specific parts of a merge request with keyboard commands. ## Troubleshooting diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 4027ec08234..ac1cd57fb99 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -33,9 +33,8 @@ which generates a commit in the merge request authored by the user that suggeste  -1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) in GitLab 13.9, - you can opt to add a custom commit message to describe your change. If you don't - specify it, the default commit message is used. It is not supported for [batch suggestions](#batch-suggestions). +1. Optionally specify a custom commit message for individual suggestions (GitLab 13.9 and later) to + describe your change. If you don't specify it, the default commit message is used.  @@ -118,6 +117,7 @@ introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) behind a feature flag, disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/227799) in GitLab 13.2. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) in GitLab 13.11. +> - Custom commit messages for batch suggestions [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326168) in GitLab 14.4. You can apply multiple suggestions at once to reduce the number of commits added to your branch to address your reviewers' requests. @@ -134,7 +134,10 @@ to your branch to address your reviewers' requests.  -1. Having added all the suggestions to your liking, when ready, select **Apply suggestions**: +1. Having added all the suggestions to your liking, when ready, select **Apply suggestions**. You + can optionally specify a custom commit message for [batch suggestions](#batch-suggestions) + (GitLab 14.4 and later) to describe your change. If you don't specify it, the default commit + message is used.  diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index 399d7958bbf..08b82462187 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -93,7 +93,7 @@ for doesn't appear immediately. The search box requires **three** alphanumeric characters to be entered for the search to begin. If you want the status check to be applied to **all** merge requests, -you can select the **Any branch** option. +you can select the **All branches** option. ## Delete a status check diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index 813e3c1c9ce..b36510c2df8 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -177,8 +177,6 @@ coverage-jdk11: # convert report from jacoco to cobertura, using relative project path - python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml needs: ["test-jdk11"] - dependencies: - - test-jdk11 artifacts: reports: cobertura: target/site/cobertura.xml @@ -215,8 +213,6 @@ coverage-jdk11: # convert report from jacoco to cobertura, using relative project path - python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml needs: ["test-jdk11"] - dependencies: - - test-jdk11 artifacts: reports: cobertura: build/cobertura.xml @@ -235,7 +231,8 @@ run tests: image: python:3 script: - pip install pytest pytest-cov - - pytest --cov=src/ tests.py + - coverage run -m pytest + - coverage report - coverage xml artifacts: reports: diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 51f1ec96c22..27487003697 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -26,7 +26,7 @@ and steps below. - A custom domain name `example.com` or subdomain `subdomain.example.com`. - Access to your domain's server control panel to set up DNS records: - A DNS A or CNAME record pointing your domain to GitLab Pages server. - - A DNS TXT record to verify your domain's ownership. + - A DNS `TXT` record to verify your domain's ownership. ### Steps @@ -48,7 +48,7 @@ Click **Create New Domain**. #### 2. Get the verification code After you add a new domain to Pages, the verification code prompts you. Copy the values from GitLab -and paste them in your domain's control panel as a TXT record on the next step. +and paste them in your domain's control panel as a `TXT` record on the next step.  @@ -76,7 +76,7 @@ Root domains (`example.com`) require: | From | DNS Record | To | | --------------------------------------------- | ---------- | --------------- | | `example.com` | A | `35.185.44.232` | -| `_gitlab-pages-verification-code.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | For projects on GitLab.com, this IP is `35.185.44.232`. For projects living in other GitLab instances (CE or EE), please contact @@ -104,7 +104,7 @@ Subdomains (`subdomain.example.com`) require: | From | DNS Record | To | | ------------------------------------------------------- | ---------- | --------------------- | | `subdomain.example.com` | CNAME | `namespace.gitlab.io` | -| `_gitlab-pages-verification-code.subdomain.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.subdomain.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | Note that, whether it's a user or a project website, the `CNAME` should point to your Pages domain (`namespace.gitlab.io`), @@ -121,15 +121,15 @@ They require: - A DNS A record for the domain. - A DNS CNAME record for the subdomain. -- A DNS TXT record for each. +- A DNS `TXT` record for each. | From | DNS Record | To | | ------------------------------------------------- | ---------- | ---------------------- | | `example.com` | A | `35.185.44.232` | -| `_gitlab-pages-verification-code.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | |---------------------------------------------------+------------+------------------------| | `www.example.com` | CNAME | `namespace.gitlab.io` | -| `_gitlab-pages-verification-code.www.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.www.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | If you're using Cloudflare, check [Redirecting `www.domain.com` to `domain.com` with Cloudflare](#redirecting-wwwdomaincom-to-domaincom-with-cloudflare). @@ -196,15 +196,15 @@ For a root domain: | From | DNS Record | To | | ------------------------------------------------- | ---------- | ---------------------- | -| `example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | -| `_gitlab-pages-verification-code.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | For a subdomain: | From | DNS Record | To | | ------------------------------------------------- | ---------- | ---------------------- | -| `www.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | -| `_gitlab-pages-verification-code.www.example.com` | TXT | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `www.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | +| `_gitlab-pages-verification-code.www.example.com` | `TXT` | `gitlab-pages-verification-code=00112233445566778899aabbccddeeff` | ### Adding more domain aliases @@ -290,8 +290,6 @@ Sublime Text, Atom, Dreamweaver, Brackets, etc). ## Force HTTPS for GitLab Pages websites -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28857) in GitLab 10.7. - To make your website's visitors even more secure, you can choose to force HTTPS for GitLab Pages. By doing so, all attempts to visit your website through HTTP are automatically redirected to HTTPS through 301. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md index 5b7f6454ef7..21f2dd51f70 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -35,7 +35,7 @@ with financial transactions. <!-- vale gitlab.Spelling = NO --> -Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [ISRG](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): +Now we have a different picture. [According to Josh Aas](https://letsencrypt.org/2015/10/29/phishing-and-malware.html), Executive Director at [Internet Security Research Group (ISRG)](https://en.wikipedia.org/wiki/Internet_Security_Research_Group): <!-- vale gitlab.rulename = YES --> diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 385a4fafa7d..283ed0b61b9 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -7,14 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Pages **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80) in GitLab Enterprise Edition 8.3. -> - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173) in GitLab Enterprise Edition 8.5. -> - [Ported](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14605) to GitLab Community Edition in GitLab 8.17. -> - Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30548) in GitLab 11.8. -> - Bundled project templates were [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47857) in GitLab 11.8. - -With GitLab Pages, you can publish static websites -directly from a repository in GitLab. +With GitLab Pages, you can publish static websites directly from a repository +in GitLab. <div class="row"> <div class="col-md-9"> @@ -32,11 +26,11 @@ directly from a repository in GitLab. <div class="col-md-3"><img src="img/ssgs_pages.png" alt="Examples of SSGs supported by Pages" class="middle display-block"></div> </div> -To publish a website with Pages, you can use any SSG, -like Gatsby, Jekyll, Hugo, Middleman, Harp, Hexo, and Brunch, just to name a few. You can also +To publish a website with Pages, you can use any static site generator, +like Gatsby, Jekyll, Hugo, Middleman, Harp, Hexo, or Brunch. You can also publish any website written directly in plain HTML, CSS, and JavaScript. -Pages does **not** support dynamic server-side processing, for instance, as `.php` and `.asp` requires. +Pages does not support dynamic server-side processing, for instance, as `.php` and `.asp` requires. Learn more about [static websites compared to dynamic websites](https://about.gitlab.com/blog/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/). @@ -45,18 +39,18 @@ Learn more about To create a GitLab Pages website: | Document | Description | -| -------- | ----------- | -| [Create a `.gitlab-ci.yml` file from scratch](getting_started/pages_from_scratch.md) | Add a Pages site to an existing project. Learn how to create and configure your own CI file. | +|----------|-------------| +| [Create a `.gitlab-ci.yml` file from scratch](getting_started/pages_from_scratch.md) | Add a Pages site to an existing project. Learn how to create and configure your own CI file. | | [Use a `.gitlab-ci.yml` template](getting_started/pages_ci_cd_template.md) | Add a Pages site to an existing project. Use a pre-populated CI template file. | -| [Fork a sample project](getting_started/pages_forked_sample_project.md) | Create a new project with Pages already configured by forking a sample project. | -| [Use a project template](getting_started/pages_new_project_template.md) | Create a new project with Pages already configured by using a template. | +| [Fork a sample project](getting_started/pages_forked_sample_project.md) | Create a new project with Pages already configured by forking a sample project. | +| [Use a project template](getting_started/pages_new_project_template.md) | Create a new project with Pages already configured by using a template. | To update a GitLab Pages website: | Document | Description | -| -------- | ----------- | +|----------|-------------| | [GitLab Pages domain names, URLs, and base URLs](getting_started_part_one.md) | Learn about GitLab Pages default domains. | -| [Explore GitLab Pages](introduction.md) | Requirements, technical aspects, specific GitLab CI/CD configuration options, Access Control, custom 404 pages, limitations, FAQ. | +| [Explore GitLab Pages](introduction.md) | Requirements, technical aspects, specific GitLab CI/CD configuration options, Access Control, custom 404 pages, limitations, and FAQ. | | [Custom domains and SSL/TLS Certificates](custom_domains_ssl_tls_certification/index.md) | Custom domains and subdomains, DNS records, and SSL/TLS certificates. | | [Let's Encrypt integration](custom_domains_ssl_tls_certification/lets_encrypt_integration.md) | Secure your Pages sites with Let's Encrypt certificates, which are automatically obtained and renewed by GitLab. | | [Redirects](redirects.md) | Set up HTTP redirects to forward one page to another. | @@ -64,7 +58,7 @@ To update a GitLab Pages website: Learn more and see examples: | Document | Description | -| -------- | ----------- | +|----------|-------------| | [Static vs dynamic websites](https://about.gitlab.com/blog/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) | Static versus dynamic site overview. | | [Modern static site generators](https://about.gitlab.com/blog/2016/06/10/ssg-overview-gitlab-pages-part-2/) | SSG overview. | | [Build any SSG site with GitLab Pages](https://about.gitlab.com/blog/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) | Use SSGs for GitLab Pages. | @@ -74,7 +68,7 @@ Learn more and see examples: To use GitLab Pages, you must create a project in GitLab to upload your website's files to. These projects can be either public, internal, or private. -GitLab always deploys your website from a very specific folder called `public` in your +GitLab always deploys your website from a specific folder called `public` in your repository. When you create a new project in GitLab, a [repository](../repository/index.md) becomes available automatically. @@ -87,7 +81,7 @@ GitLab Pages website. You can either use the GitLab [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-default-domain-names), `*.gitlab.io`, or your own domain (`example.com`). In that case, you -need administrator access to your domain's registrar (or control panel) to set it up with Pages. +must have the Administrator role in your domain's registrar (or control panel) to set it up with Pages. The following diagrams show the workflows you might follow to get started with Pages. @@ -95,24 +89,21 @@ The following diagrams show the workflows you might follow to get started with P ## Access to your Pages site -If you're using GitLab Pages default domain (`.gitlab.io`), -your website is automatically secure and available under -HTTPS. If you're using your own custom domain, you can -optionally secure it with SSL/TLS certificates. +If you're using GitLab Pages default domain (`.gitlab.io`), your website is +automatically secure and available under HTTPS. If you're using your own custom +domain, you can optionally secure it with SSL/TLS certificates. If you're using GitLab.com, your website is publicly available to the internet. To restrict access to your website, enable [GitLab Pages Access Control](pages_access_control.md). -If you're using a self-managed instance (Free, Premium, or Ultimate), -your websites are published on your own server, according to the -[Pages settings](../../../administration/pages/index.md) chosen by your sysadmin, -who can make them public or internal. +If you're using a self-managed instance, your websites are published on your +own server, according to the [Pages settings](../../../administration/pages/index.md) +chosen by your sysadmin, who can make them public or internal. ## Pages examples -There are some great examples of GitLab Pages websites built for -specific reasons. These examples can teach you advanced techniques -to use and adapt to your own needs: +These GitLab Pages website examples can teach you advanced techniques to use +and adapt for your own needs: - [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/blog/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/). - [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/blog/2016/07/29/the-basics-of-gitlab-ci/). @@ -122,27 +113,27 @@ to use and adapt to your own needs: ## Administer GitLab Pages for self-managed instances -If you are running a self-managed instance of GitLab (GitLab Community Edition and Enterprise Editions), +If you are running a self-managed instance of GitLab, [follow the administration steps](../../../administration/pages/index.md) to configure Pages. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a [video tutorial](https://www.youtube.com/watch?v=dD8c7WNcc6s) about how to get started with GitLab Pages administration. ## Security for GitLab Pages -If your username is `foo`, your GitLab Pages website is located at `foo.gitlab.io`. -GitLab allows usernames to contain a `.`, so a user named `bar.foo` could create -a GitLab Pages website `bar.foo.gitlab.io` that effectively is a subdomain of your -`foo.gitlab.io` website. Be careful if you use JavaScript to set cookies for your website. +If your username is `example`, your GitLab Pages website is located at `example.gitlab.io`. +GitLab allows usernames to contain a `.`, so a user named `bar.example` could create +a GitLab Pages website `bar.example.gitlab.io` that effectively is a subdomain of your +`example.gitlab.io` website. Be careful if you use JavaScript to set cookies for your website. The safe way to manually set cookies with JavaScript is to not specify the `domain` at all: ```javascript -// Safe: This cookie is only visible to foo.gitlab.io +// Safe: This cookie is only visible to example.gitlab.io document.cookie = "key=value"; -// Unsafe: This cookie is visible to foo.gitlab.io and its subdomains, +// Unsafe: This cookie is visible to example.gitlab.io and its subdomains, // regardless of the presence of the leading dot. -document.cookie = "key=value;domain=.foo.gitlab.io"; -document.cookie = "key=value;domain=foo.gitlab.io"; +document.cookie = "key=value;domain=.example.gitlab.io"; +document.cookie = "key=value;domain=example.gitlab.io"; ``` This issue doesn't affect users with a custom domain, or users who don't set any diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 94656c91e98..45c2f1aaf04 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -118,7 +118,7 @@ pages: paths: - public only: - - master + - main ``` ### `.gitlab-ci.yml` for a static site generator @@ -133,7 +133,7 @@ the `pages` job with the [`only` parameter](../../../ci/yaml/index.md#only--exce whenever a new commit is pushed to a branch used specifically for your pages. -That way, you can have your project's code in the `master` branch and use an +That way, you can have your project's code in the `main` branch and use an orphan branch (let's name it `pages`) to host your static generator site. You can create a new empty branch like this: @@ -163,7 +163,7 @@ pages: - pages ``` -See an example that has different files in the [`master` branch](https://gitlab.com/pages/jekyll-branched/tree/master) +See an example that has different files in the [`main` branch](https://gitlab.com/pages/jekyll-branched/tree/main) and the source files for Jekyll are in a [`pages` branch](https://gitlab.com/pages/jekyll-branched/tree/pages) which also includes `.gitlab-ci.yml`. diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index c17133e72cf..305bbb2ded0 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -61,7 +61,7 @@ time as pushing changes: | Push option | Description | Introduced in version | | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------- | | `merge_request.create` | Create a new merge request for the pushed branch. | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | -| `merge_request.target=<branch_name>` | Set the target of the merge request to a particular branch. | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | +| `merge_request.target=<branch_name>` | Set the target of the merge request to a particular branch or upstream project, such as: `git push -o merge_request.target=project_path/branch` | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | | `merge_request.merge_when_pipeline_succeeds` | Set the merge request to [merge when its pipeline succeeds](merge_requests/merge_when_pipeline_succeeds.md). | [11.10](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26752) | | `merge_request.remove_source_branch` | Set the merge request to remove the source branch when it's merged. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | | `merge_request.title="<title>"` | Set the title of the merge request. Ex: `git push -o merge_request.title="The title I want"`. | [12.2](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/64320) | diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 52b59d70302..45a83394c41 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -103,7 +103,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/target_branch <local branch name>` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Set target branch. | | `/title <new title>` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Change title. | | `/todo` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Add a to-do item. | -| `/unapprove` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Unapprove the merge request. ([introduced in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/8003)| +| `/unapprove` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Unapprove the merge request. ([introduced in GitLab 14.3](https://gitlab.com/gitlab-org/gitlab/-/issues/8103)| | `/unassign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Remove specific assignees. | | `/unassign` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Remove all assignees. | | `/unassign_reviewer @user1 @user2` or `/remove_reviewer @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Remove specific reviewers. | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 49b5ec2ca60..9e4d621dbc6 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -218,7 +218,7 @@ To set a deploy freeze window in the UI, complete these steps: 1. Scroll to **Deploy freezes**. 1. Click **Expand** to see the deploy freeze table. 1. Click **Add deploy freeze** to open the deploy freeze modal. -1. Enter the start time, end time, and timezone of the desired deploy freeze period. +1. Enter the start time, end time, and time zone of the desired deploy freeze period. 1. Click **Add deploy freeze** in the modal. 1. After the deploy freeze is saved, you can edit it by selecting the edit button (**{pencil}**) and remove it by selecting the delete button (**{remove}**).  diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index a1ea929bb49..5cd025f017d 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -11,8 +11,9 @@ When you create a new [project](../../index.md), GitLab creates a default branch in the repository. A default branch has special configuration options not shared by other branches: +- It cannot be deleted. - It's [initially protected](../../protected_branches.md#protected-branches) against - accidental deletion and forced pushes. + forced pushes. - When a merge request uses an [issue closing pattern](../../issues/managing_issues.md#closing-issues-automatically) to close an issue, the work is merged into this branch. @@ -97,7 +98,7 @@ Ensure they understand the scope of this change includes references to the old branch name in related code and scripts. When changing the default branch name for an existing repository, you should preserve -the history of your default branch by renaming it, instead of deleting it. This example +the history of your default branch by renaming it, instead of creating a new branch. This example renames a Git repository's (`example`) default branch. 1. On your local command line, navigate to your `example` repository, and ensure @@ -195,3 +196,32 @@ To fix the problem: ``` 1. In GitLab, [change the default branch](#change-the-default-branch-name-for-a-project) to the one you intend to use. + +### Query GraphQL for default branches + +You can use a [GraphQL query](../../../../api/graphql/index.md) +to retrieve the default branches for all projects in a group. + +To return all projects in a single page of results, replace `GROUPNAME` with the +full path to your group. GitLab returns the first page of results. If `hasNextPage` +is `true`, you can request the next page by replacing the `null` in `after: null` +with the value of `endCursor`: + +```graphql +{ + group(fullPath: "GROUPNAME") { + projects(after: null) { + pageInfo { + hasNextPage + endCursor + } + nodes { + name + repository { + rootRef + } + } + } + } +} +``` diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index 91858ff9a65..351daaaca3b 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -2,7 +2,6 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: concepts, howto --- # Branches **(FREE)** @@ -57,8 +56,6 @@ To compare branches in a repository: ## Delete merged branches -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6449) in GitLab 8.14. -  This feature allows merged branches to be deleted in bulk. Only branches that @@ -83,8 +80,6 @@ Search results appear in the following order: ## Branch filter search box -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22166) in GitLab 11.5. -  This feature allows you to search and select branches quickly. Search results appear in the following order: diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 33ab5f6580d..bf4ef21e31b 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -52,7 +52,7 @@ of the fork must manually change the visibility. Issue ## Repository mirroring -You can use [repository mirroring](repository_mirroring.md) to keep your fork synced with the original repository. You can also use `git remote add upstream` to achieve the same result. +You can use [repository mirroring](mirror/index.md) to keep your fork synced with the original repository. You can also use `git remote add upstream` to achieve the same result. The main difference is that with repository mirroring, your remote fork is automatically kept up-to-date. diff --git a/doc/user/project/repository/img/web_editor_new_branch_dropdown_v14_1.png b/doc/user/project/repository/img/web_editor_new_branch_dropdown_v14_1.png Binary files differindex 20d76797977..df5e803d77a 100644 --- a/doc/user/project/repository/img/web_editor_new_branch_dropdown_v14_1.png +++ b/doc/user/project/repository/img/web_editor_new_branch_dropdown_v14_1.png diff --git a/doc/user/project/repository/img/web_editor_new_branch_from_issue_v14_1.png b/doc/user/project/repository/img/web_editor_new_branch_from_issue_v14_1.png Binary files differindex 150ec3ebf90..732173d9c1b 100644 --- a/doc/user/project/repository/img/web_editor_new_branch_from_issue_v14_1.png +++ b/doc/user/project/repository/img/web_editor_new_branch_from_issue_v14_1.png diff --git a/doc/user/project/repository/img/web_editor_new_directory_dropdown_v14_1.png b/doc/user/project/repository/img/web_editor_new_directory_dropdown_v14_1.png Binary files differindex 326605baaab..bbdb9bca199 100644 --- a/doc/user/project/repository/img/web_editor_new_directory_dropdown_v14_1.png +++ b/doc/user/project/repository/img/web_editor_new_directory_dropdown_v14_1.png diff --git a/doc/user/project/repository/img/web_editor_new_file_dropdown_v14_1.png b/doc/user/project/repository/img/web_editor_new_file_dropdown_v14_1.png Binary files differindex 4a4c1f8cd11..1a92555457a 100644 --- a/doc/user/project/repository/img/web_editor_new_file_dropdown_v14_1.png +++ b/doc/user/project/repository/img/web_editor_new_file_dropdown_v14_1.png diff --git a/doc/user/project/repository/img/web_editor_template_dropdown_mit_license_v14_1.png b/doc/user/project/repository/img/web_editor_template_dropdown_mit_license_v14_1.png Binary files differindex d0bdefaa437..5a5562ed38c 100644 --- a/doc/user/project/repository/img/web_editor_template_dropdown_mit_license_v14_1.png +++ b/doc/user/project/repository/img/web_editor_template_dropdown_mit_license_v14_1.png diff --git a/doc/user/project/repository/img/web_editor_upload_file_dropdown_v14_1.png b/doc/user/project/repository/img/web_editor_upload_file_dropdown_v14_1.png Binary files differindex 1b2fa59726a..ad949aae8ce 100644 --- a/doc/user/project/repository/img/web_editor_upload_file_dropdown_v14_1.png +++ b/doc/user/project/repository/img/web_editor_upload_file_dropdown_v14_1.png diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md new file mode 100644 index 00000000000..c4254655fcc --- /dev/null +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -0,0 +1,171 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.html' +--- + +# Bidirectional mirroring **(PREMIUM)** + +> Moved to GitLab Premium in 13.9. + +WARNING: +Bidirectional mirroring may cause conflicts. + +Bidirectional [mirroring](index.md) configures two repositories to both pull from, +and push to, each other. There is no guarantee that either repository can update +without errors. + +## Reduce conflicts in bidirectional mirroring + +If you configure bidirectional mirroring, prepare your repositories for +conflicts. Configure them to reduce conflicts, and how to settle them when they occur: + +- [Mirror only protected branches](index.md#mirror-only-protected-branches). Rewriting + any mirrored commit on either remote causes conflicts and mirroring to fail. +- [Protect the branches](../../protected_branches.md) you want to mirror on both + remotes to prevent conflicts caused by rewriting history. +- Reduce mirroring delay with a [push event webhook](../../integrations/webhook_events.md#push-events). + Bidirectional mirroring creates a race condition where commits made close together + to the same branch cause conflicts. Push event webhooks can help mitigate the race + condition. Push mirroring from GitLab is rate limited to once per minute when only + push mirroring protected branches. +- Prevent conflicts [using a pre-receive hook](#prevent-conflicts-by-using-a-pre-receive-hook). + +## Configure a webhook to trigger an immediate pull to GitLab + +A [push event webhook](../../integrations/webhook_events.md#push-events) in the downstream +instance can help reduce race conditions by syncing changes more frequently. + +Prerequisites: + +- You have configured the [push](push.md#set-up-a-push-mirror-to-another-gitlab-instance-with-2fa-activated) +and [pull](pull.md#pull-from-a-remote-repository) mirrors in the upstream GitLab instance. + +To create the webhook in the downstream instance: + +1. Create a [personal access token](../../../profile/personal_access_tokens.md) with `API` scope. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Webhooks**. +1. Add the webhook **URL**, which (in this case) uses the + [Pull Mirror API](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project) + request to trigger an immediate pull after a repository update: + + ```plaintext + https://gitlab.example.com/api/v4/projects/:id/mirror/pull?private_token=<your_access_token> + ``` + +1. Select **Push Events**. +1. Select **Add Webhook**. + +To test the integration, select **Test** and confirm GitLab doesn't return an error message. + +## Prevent conflicts by using a pre-receive hook + +WARNING: +This solution negatively affects the performance of Git push operations, because +they are proxied to the upstream Git repository. + +In this configuration, one Git repository acts as the authoritative upstream, and +the other as downstream. This server-side `pre-receive` hook accepts a push only +after first pushing the commit to the upstream repository. Install this hook on +your downstream repository. + +For example: + +```shell +#!/usr/bin/env bash + +# --- Assume only one push mirror target +# Push mirroring remotes are named `remote_mirror_<id>`. +# This line finds the first remote and uses that. +TARGET_REPO=$(git remote | grep -m 1 remote_mirror) + +proxy_push() +{ + # --- Arguments + OLDREV=$(git rev-parse $1) + NEWREV=$(git rev-parse $2) + REFNAME="$3" + + # --- Pattern of branches to proxy pushes + allowlist=$(expr "$branch" : "\(master\)") + + case "$refname" in + refs/heads/*) + branch=$(expr "$refname" : "refs/heads/\(.*\)") + + if [ "$allowlist" = "$branch" ]; then + # handle https://git-scm.com/docs/git-receive-pack#_quarantine_environment + unset GIT_QUARANTINE_PATH + error="$(git push --quiet $TARGET_REPO $NEWREV:$REFNAME 2>&1)" + fail=$? + + if [ "$fail" != "0" ]; then + echo >&2 "" + echo >&2 " Error: updates were rejected by upstream server" + echo >&2 " This is usually caused by another repository pushing changes" + echo >&2 " to the same ref. You may want to first integrate remote changes" + echo >&2 "" + return + fi + fi + ;; + esac +} + +# Allow dual mode: run from the command line just like the update hook, or +# if no arguments are given, then run as a hook script: +if [ -n "$1" -a -n "$2" -a -n "$3" ]; then + # Output to the terminal in command line mode. If someone wanted to + # resend an email, they could redirect the output to sendmail themselves + PAGER= proxy_push $2 $3 $1 +else + # Push is proxied upstream one ref at a time. It is possible for some refs + # to succeed, and others to fail. This results in a failed push. + while read oldrev newrev refname + do + proxy_push $oldrev $newrev $refname + done +fi +``` + +This sample has a few limitations: + +- It may not work for your use case without modification: + - It doesn't regard different types of authentication mechanisms for the mirror. + - It doesn't work with forced updates (rewriting history). + - Only branches that match the `allowlist` patterns are proxy pushed. +- The script circumvents the Git hook quarantine environment because the update of `$TARGET_REPO` + is seen as a ref update, and Git displays warnings about it. + +## Mirror with Perforce Helix via Git Fusion **(PREMIUM)** + +> Moved to GitLab Premium in 13.9. + +WARNING: +Bidirectional mirroring should not be used as a permanent configuration. Refer to +[Migrating from Perforce Helix](../../import/perforce.md) for alternative migration approaches. + +[Git Fusion](https://www.perforce.com/manuals/git-fusion/#Git-Fusion/section_avy_hyc_gl.html) provides a Git interface +to [Perforce Helix](https://www.perforce.com/products). GitLab can use the Perforce Helix +interface to bidirectionally mirror projects. It can help when migrating from Perforce Helix +to GitLab, if overlapping Perforce Helix workspaces cannot be migrated simultaneously. + +If you mirror with Perforce Helix, mirror only protected branches. Perforce Helix +rejects any pushes that rewrite history. Only the fewest number of branches should be mirrored +due to the performance limitations of Git Fusion. + +When you configure mirroring with Perforce Helix by using Git Fusion, we recommend these Git Fusion +settings: + +- Disable `change-pusher`. Otherwise, every commit is rewritten as being committed + by the mirroring account, rather than mapping to existing Perforce Helix users or the `unknown_git` user. +- Use the `unknown_git` user as the commit author, if the GitLab user doesn't exist in + Perforce Helix. + +Read about [Git Fusion settings on Perforce.com](https://www.perforce.com/manuals/git-fusion/Content/Git-Fusion/section_vss_bdw_w3.html#section_zdp_zz1_3l). + +## Related topics + +- [Configure server hooks](../../../../administration/server_hooks.md) on a GitLab server. diff --git a/doc/user/project/repository/img/repository_mirroring_copy_ssh_public_key_button.png b/doc/user/project/repository/mirror/img/repository_mirroring_copy_ssh_public_key_button.png Binary files differindex e20dae09a4d..e20dae09a4d 100644 --- a/doc/user/project/repository/img/repository_mirroring_copy_ssh_public_key_button.png +++ b/doc/user/project/repository/mirror/img/repository_mirroring_copy_ssh_public_key_button.png diff --git a/doc/user/project/repository/img/repository_mirroring_force_update.png b/doc/user/project/repository/mirror/img/repository_mirroring_force_update.png Binary files differindex 1e6dcb9ea08..1e6dcb9ea08 100644 --- a/doc/user/project/repository/img/repository_mirroring_force_update.png +++ b/doc/user/project/repository/mirror/img/repository_mirroring_force_update.png diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md new file mode 100644 index 00000000000..4532a80c2f5 --- /dev/null +++ b/doc/user/project/repository/mirror/index.md @@ -0,0 +1,224 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.html' +--- + +# Repository mirroring **(FREE)** + +Repository mirroring allows for the mirroring of repositories to and from external sources. You +can use it to mirror branches, tags, and commits between repositories. It helps you use +a repository outside of GitLab. + +A repository mirror at GitLab updates automatically. You can also manually trigger an update: + +- At most once every five minutes on GitLab.com. +- According to a [limit set by the administrator](../../../../administration/instance_limits.md#pull-mirroring-interval) + on self-managed instances. + +There are two kinds of repository mirroring supported by GitLab: + +- [Push](push.md): for mirroring a GitLab repository to another location. +- [Pull](pull.md): for mirroring a repository from another location to GitLab. + +When the mirror repository is updated, all new branches, tags, and commits are visible in the +project's activity feed. + +Users with the [Maintainer role](../../../permissions.md) for the project can also force an +immediate update, unless: + +- The mirror is already being updated. +- The [limit for pull mirroring interval seconds](../../../../administration/instance_limits.md#pull-mirroring-interval) has not elapsed after its last update. + +For security reasons, the URL to the original repository is only displayed to users with the +[Maintainer role](../../../permissions.md) or the [Owner role](../../../permissions.md) for the mirrored +project. + +## Use cases + +The following are some possible use cases for repository mirroring: + +- You migrated to GitLab but still must keep your project in another source. In that case, you + can set it up to mirror to GitLab (pull) and all the essential history of commits, tags, + and branches are available in your GitLab instance. **(PREMIUM)** +- You have old projects in another source that you don't use actively anymore, but don't want to + remove for archiving purposes. In that case, you can create a push mirror so that your active + GitLab repository can push its changes to the old location. +- You are a GitLab self-managed user for privacy reasons and your instance is closed to the public, + but you still have certain software components that you want open sourced. In this case, utilizing + GitLab to be your primary repository which is closed from the public, and using push mirroring to a + GitLab.com repository that's public, allows you to open source specific projects and contribute back + to the open source community. + +## Mirror only protected branches **(PREMIUM)** + +> Moved to GitLab Premium in 13.9. + +Based on the mirror direction that you choose, you can opt to mirror only the +[protected branches](../../protected_branches.md) in the mirroring project, +either from or to your remote repository. For pull mirroring, non-protected branches in +the mirroring project are not mirrored and can diverge. + +To use this option, check the **Only mirror protected branches** box when +creating a repository mirror. **(PREMIUM)** + +## SSH authentication + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22982) in GitLab 11.6 for Push mirroring. + +SSH authentication is mutual: + +- You have to prove to the server that you're allowed to access the repository. +- The server also has to prove to *you* that it's who it claims to be. + +You provide your credentials as a password or public key. The server that the +other repository resides on provides its credentials as a "host key", the +fingerprint of which needs to be verified manually. + +If you're mirroring over SSH (using an `ssh://` URL), you can authenticate using: + +- Password-based authentication, just as over HTTPS. +- Public key authentication. This is often more secure than password authentication, + especially when the other repository supports [deploy keys](../../deploy_keys/index.md). + +To get started: + +1. In your project, go to **Settings > Repository**, and then expand the **Mirroring repositories** section. +1. Enter an `ssh://` URL for mirroring. + +NOTE: +SCP-style URLs (that is, `git@example.com:group/project.git`) are not supported at this time. + +Entering the URL adds two buttons to the page: + +- **Detect host keys**. +- **Input host keys manually**. + +If you select the: + +- **Detect host keys** button, GitLab fetches the host keys from the server and display the fingerprints. +- **Input host keys manually** button, a field is displayed where you can paste in host keys. + +Assuming you used the former, you now must verify that the fingerprints are +those you expect. GitLab.com and other code hosting sites publish their +fingerprints in the open for you to check: + +- [AWS CodeCommit](https://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints) +- [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/) +- [GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints) +- [GitLab.com](../../../gitlab_com/index.md#ssh-host-keys-fingerprints) +- [Launchpad](https://help.launchpad.net/SSHFingerprints) +- [Savannah](http://savannah.gnu.org/maintenance/SshAccess/) +- [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/) + +Other providers vary. If you're running self-managed GitLab, or otherwise +have access to the server for the other repository, you can securely gather the +key fingerprints: + +```shell +$ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f - +256 MD5:f4:28:9f:23:99:15:21:1b:bf:ed:1f:8e:a0:76:b2:9d root@example.com (ECDSA) +256 MD5:e6:eb:45:8a:3c:59:35:5f:e9:5b:80:12:be:7e:22:73 root@example.com (ED25519) +2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA) +``` + +NOTE: +You must exclude `-E md5` for some older versions of SSH. + +When mirroring the repository, GitLab checks that at least one of the +stored host keys matches before connecting. This can prevent malicious code from +being injected into your mirror, or your password being stolen. + +### SSH public key authentication + +To use SSH public key authentication, you must also choose that option +from the **Authentication method** dropdown. When the mirror is created, +GitLab generates a 4096-bit RSA key that can be copied by selecting the **Copy SSH public key** button. + + + +You then must add the public SSH key to the other repository's configuration: + +- If the other repository is hosted on GitLab, you should add the public SSH key + as a [deploy key](../../../project/deploy_keys/index.md). +- If the other repository is hosted elsewhere, you must add the key to + your user's `authorized_keys` file. Paste the entire public SSH key into the + file on its own line and save it. + +If you must change the key at any time, you can remove and re-add the mirror +to generate a new key. Update the other repository with the new +key to keep the mirror running. + +NOTE: +The generated keys are stored in the GitLab database, not in the file system. Therefore, +SSH public key authentication for mirrors cannot be used in a pre-receive hook. + +## Force an update **(FREE)** + +While mirrors are scheduled to update automatically, you can always force an update by using the +update button which is available on the **Mirroring repositories** section of the **Repository Settings** page. + + + +## Resources + +- Configure a [Pull Mirroring Interval](../../../../administration/instance_limits.md#pull-mirroring-interval) +- [Disable mirrors for a project](../../../admin_area/settings/visibility_and_access_controls.md#enable-project-mirroring) +- [Secrets file and mirroring](../../../../raketasks/backup_restore.md#when-the-secrets-file-is-lost) + +## Troubleshooting + +Should an error occur during a push, GitLab displays an **Error** highlight for that repository. Details on the error can then be seen by hovering over the highlight text. + +### 13:Received RST_STREAM with error code 2 with GitHub + +If you receive a "13:Received RST_STREAM with error code 2" message while mirroring to a GitHub repository, +your GitHub settings might be set to block pushes that expose your email address used in commits. Either +set your email address on GitHub to be public, or disable the [Block command line pushes that expose my email](https://github.com/settings/emails) setting. + +### 4:Deadline Exceeded + +When upgrading to GitLab 11.11.8 or newer, a change in how usernames are represented means that you +must update your mirroring username and password to ensure that `%40` characters are replaced with `@`. + +### Connection blocked because server only allows public key authentication + +As the error indicates, the connection is getting blocked between GitLab and the remote repository. Even if a +[TCP Check](../../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) is successful, +you must check any networking components in the route from GitLab to the remote Server to ensure there's no blockage. + +For example, we've seen this error when a Firewall was performing a `Deep SSH Inspection` on outgoing packets. + +### Could not read username: terminal prompts disabled + +If you receive this error after creating a new project using +[GitLab CI/CD for external repositories](../../../../ci/ci_cd_for_external_repos/): + +```plaintext +"2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': terminal prompts disabled\n": exit status 128." +``` + +Check if the repository owner is specified in the URL of your mirrored repository: + +1. Go to your project. +1. On the left sidebar, select **Settings > Repository**. +1. Select **Mirroring repositories**. +1. If no repository owner is specified, delete and add the URL again in this format: + + ```plaintext + https://**<repo_owner>**@bitbucket.org/<accountname>/<reponame>.git + ``` + +The repository owner is needed for Bitbucket to connect to the repository for mirroring. + +### Pull mirror is missing LFS files + +In some cases, pull mirroring does not transfer LFS files. This issue occurs when: + +- You use an SSH repository URL. The workaround is to use an HTTPS repository URL instead. + There is [an issue to fix this for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). +- You're using GitLab 14.0 or older, and the source repository is a public Bitbucket URL. + This was [fixed in GitLab 14.0.6](https://gitlab.com/gitlab-org/gitlab/-/issues/335123). +- You mirror an external repository using object storage. + There is [an issue to fix this](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md new file mode 100644 index 00000000000..d1943cbfd71 --- /dev/null +++ b/doc/user/project/repository/mirror/pull.md @@ -0,0 +1,121 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.html' +--- + +# Pull from a remote repository **(PREMIUM)** + +> Moved to GitLab Premium in 13.9. + +You can use the GitLab interface to browse the content and activity of a repository, +even if it isn't hosted on GitLab. Create a pull [mirror](index.md) to copy the +branches, tags, and commits from an upstream repository to yours. + +Unlike [push mirrors](push.md), pull mirrors retrieve changes from an upstream (remote) +repository on a scheduled basis. To prevent the mirror from diverging from the upstream +repository, don't push commits directly to the downstream mirror. Push commits to +the upstream repository instead. Changes in the remote repository are pulled into the GitLab repository, either: + +- Automatically in a certain period of time. Self-managed instances can + configure [pull mirroring intervals](../../../../administration/instance_limits.md#pull-mirroring-interval). +- When an administrator [force-updates the mirror](index.md#force-an-update). +- When an [API call triggers an update](#trigger-an-update-by-using-the-api). + +By default, if any branch or tag on the downstream pull mirror diverges from the +local repository, GitLab stops updating the branch. This prevents data loss. +Deleted branches and tags in the upstream repository are not reflected in the +downstream repository. + +## How pull mirroring works + +After you configure a GitLab repository as a pull mirror: + +1. GitLab adds the repository to a queue. +1. Once per minute, a Sidekiq cron job schedules repository mirrors to update, based on: + - Available capacity, determined by Sidekiq settings. For GitLab.com, read + [GitLab.com Sidekiq settings](../../../gitlab_com/index.md#sidekiq). + - How many mirrors are already in the queue and due for updates. Being due depends + on when the repository mirror was last updated, and how many times updates have been retried. +1. Sidekiq becomes available to process updates, mirrors are updated. If the update process: + - **Succeeds**: An update is enqueued again with at least a 30 minute wait. + - **Fails**: The update is attempted again later. After 14 failures, a mirror is marked as a + [hard failure](#hard-failure) and is no longer enqueued for updates. A branch diverging + from its upstream counterpart can cause failures. To prevent branches from + diverging, configure [Overwrite diverged branches](#overwrite-diverged-branches) when + you create your mirror. + +## Configure pull mirroring + +Prerequisite: + +- If your remote repository is on GitHub and you have + [two-factor authentication (2FA) configured](https://docs.github.com/en/authentication/securing-your-account-with-two-factor-authentication-2fa), + create a [personal access token for GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) + with the `repo` scope. If 2FA is enabled, this personal access + token serves as your GitHub password. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. Enter the **Git repository URL**. Include the username + in the URL, if required: `https://MYUSERNAME@github.com/GROUPNAME/PROJECTNAME.git` +1. In **Mirror direction**, select **Pull**. +1. In **Authentication method**, select your authentication method. +1. Select any of the options you need: + - [**Overwrite diverged branches**](#overwrite-diverged-branches) + - [**Trigger pipelines for mirror updates**](#trigger-pipelines-for-mirror-updates) + - **Only mirror protected branches** +1. To save the configuration, select **Mirror repository**. + +### Overwrite diverged branches + +> Moved to GitLab Premium in 13.9. + +To always update your local branches with remote versions, even if they have +diverged from the remote, select **Overwrite diverged branches** when you +create a mirror. + +WARNING: +For mirrored branches, enabling this option results in the loss of local changes. + +### Trigger pipelines for mirror updates + +> Moved to GitLab Premium in 13.9. + +If this option is enabled, pipelines trigger when branches or tags are +updated from the remote repository. Depending on the activity of the remote +repository, this may greatly increase the load on your CI runners. Only enable +this feature if you know they can handle the load. CI uses the credentials +assigned when you set up pull mirroring. + +## Trigger an update by using the API + +> Moved to GitLab Premium in 13.9. + +Pull mirroring uses polling to detect new branches and commits added upstream, +often minutes afterwards. If you notify GitLab by +[API](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project), +updates are pulled immediately. + +For more information, read +[Start the pull mirroring process for a project](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project). + +## Hard failure + +> Moved to GitLab Premium in 13.9. + +After 14 consecutive unsuccessful retries, the mirroring process is marked as a hard failure +and mirroring attempts stop. This failure is visible in either the: + +- Project's main dashboard. +- Pull mirror settings page. + +You can resume the project mirroring again by [forcing an update](index.md#force-an-update). + +## Related topics + +- Configure [pull mirroring intervals](../../../../administration/instance_limits.md#pull-mirroring-interval) + on self-managed instances. +- Configure [pull mirroring through the API](../../../../api/projects.md#configure-pull-mirroring-for-a-project). diff --git a/doc/user/project/repository/mirror/push.md b/doc/user/project/repository/mirror/push.md new file mode 100644 index 00000000000..498b8d063a9 --- /dev/null +++ b/doc/user/project/repository/mirror/push.md @@ -0,0 +1,197 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.html' +--- + +# Push mirroring **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40137) in GitLab 13.5: LFS support over HTTPS. + +A _push mirror_ is a downstream repository that [mirrors](index.md) the commits made +to the upstream repository. Push mirrors passively receive copies of the commits made to the +upstream repository. To prevent the mirror from diverging from the upstream +repository, don't push commits directly to the downstream mirror. Push commits to +the upstream repository instead. + +While [pull mirroring](pull.md) periodically retrieves updates from the upstream repository, +push mirrors only receive changes when: + +- Commits are pushed to the upstream GitLab repository. +- An administrator [force-updates the mirror](index.md#force-an-update). + +When you push a change to the upstream repository, the push mirror receives it: + +- Within five minutes. +- Within one minute, if you enabled **Only mirror protected branches**. + +In the case of a diverged branch, an error displays in the **Mirroring repositories** +section. + +## Configure push mirroring + +To set up push mirroring for an existing project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. Enter a repository URL. +1. In the **Mirror direction** dropdown list, select **Push**. +1. Select an **Authentication method**. + You can authenticate with either a password or an [SSH key](index.md#ssh-authentication). +1. Select **Only mirror protected branches**, if necessary. +1. Select **Keep divergent refs**, if desired. +1. To save the configuration, select **Mirror repository**. + +### Configure push mirrors through the API + +You can also create and modify project push mirrors through the +[remote mirrors API](../../../../api/remote_mirrors.md). + +## Keep divergent refs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208828) in GitLab 13.0. + +By default, if any ref (branch or tag) on the remote (downstream) mirror diverges from the +local repository, the upstream repository overwrites any changes on the remote: + +1. A repository mirrors `main` and `develop` branches to a remote. +1. A new commit is added to `develop` on the remote mirror. +1. The next push updates the remote mirror to match the upstream repository. +1. The new commit added to `develop` on the remote mirror is lost. + +If **Keep divergent refs** is selected, the changes are handled differently: + +1. Updates to the `develop` branch on the remote mirror are skipped. +1. The `develop` branch on the remote mirror preserves the commit that does not + exist on the upstream repository. Any refs that exist in the remote mirror, + but not the upstream, are left untouched. +1. The update is marked failed. + +After you create a mirror, you can only modify the value of **Keep divergent refs** +through the [remote mirrors API](../../../../api/remote_mirrors.md). + +## Set up a push mirror from GitLab to GitHub + +To configure a mirror from GitLab to GitHub: + +1. Create a [GitHub personal access token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) + with `public_repo` selected. +1. Enter a **Git repository URL** with this format: + `https://<your_github_username>@github.com/<your_github_group>/<your_github_project>.git`. +1. For **Password**, enter your GitHub personal access token. +1. Select **Mirror repository**. + +The mirrored repository is listed. For example: + +```plaintext +https://*****:*****@github.com/<your_github_group>/<your_github_project>.git +``` + +The repository pushes shortly thereafter. To force a push, select **Update now** (**{retry}**). + +## Set up a push mirror from GitLab to AWS CodeCommit + +AWS CodeCommit push mirroring is the best way to connect GitLab repositories to +AWS CodePipeline. GitLab is not yet supported as one of their Source Code Management (SCM) providers. +Each new AWS CodePipeline needs significant AWS infrastructure setup. It also +requires an individual pipeline per branch. + +If AWS CodeDeploy is the final step of a CodePipeline, you can, instead combine +these tools to create a deployment: + +- GitLab CI/CD pipelines. +- The AWS CLI in the final job in `.gitlab-ci.yml` to deploy to CodeDeploy. + +NOTE: +GitLab-to-AWS-CodeCommit push mirroring cannot use SSH authentication until [GitLab issue 34014](https://gitlab.com/gitlab-org/gitlab/-/issues/34014) is resolved. + +To set up a mirror from GitLab to AWS CodeCommit: + +1. In the AWS IAM console, create an IAM user. +1. Add the following least privileges permissions for repository mirroring as an **inline policy**. + + The Amazon Resource Names (ARNs) must explicitly include the region and account. This IAM policy + grants privilege for mirroring access to two sample repositories. These permissions have + been tested to be the minimum (least privileged) required for mirroring: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "MinimumGitLabPushMirroringPermissions", + "Effect": "Allow", + "Action": [ + "codecommit:GitPull", + "codecommit:GitPush" + ], + "Resource": [ + "arn:aws:codecommit:us-east-1:111111111111:MyDestinationRepo", + "arn:aws:codecommit:us-east-1:111111111111:MyDemo*" + ] + } + ] + } + ``` + +1. After the user is created, select the AWS IAM user name. +1. Select the **Security credentials** tab. +1. Under **HTTPS Git credentials for AWS CodeCommit**, select **Generate credentials**. + + NOTE: + This Git user ID and password is specific to communicating with CodeCommit. Do + not confuse it with the IAM user ID or AWS keys of this user. + +1. Copy or download the special Git HTTPS user ID and password. +1. In the AWS CodeCommit console, create a new repository to mirror from your GitLab repository. +1. Open your new repository, and then select **Clone URL > Clone HTTPS** (not **Clone HTTPS (GRC)**). +1. In GitLab, open the repository to be push-mirrored. +1. On the left sidebar, select **Settings > Repository**, and then expand **Mirroring repositories**. +1. Fill in the **Git repository URL** field using this format: + + ```plaintext + https://<your_aws_git_userid>@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> + ``` + + Replace `<your_aws_git_userid>` with the AWS **special HTTPS Git user ID** + from the IAM Git credentials created earlier. Replace `<your_codecommit_repo>` + with the name of your repository in CodeCommit. + +1. For **Mirror direction**, select **Push**. +1. For **Authentication method**, select **Password**. Fill in the **Password** box + with the special IAM Git clone user ID **password** created earlier in AWS. +1. Leave the option **Only mirror protected branches** for CodeCommit. It pushes more + frequently (from every five minutes to every minute). + + CodePipeline requires individual pipeline setups for named branches you want + to have a AWS CI setup for. Because feature branches with dynamic names are unsupported, + configuring **Only mirror protected branches** doesn't cause flexibility problems + with CodePipeline integration. You must also protect all the named branches you + want to build CodePipelines for. + +1. Select **Mirror repository**. You should see the mirrored repository appear: + + ```plaintext + https://*****:*****@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> + ``` + +To test mirroring by forcing a push, select **Update now** (the half-circle arrows). +If **Last successful update** shows a date, you have configured mirroring correctly. +If it isn't working correctly, a red `error` tag appears, and shows the error message as hover text. + +## Set up a push mirror to another GitLab instance with 2FA activated + +1. On the destination GitLab instance, create a + [personal access token](../../../profile/personal_access_tokens.md) with `write_repository` scope. +1. On the source GitLab instance: + 1. Enter the **Git repository URL** using this format: + `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. + 1. Enter the **Password**. Use the GitLab personal access token created on the + destination GitLab instance. + 1. Select **Mirror repository**. + +## Related topics + +- [Remote mirrors API](../../../../api/remote_mirrors.md). diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 81429ea5384..0094e0b1b15 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -63,6 +63,12 @@ To purge files from a GitLab repository: git clone --bare --mirror /path/to/project.bundle ``` +1. Navigate to the `project.git` directory: + + ```shell + cd project.git + ``` + 1. Using `git filter-repo`, purge any files from the history of your repository. Because we are trying to remove internal refs, we rely on the `commit-map` produced by each run to tell us which internal refs to remove. diff --git a/doc/user/project/repository/repository_mirroring.md b/doc/user/project/repository/repository_mirroring.md index 5a02a35fce1..8fbe5aec6a3 100644 --- a/doc/user/project/repository/repository_mirroring.md +++ b/doc/user/project/repository/repository_mirroring.md @@ -1,635 +1,9 @@ --- -stage: Create -group: Source Code -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.html' +redirect_to: 'mirror/index.md' +remove_date: '2022-03-22' --- -# Repository mirroring **(FREE)** +This document was moved to [another location](mirror/index.md). -Repository mirroring allows for the mirroring of repositories to and from external sources. You -can use it to mirror branches, tags, and commits between repositories. It helps you use -a repository outside of GitLab. - -A repository mirror at GitLab updates automatically. You can also manually trigger an update: - -- At most once every five minutes on GitLab.com. -- According to a [limit set by the administrator](../../../administration/instance_limits.md#pull-mirroring-interval) - on self-managed instances. - -There are two kinds of repository mirroring supported by GitLab: - -- [Push](#push-to-a-remote-repository): for mirroring a GitLab repository to another location. -- [Pull](#pull-from-a-remote-repository): for mirroring a repository from another location to GitLab. - -When the mirror repository is updated, all new branches, tags, and commits are visible in the -project's activity feed. - -Users with the [Maintainer role](../../permissions.md) for the project can also force an -immediate update, unless: - -- The mirror is already being updated. -- The [limit for pull mirroring interval seconds](../../../administration/instance_limits.md#pull-mirroring-interval) has not elapsed since its last update. - -For security reasons, the URL to the original repository is only displayed to users with the -[Maintainer role](../../permissions.md) or the [Owner role](../../permissions.md) for the mirrored -project. - -## Use cases - -The following are some possible use cases for repository mirroring: - -- You migrated to GitLab but still need to keep your project in another source. In that case, you - can set it up to mirror to GitLab (pull) and all the essential history of commits, tags, - and branches are available in your GitLab instance. **(PREMIUM)** -- You have old projects in another source that you don't use actively anymore, but don't want to - remove for archiving purposes. In that case, you can create a push mirror so that your active - GitLab repository can push its changes to the old location. -- You are a GitLab self-managed user for privacy reasons and your instance is closed to the public, - but you still have certain software components that you want open sourced. In this case, utilizing - GitLab to be your primary repository which is closed from the public, and using push mirroring to a - GitLab.com repository that's public, allows you to open source specific projects and contribute back - to the open source community. - -## Push to a remote repository - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40137) in GitLab 13.5: LFS support over HTTPS. - -For an existing project, you can set up push mirroring as follows: - -1. In your project, go to **Settings > Repository**, and then expand the **Mirroring repositories** section. -1. Enter a repository URL. -1. In the **Mirror direction** dropdown, select **Push**. -1. Select an authentication method from the **Authentication method** dropdown. - You can authenticate with either a password or an [SSH key](#ssh-authentication). -1. Select the **Only mirror protected branches** checkbox, if necessary. -1. Select the **Keep divergent refs** checkbox, if desired. -1. Select **Mirror repository** to save the configuration. - -When push mirroring is enabled, only push commits directly to the mirrored repository to prevent the -mirror diverging. - -Unlike [pull mirroring](#how-it-works), the mirrored repository is not periodically auto-synced. -The mirrored repository receives all changes only when: - -- Commits are pushed to GitLab. -- A [forced update](#force-an-update) is initiated. - -Changes pushed to files in the repository are automatically pushed to the remote mirror at least: - -- Within five minutes of being received. -- Within one minute if **Only mirror protected branches** is enabled. - -In the case of a diverged branch, an error displays in the **Mirroring repositories** -section. - -### Configure push mirrors through the API - -You can also create and modify project push mirrors through the -[remote mirrors API](../../../api/remote_mirrors.md). - -### Keep divergent refs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208828) in GitLab 13.0. - -By default, if any ref (branch or tag) on the remote mirror has diverged from the local repository, the local differences are forced to the remote. - -For example, if a repository has `main` and `develop` branches that -have been mirrored to a remote, and then a new commit is added to `develop` on -the remote mirror. The next push updates all of the references on the remote mirror to match -the local repository, and the new commit added to the remote `develop` branch is lost. - -With the **Keep divergent refs** option enabled, the `develop` branch is -skipped, causing only `main` to be updated. The mirror status -reflects that `develop` has diverged and was skipped, and be marked as a -failed update. Refs that exist in the mirror repository but not in the local -repository are left untouched. - -NOTE: -After the mirror is created, this option can only be modified via the [API](../../../api/remote_mirrors.md). - -### Set up a push mirror from GitLab to GitHub - -To set up a mirror from GitLab to GitHub, you need to follow these steps: - -1. Create a [GitHub personal access token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) with the `public_repo` box checked. -1. Fill in the **Git repository URL** field using this format: `https://<your_github_username>@github.com/<your_github_group>/<your_github_project>.git`. -1. Fill in **Password** field with your GitHub personal access token. -1. Select **Mirror repository**. - -The mirrored repository is listed. For example, `https://*****:*****@github.com/<your_github_group>/<your_github_project>.git`. - -The repository pushes shortly thereafter. To force a push, select the **Update now** (**{retry}**) button. - -### Set up a push mirror from GitLab to AWS CodeCommit - -AWS CodeCommit push mirroring is the best way to connect GitLab repositories to -AWS CodePipeline, as GitLab isn't yet supported as one of their Source Code Management (SCM) providers. - -Each new AWS CodePipeline needs significant AWS infrastructure setup. It also -requires an individual pipeline per branch. - -If AWS CodeDeploy is the final step of a CodePipeline, you can, instead, leverage -GitLab CI/CD pipelines and use the AWS CLI in the final job in `.gitlab-ci.yml` -to deploy to CodeDeploy. - -NOTE: -GitLab-to-AWS-CodeCommit push mirroring cannot use SSH authentication until [GitLab issue 34014](https://gitlab.com/gitlab-org/gitlab/-/issues/34014) is resolved. - -To set up a mirror from GitLab to AWS CodeCommit: - -1. In the AWS IAM console, create an IAM user. -1. Add the following least privileges permissions for repository mirroring as an "inline policy". - - The Amazon Resource Names (ARNs) must explicitly include the region and account. The IAM policy - below grants privilege for mirroring access to two sample repositories. These permissions have - been tested to be the minimum (least privileged) required for mirroring: - - ```json - { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "MinimumGitLabPushMirroringPermissions", - "Effect": "Allow", - "Action": [ - "codecommit:GitPull", - "codecommit:GitPush" - ], - "Resource": [ - "arn:aws:codecommit:us-east-1:111111111111:MyDestinationRepo", - "arn:aws:codecommit:us-east-1:111111111111:MyDemo*" - ] - } - ] - } - ``` - -1. After the user was created, select the AWS IAM user name. -1. Select the **Security credentials** tab. -1. Under **HTTPS Git credentials for AWS CodeCommit** select **Generate credentials**. - - NOTE: - This Git user ID and password is specific to communicating with CodeCommit. Do - not confuse it with the IAM user ID or AWS keys of this user. - -1. Copy or download special Git HTTPS user ID and password. -1. In the AWS CodeCommit console, create a new repository to mirror from your GitLab repository. -1. Open your new repository, and then select **Clone URL > Clone HTTPS** (not **Clone HTTPS (GRC)**). -1. In GitLab, open the repository to be push-mirrored. -1. Go to **Settings > Repository**, and then expand **Mirroring repositories**. -1. Fill in the **Git repository URL** field using this format: - - ```plaintext - https://<your_aws_git_userid>@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> - ``` - - Replace `<your_aws_git_userid>` with the AWS **special HTTPS Git user ID** from the IAM Git - credentials created earlier. Replace `<your_codecommit_repo>` with the name of your repository in CodeCommit. - -1. For **Mirror direction**, select **Push**. -1. For **Authentication method**, select **Password** and fill in the **Password** field with the special IAM Git clone user ID **password** created earlier in AWS. -1. The option **Only mirror protected branches** should be good for CodeCommit as it pushes more - frequently (from every five minutes to every minute). - CodePipeline requires individual pipeline setups for named branches you wish to have a AWS CI setup for. Because feature branches that have dynamic names are unsupported, configuring **Only mirror protected branches** doesn't cause flexibility problems with CodePipeline integration as long as you are also willing to protect all the named branches you want to build CodePipelines for. - -1. Select **Mirror repository**. You should see the mirrored repository appear: - - ```plaintext - https://*****:*****@git-codecommit.<aws-region>.amazonaws.com/v1/repos/<your_codecommit_repo> - ``` - -To test mirroring by forcing a push, select the half-circle arrows button (hover text is **Update now**). -If **Last successful update** shows a date, you have configured mirroring correctly. -If it isn't working correctly, a red `error` tag appears and shows the error message as hover text. - -### Set up a push mirror to another GitLab instance with 2FA activated - -1. On the destination GitLab instance, create a [personal access token](../../profile/personal_access_tokens.md) with `write_repository` scope. -1. On the source GitLab instance: - 1. Fill in the **Git repository URL** field using this format: `https://oauth2@<destination host>/<your_gitlab_group_or_name>/<your_gitlab_project>.git`. - 1. Fill in the **Password** field with the GitLab personal access token created on the destination GitLab instance. - 1. Select **Mirror repository**. - -## Pull from a remote repository **(PREMIUM)** - -> - [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab/-/issues/10871) in GitLab 11.11. -> - Moved to GitLab Premium in 13.9. - -You can set up a repository to automatically have its branches, tags, and commits updated from an -upstream repository. - -If a repository you're interested in is located on a different server, and you want -to browse its content and its activity using the GitLab interface, you can configure -mirror pulling: - -1. If your remote repository is on GitHub and you have - [two-factor authentication (2FA) configured](https://docs.github.com/en/github/authenticating-to-github/securing-your-account-with-two-factor-authentication-2fa), - create a [personal access token for GitHub](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token). - with the `repo` scope. If 2FA is enabled, this personal access - token serves as your GitHub password. -1. In your project, go to **Settings > Repository**, and then expand the - **Mirroring repositories** section. -1. In the **Git repository URL** field, enter a repository URL. Include the username - in the URL if required: `https://MYUSERNAME@github.com/group/PROJECTNAME.git` -1. In the **Mirror direction** dropdown, select **Pull**. -1. In the **Authentication method** dropdown, select your authentication method. -1. Select from the following checkboxes, if needed: - - **Overwrite diverged branches** - - **Trigger pipelines for mirror updates** - - **Only mirror protected branches** -1. Select **Mirror repository** to save the configuration. - -Because GitLab is now set to pull changes from the upstream repository, you should not push commits -directly to the repository on GitLab. Instead, any commits should be pushed to the remote repository. -Changes pushed to the remote repository are pulled into the GitLab repository, either: - -- Automatically in a certain period of time. -- When a [forced update](#force-an-update) is initiated. - -WARNING: -If you do manually update a branch in the GitLab repository, the branch becomes diverged from -upstream, and GitLab no longer automatically updates this branch to prevent any changes from being lost. -Deleted branches and tags in the upstream repository are not reflected in the GitLab repository. - -### How it works - -After the pull mirroring feature has been enabled for a repository, the repository is added to a queue. - -Once per minute, a Sidekiq cron job schedules repository mirrors to update, based on: - -- The capacity available. This is determined by Sidekiq settings. For GitLab.com, see [GitLab.com Sidekiq settings](../../gitlab_com/index.md#sidekiq). -- The number of repository mirrors already in the queue that are due to be updated. Being due depends on when the repository mirror was last updated and how many times it's been retried. - -Repository mirrors are updated as Sidekiq becomes available to process them. If the process of updating the repository mirror: - -- **Succeeds**: An update is enqueued again with at least a 30 minute wait. -- **Fails**: (For example, a branch diverged from upstream.), The update attempted again later. Mirrors can fail - up to 14 times before they are no longer enqueued for updates. - -### Overwrite diverged branches **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -You can choose to always update your local branches with remote versions, even if they have -diverged from the remote. - -WARNING: -For mirrored branches, enabling this option results in the loss of local changes. - -To use this option, check the **Overwrite diverged branches** box when creating a repository mirror. - -### Trigger pipelines for mirror updates **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -If this option is enabled, pipelines trigger when branches or tags are -updated from the remote repository. Depending on the activity of the remote -repository, this may greatly increase the load on your CI runners. Only enable -this if you know they can handle the load. CI uses the credentials -assigned when you set up pull mirroring. - -### Hard failure **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -After 14 consecutive unsuccessful retries, the mirroring process is marked as a hard failure -and mirroring attempts stop. This failure is visible in either the: - -- Project's main dashboard. -- Pull mirror settings page. - -You can resume the project mirroring again by [forcing an update](#force-an-update). - -### Trigger an update using the API **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -Pull mirroring uses polling to detect new branches and commits added upstream, often minutes -afterwards. If you notify GitLab by [API](../../../api/projects.md#start-the-pull-mirroring-process-for-a-project), -updates are pulled immediately. - -For more information, see [Start the pull mirroring process for a Project](../../../api/projects.md#start-the-pull-mirroring-process-for-a-project). - -## Mirror only protected branches **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -Based on the mirror direction that you choose, you can opt to mirror only the -[protected branches](../protected_branches.md) in the mirroring project, -either from or to your remote repository. For pull mirroring, non-protected branches in -the mirroring project are not mirrored and can diverge. - -To use this option, check the **Only mirror protected branches** box when -creating a repository mirror. **(PREMIUM)** - -## SSH authentication - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22982) in GitLab 11.6 for Push mirroring. - -SSH authentication is mutual: - -- You have to prove to the server that you're allowed to access the repository. -- The server also has to prove to *you* that it's who it claims to be. - -You provide your credentials as a password or public key. The server that the -other repository resides on provides its credentials as a "host key", the -fingerprint of which needs to be verified manually. - -If you're mirroring over SSH (using an `ssh://` URL), you can authenticate using: - -- Password-based authentication, just as over HTTPS. -- Public key authentication. This is often more secure than password authentication, - especially when the other repository supports [deploy keys](../deploy_keys/index.md). - -To get started: - -1. In your project, go to **Settings > Repository**, and then expand the **Mirroring repositories** section. -1. Enter an `ssh://` URL for mirroring. - -NOTE: -SCP-style URLs (that is, `git@example.com:group/project.git`) are not supported at this time. - -Entering the URL adds two buttons to the page: - -- **Detect host keys**. -- **Input host keys manually**. - -If you select the: - -- **Detect host keys** button, GitLab fetches the host keys from the server and display the fingerprints. -- **Input host keys manually** button, a field is displayed where you can paste in host keys. - -Assuming you used the former, you now need to verify that the fingerprints are -those you expect. GitLab.com and other code hosting sites publish their -fingerprints in the open for you to check: - -- [AWS CodeCommit](https://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints) -- [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/) -- [GitHub](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints) -- [GitLab.com](../../gitlab_com/index.md#ssh-host-keys-fingerprints) -- [Launchpad](https://help.launchpad.net/SSHFingerprints) -- [Savannah](http://savannah.gnu.org/maintenance/SshAccess/) -- [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/) - -Other providers vary. If you're running self-managed GitLab, or otherwise -have access to the server for the other repository, you can securely gather the -key fingerprints: - -```shell -$ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f - -256 MD5:f4:28:9f:23:99:15:21:1b:bf:ed:1f:8e:a0:76:b2:9d root@example.com (ECDSA) -256 MD5:e6:eb:45:8a:3c:59:35:5f:e9:5b:80:12:be:7e:22:73 root@example.com (ED25519) -2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA) -``` - -NOTE: -You may need to exclude `-E md5` for some older versions of SSH. - -When mirroring the repository, GitLab checks that at least one of the -stored host keys matches before connecting. This can prevent malicious code from -being injected into your mirror, or your password being stolen. - -### SSH public key authentication - -To use SSH public key authentication, you must also choose that option -from the **Authentication method** dropdown. When the mirror is created, -GitLab generates a 4096-bit RSA key that can be copied by selecting the **Copy SSH public key** button. - - - -You then need to add the public SSH key to the other repository's configuration: - -- If the other repository is hosted on GitLab, you should add the public SSH key - as a [deploy key](../../project/deploy_keys/index.md). -- If the other repository is hosted elsewhere, you may need to add the key to - your user's `authorized_keys` file. Paste the entire public SSH key into the - file on its own line and save it. - -If you need to change the key at any time, you can remove and re-add the mirror -to generate a new key. Update the other repository with the new -key to keep the mirror running. - -NOTE: -The generated keys are stored in the GitLab database, not in the file system. Therefore, -SSH public key authentication for mirrors cannot be used in a pre-receive hook. - -## Force an update **(FREE)** - -While mirrors are scheduled to update automatically, you can always force an update by using the -update button which is available on the **Mirroring repositories** section of the **Repository Settings** page. - - - -## Bidirectional mirroring **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -WARNING: -Bidirectional mirroring may cause conflicts. - -If you configure a GitLab repository to both pull from, and push to, the same remote source, there -is no guarantee that either repository updates correctly. If you set up a repository for -bidirectional mirroring, you should prepare for the likely conflicts by deciding who resolves -them and how. - -Rewriting any mirrored commit on either remote causes conflicts and mirroring to fail. This can -be prevented by [mirroring only protected branches](#mirror-only-protected-branches). - -You should [protect the branches](../protected_branches.md) you wish to mirror on both -remotes to prevent conflicts caused by rewriting history. - -Bidirectional mirroring also creates a race condition where commits made close together to the same -branch causes conflicts. The race condition can be mitigated by reducing the mirroring delay by using -a [Push event webhook](../integrations/webhooks.md#push-events) to trigger an immediate -pull to GitLab. Push mirroring from GitLab is rate limited to once per minute when only push mirroring -protected branches. - -### Configure a webhook to trigger an immediate pull to GitLab - -Assuming you have already configured the [push](#set-up-a-push-mirror-to-another-gitlab-instance-with-2fa-activated) -and [pull](#pull-from-a-remote-repository) mirrors in the upstream GitLab instance, to trigger an -immediate pull as suggested above, you must configure a [Push Event Web Hook](../integrations/webhooks.md#push-events) -in the downstream instance. - -To do this: - -1. Create a [personal access token](../../profile/personal_access_tokens.md) with `API` scope. -1. In your project, go to **Settings > Webhooks**. -1. Add the webhook URL which (in this case) uses the [Pull Mirror API](../../../api/projects.md#start-the-pull-mirroring-process-for-a-project) - request to trigger an immediate pull after updates to the repository. - - ```plaintext - https://gitlab.example.com/api/v4/projects/:id/mirror/pull?private_token=<your_access_token> - ``` - -1. Ensure the **Push Events** checkbox is selected. -1. Select **Add Webhook** to save the webhook. - -To test the integration, select the **Test** button and confirm GitLab doesn't return an error message. - -### Prevent conflicts using a pre-receive hook - -WARNING: -The solution proposed negatively affects the performance of -Git push operations because they are proxied to the upstream Git -repository. - -A server-side `pre-receive` hook can be used to prevent the race condition -described above by only accepting the push after first pushing the commit to -the upstream Git repository. In this configuration one Git repository acts as -the authoritative upstream, and the other as downstream. The `pre-receive` hook -is installed on the downstream repository. - -Read about [configuring Server hooks](../../../administration/server_hooks.md) on the GitLab server. - -A sample `pre-receive` hook is provided below. - -```shell -#!/usr/bin/env bash - -# --- Assume only one push mirror target -# Push mirroring remotes are named `remote_mirror_<id>`, this finds the first remote and uses that. -TARGET_REPO=$(git remote | grep -m 1 remote_mirror) - -proxy_push() -{ - # --- Arguments - OLDREV=$(git rev-parse $1) - NEWREV=$(git rev-parse $2) - REFNAME="$3" - - # --- Pattern of branches to proxy pushes - allowlist=$(expr "$branch" : "\(master\)") - - case "$refname" in - refs/heads/*) - branch=$(expr "$refname" : "refs/heads/\(.*\)") - - if [ "$allowlist" = "$branch" ]; then - unset GIT_QUARANTINE_PATH # handle https://git-scm.com/docs/git-receive-pack#_quarantine_environment - error="$(git push --quiet $TARGET_REPO $NEWREV:$REFNAME 2>&1)" - fail=$? - - if [ "$fail" != "0" ]; then - echo >&2 "" - echo >&2 " Error: updates were rejected by upstream server" - echo >&2 " This is usually caused by another repository pushing changes" - echo >&2 " to the same ref. You may want to first integrate remote changes" - echo >&2 "" - return - fi - fi - ;; - esac -} - -# Allow dual mode: run from the command line just like the update hook, or -# if no arguments are given then run as a hook script -if [ -n "$1" -a -n "$2" -a -n "$3" ]; then - # Output to the terminal in command line mode - if someone wanted to - # resend an email; they could redirect the output to sendmail - # themselves - PAGER= proxy_push $2 $3 $1 -else - # Push is proxied upstream one ref at a time. Because of this it is possible - # for some refs to succeed, and others to fail. This will result in a failed - # push. - while read oldrev newrev refname - do - proxy_push $oldrev $newrev $refname - done -fi -``` - -Note that this sample has a few limitations: - -- This example may not work verbatim for your use case and might need modification. - - It doesn't regard different types of authentication mechanisms for the mirror. - - It doesn't work with forced updates (rewriting history). - - Only branches that match the `allowlist` patterns are proxy pushed. -- The script circumvents the Git hook quarantine environment because the update of `$TARGET_REPO` - is seen as a ref update, and Git displays warnings about it. - -### Mirror with Perforce Helix via Git Fusion **(PREMIUM)** - -> Moved to GitLab Premium in 13.9. - -WARNING: -Bidirectional mirroring should not be used as a permanent configuration. Refer to -[Migrating from Perforce Helix](../import/perforce.md) for alternative migration approaches. - -[Git Fusion](https://www.perforce.com/manuals/git-fusion/#Git-Fusion/section_avy_hyc_gl.html) provides a Git interface -to [Perforce Helix](https://www.perforce.com/products) which can be used by GitLab to bidirectionally -mirror projects with GitLab. This can help you in some situations when migrating from Perforce Helix -to GitLab where overlapping Perforce Helix workspaces cannot be migrated simultaneously to GitLab. - -If using mirroring with Perforce Helix, you should only mirror protected branches. Perforce Helix -rejects any pushes that rewrite history. Only the fewest number of branches should be mirrored -due to the performance limitations of Git Fusion. - -When configuring mirroring with Perforce Helix via Git Fusion, the following Git Fusion -settings are recommended: - -- `change-pusher` should be disabled. Otherwise, every commit is rewritten as being committed - by the mirroring account, rather than being mapped to existing Perforce Helix users or the `unknown_git` user. -- `unknown_git` user is used as the commit author if the GitLab user doesn't exist in - Perforce Helix. - -Read about [Git Fusion settings on Perforce.com](https://www.perforce.com/manuals/git-fusion/Content/Git-Fusion/section_vss_bdw_w3.html#section_zdp_zz1_3l). - -## Troubleshooting - -Should an error occur during a push, GitLab displays an **Error** highlight for that repository. Details on the error can then be seen by hovering over the highlight text. - -### 13:Received RST_STREAM with error code 2 with GitHub - -If you receive a "13:Received RST_STREAM with error code 2" message while mirroring to a GitHub repository, -your GitHub settings might be set to block pushes that expose your email address used in commits. Either -set your email address on GitHub to be public, or disable the [Block command line pushes that expose my email](https://github.com/settings/emails) setting. - -### 4:Deadline Exceeded - -When upgrading to GitLab 11.11.8 or newer, a change in how usernames are represented means that you -may need to update your mirroring username and password to ensure that `%40` characters are replaced with `@`. - -### Connection blocked because server only allows public key authentication - -As the error indicates, the connection is getting blocked between GitLab and the remote repository. Even if a -[TCP Check](../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) is successful, -you must check any networking components in the route from GitLab to the remote Server to ensure there's no blockage. - -For example, we've seen this error when a Firewall was performing a `Deep SSH Inspection` on outgoing packets. - -### Could not read username: terminal prompts disabled - -If you receive this error after creating a new project using -[GitLab CI/CD for external repositories](../../../ci/ci_cd_for_external_repos/): - -```plaintext -"2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': terminal prompts disabled\n": exit status 128." -``` - -Check if the repository owner is specified in the URL of your mirrored repository: - -1. Go to your project. -1. On the left sidebar, select **Settings > Repository**. -1. Select **Mirroring repositories**. -1. If no repository owner is specified, delete and add the URL again in this format: - - ```plaintext - https://**<repo_owner>**@bitbucket.org/<accountname>/<reponame>.git - ``` - -The repository owner is needed for Bitbucket to connect to the repository for mirroring. - -### Pull mirror is missing LFS files - -In some cases, pull mirroring does not transfer LFS files. This issue occurs when: - -- You use an SSH repository URL. The workaround is to use an HTTPS repository URL instead. - There is [an issue to fix this for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). -- You're using GitLab 14.0 or older, and the source repository is a public Bitbucket URL. - This was [fixed in GitLab 14.0.6](https://gitlab.com/gitlab-org/gitlab/-/issues/335123). -- You mirror an external repository using object storage. - There is [an issue to fix this](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). +<!-- This redirect file can be deleted after <2022-03-22>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 3be3b1682bb..661bd3e0598 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -7,8 +7,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Requirements Management **(ULTIMATE)** +NOTE: +In 14.4, Requirements was moved under **Issues**. + > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2703) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.10. > - The ability to add and edit a requirement's long description [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/224622) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.5. +> - [Moved under Issues](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70748) in 14.4 With requirements, you can set criteria to check your products against. They can be based on users, stakeholders, system, software, or anything else you find important to capture. @@ -38,7 +42,7 @@ Users with Reporter or higher [permissions](../../permissions.md) can create req To create a requirement: -1. In a project, go to **Requirements**. +1. In a project, go to **Issues > Requirements**. 1. Select **New requirement**. 1. Enter a title and description and select **Create requirement**. @@ -107,7 +111,7 @@ You can search for a requirement from the requirements list page based on the fo To search for a requirement: -1. In a project, go to **Requirements > List**. +1. In a project, go to **Issues > Requirements > List**. 1. Select the **Search or filter results** field. A dropdown menu appears. 1. Select the requirement author or status from the dropdown or enter plain text to search by requirement title. 1. Press <kbd>Enter</kbd> on your keyboard to filter the list. @@ -222,7 +226,7 @@ Before you import your file: To import requirements: -1. In a project, go to **Requirements**. +1. In a project, go to **Issues > Requirements**. - If the project already has existing requirements, select the import icon (**{import}**) in the top right. - For a project without any requirements, select **Import CSV** in the middle of the page. @@ -281,7 +285,7 @@ Users with Reporter or higher [permissions](../../permissions.md) can export req To export requirements: -1. In a project, go to **Requirements**. +1. In a project, go to **Issues > Requirements**. 1. In the top right, select the **Export as CSV** icon (**{export}**). A confirmation modal appears. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 69215e03f28..6c3bf731cf8 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -141,6 +141,11 @@ The following items are **not** exported: - Merge Request Approvers - Repository size limits +These content rules also apply to creating projects from templates on the +[group](../../group/custom_project_templates.md) +or [instance](../../admin_area/custom_project_templates.md) +levels, because the same export and import mechanisms are used. + NOTE: For more details on the specific data persisted in a project export, see the [`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) file. @@ -221,6 +226,15 @@ GitLab.com may have [different settings](../../gitlab_com/index.md#importexport) ## Troubleshooting +### Project fails to import due to mismatch + +If the [shared runners enablement](../../../ci/runners/runners_scope.md#enable-shared-runners) +does not match between the exported project, and the project import, the project fails to import. +Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either: + +- Ensure shared runners are enabled in both the source and destination projects. +- Disable shared runners on the parent group when you import the project. + ### Import workaround for large repositories [Maximum import size limitations](#import-the-project) @@ -258,7 +272,7 @@ reduce the repository size for another import attempt. git gc --prune=now --aggressive # Prepare recreating an importable file - git bundle create ../project.bundle smaller-tmp-main + git bundle create ../project.bundle <default-branch-name> cd .. mv project/ ../"$EXPORT"-project cd .. @@ -276,3 +290,29 @@ reduce the repository size for another import attempt. its [default branch](../repository/branches/default.md), and delete the temporary, `smaller-tmp-main` branch, and the local, temporary data. + +### Manually execute export steps + +Exports sometimes fail without giving enough information to troubleshoot. In these cases, it can be +helpful to [execute the export process manually within rails](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/project-export.md#export-a-project-via-rails-console). +Execute each line individually, rather than pasting the entire block at once, so you can see any +errors each command returns. + +```shell +u = User.find_by_username('someuser') +p = Project.find_by_full_path('some/project') +e = Projects::ImportExport::ExportService.new(p,u) + +e.send(:version_saver).send(:save) +e.send(:avatar_saver).send(:save) +e.send(:project_tree_saver).send(:save) +e.send(:uploads_saver).send(:save) +e.send(:wiki_repo_saver).send(:save) +e.send(:lfs_saver).send(:save) +e.send(:snippets_repo_saver).send(:save) +e.send(:design_repo_saver).send(:save) + +s = Gitlab::ImportExport::Saver.new(exportable: p, shared:p.import_export_shared) +s.send(:compress_and_save) +s.send(:save_upload) +``` diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 8b159a75451..30def6ae80f 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Manage +group: Workspace info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, index, howto --- @@ -39,17 +39,19 @@ You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#li > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/287779) in GitLab 13.12. -You can create a framework label to identify that your project has certain compliance requirements -or needs additional oversight. +You can create a compliance framework label to identify that your project has certain compliance +requirements or needs additional oversight. The label can optionally apply +[compliance pipeline configuration](#compliance-pipeline-configuration). Group owners can create, edit, and delete compliance frameworks: -1. Go to the group's **Settings** > **General**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings** > **General**. 1. Expand the **Compliance frameworks** section. -Compliance frameworks created can then be assigned to any number of projects using: +Compliance frameworks created can then be assigned to projects within the group using: -- The project settings page inside the group or subgroups. +- The GitLab UI, using the project settings page. - In [GitLab 14.2](https://gitlab.com/gitlab-org/gitlab/-/issues/333249) and later, using the [GraphQL API](../../../api/graphql/reference/index.md#mutationprojectsetcomplianceframework). @@ -64,24 +66,32 @@ read-only view to discourage this behavior. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/300324) in GitLab 13.11. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/331231) in GitLab 14.2. -Group owners can use the compliance pipeline configuration to define compliance requirements -such as scans or tests, and enforce them in individual projects. +Group owners can use compliance pipeline configuration to add additional pipeline configuration to +projects to define compliance requirements such as scans or tests. -The [custom compliance framework](#compliance-frameworks) feature allows group owners to specify the location -of a compliance pipeline configuration stored and managed in a dedicated project, distinct from a developer's project. +[Compliance frameworks](#compliance-frameworks) allow group owners to specify the location of +compliance pipeline configuration stored and managed in dedicated projects, separate from regular +projects. -When you set up the compliance pipeline configuration field, use the -`file@group/project` format. For example, you can configure -`.compliance-gitlab-ci.yml@compliance-group/compliance-project`. -This field is inherited by projects where the compliance framework label is applied. The result -forces the project to run the compliance configurations. +When you set up the compliance framework, use the **Compliance pipeline configuration** box to link +the compliance framework to specific CI/CD configuration. Use the +`path/file.y[a]ml@group-name/project-name` format. For example: -When a project with a custom label executes a pipeline, it begins by evaluating the compliance pipeline configuration. -The custom pipeline configuration can then execute any included individual project configuration. +- `.compliance-ci.yml@gitlab-org/gitlab`. +- `.compliance-ci.yaml@gitlab-org/gitlab`. -The user running the pipeline in the project should at least have Reporter access to the compliance project. +This configuration is inherited by projects where the compliance framework label is applied. The +result forces projects with the label to run the compliance CI/CD configuration in addition to +the project's own CI/CD configuration. When a project with a compliance framework label executes a +pipeline, it evaluates configuration in the following order: -Example `.compliance-gitlab-ci.yml` +1. Compliance pipeline configuration. +1. Project-specific pipeline configuration. + +The user running the pipeline in the project must at least have the Reporter role on the compliance +project. + +Example `.compliance-gitlab-ci.yml`: ```yaml # Allows compliance team to control the ordering and interweaving of stages/jobs. @@ -94,10 +104,10 @@ stages: - deploy - post-compliance -variables: # Can be overridden by setting a job-specific variable in project's local .gitlab-ci.yml +variables: # Can be overridden by setting a job-specific variable in project's local .gitlab-ci.yml FOO: sast -sast: # None of these attributes can be overridden by a project's local .gitlab-ci.yml +sast: # None of these attributes can be overridden by a project's local .gitlab-ci.yml variables: FOO: sast image: ruby:2.6 @@ -144,10 +154,10 @@ audit trail: after_script: - "# No after scripts." -include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) +include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) project: '$CI_PROJECT_PATH' file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch. + ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch. ``` ##### Ensure compliance jobs are always run @@ -182,6 +192,20 @@ cannot change them: This ensures that your job uses the settings you intend and that they are not overridden by project-level pipelines. +##### Avoid parent and child pipelines + +Compliance pipelines start on the run of _every_ pipeline in a relevant project. This means that if a pipeline in the relevant project +triggers a child pipeline, the compliance pipeline runs first. This can trigger the parent pipeline, instead of the child pipeline. + +Therefore, in projects with compliance frameworks, we recommend replacing +[parent-child pipelines](../../../ci/pipelines/parent_child_pipelines.md) with the following: + +- Direct [`include`](../../../ci/yaml/index.md#include) statements that provide the parent pipeline with child pipeline configuration. +- Child pipelines placed in another project that are run using the [trigger API](../../../ci/triggers/) rather than the parent-child + pipeline feature. + +This alternative ensures the compliance pipeline does not re-start the parent pipeline. + ### Sharing and permissions For your repository, you can set up features such as public access, repository features, diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 010a63b7957..71cf0c03549 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -26,7 +26,7 @@ and from merge requests: 1. Select **Edit in Web IDE** to open the editor. - *When viewing a merge request* - 1. Go to your merge request, and select the **Overview** tab. - 1. Scroll to the widgets area, after the merge request description. + 1. Scroll to the widgets section, after the merge request description. 1. Select **Edit in Web IDE** if it is visible. 1. If **Edit in Web IDE** is not visible: 1. Select the **(angle-down)** next to **Open in Gitpod**. @@ -231,7 +231,7 @@ left. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19318) in [GitLab Free](https://about.gitlab.com/pricing/) 11.0. To switch between your authored and assigned merge requests, click the -dropdown in the top of the sidebar to open a list of merge requests. You need to commit or discard all your changes before switching to a different merge +dropdown in the top of the sidebar to open a list of merge requests. You must commit or discard all your changes before switching to a different merge request. ## Switching branches @@ -240,7 +240,7 @@ request. To switch between branches of the current project repository, click the dropdown in the top of the sidebar to open a list of branches. -You need to commit or discard all your changes before switching to a +You must commit or discard all your changes before switching to a different branch. ## Markdown editing @@ -324,7 +324,7 @@ An example `package.json`: WARNING: Interactive Web Terminals for the Web IDE is currently in **Beta**. GitLab.com shared runners [do not yet support Interactive Web Terminals](https://gitlab.com/gitlab-org/gitlab/-/issues/24674), -so you would need to use your own private runner to make use of this feature. +so you must use your own private runner to make use of this feature. [Interactive Web Terminals](../../../ci/interactive_web_terminal/index.md) give the project [Maintainers](../../permissions.md#project-members-permissions) @@ -333,14 +333,14 @@ GitLab, including through the Web IDE. ### Runner configuration -Some things need to be configured in the runner for the interactive web terminal +Some things must be configured in the runner for the interactive web terminal to work: - The runner needs to have [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). This section requires at least a `session_timeout` value (which defaults to 1800 seconds) and a `listen_address` value. If `advertise_address` is not defined, `listen_address` is used. -- If you are using a reverse proxy with your GitLab instance, web terminals need to be +- If you are using a reverse proxy with your GitLab instance, web terminals must be [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **(ULTIMATE SELF)** If you have the terminal open and the job has finished with its tasks, the @@ -355,7 +355,7 @@ The [File Sync](#file-syncing-to-web-terminal) feature is supported on Kubernete ### Web IDE configuration file -In order to enable the Web IDE terminals you need to create the file +To enable the Web IDE terminals you must create the file `.gitlab/.gitlab-webide.yml` inside the repository's root. This file is fairly similar to the [CI configuration file](../../../ci/yaml/index.md) syntax but with some restrictions: @@ -456,7 +456,7 @@ terminal: ``` - The `webide-file-sync` executable must start **after** the project - directory is available. This is why we need to add `sleep 5` to the `command`. + directory is available. This is why we must add `sleep 5` to the `command`. See [this issue](https://gitlab.com/gitlab-org/webide-file-sync/-/issues/7) for more information. - `$CI_PROJECT_DIR` is a diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index d0a1f485fa8..e2a8167b14c 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -13,8 +13,14 @@ in each GitLab project. Every wiki is a separate Git repository, so you can crea wiki pages in the web interface, or [locally using Git](#create-or-edit-wiki-pages-locally). To access the wiki for a project or group, go to the page for your project or group -and, in the left sidebar, select **Wiki**. If **Wiki** is not listed in the -left sidebar, a project administrator has [disabled it](#enable-or-disable-a-project-wiki). +and either: + +- In the left sidebar, select **Wiki**. +- On any page in the project, use the <kbd>g</kbd> + <kbd>w</kbd> + [wiki keyboard shortcut](../../shortcuts.md). + +If **Wiki** is not listed in the left sidebar of your project, a project administrator +has [disabled it](#enable-or-disable-a-project-wiki). GitLab wikis support Markdown, RDoc, AsciiDoc, and Org for content. Wiki pages written in Markdown support all [Markdown features](../../markdown.md), @@ -130,8 +136,9 @@ may not be able to check out the wiki locally afterward. You need at least the [Developer role](../../permissions.md) to edit a wiki page: 1. Go to your project or group and select **Wiki**. -1. Go to the page you want to edit. -1. Select the edit icon (**{pencil}**). +1. Go to the page you want to edit, and either: + - Use the <kbd>e</kbd> wiki [keyboard shortcut](../../shortcuts.md#wiki-pages). + - Select the edit icon (**{pencil}**). 1. Edit the content. 1. Select **Save changes**. @@ -142,7 +149,7 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). ## Delete a wiki page -You need at least the [Maintainer role](../../permissions.md) to delete a wiki page: +You need at least the [Developer role](../../permissions.md) to delete a wiki page: 1. Go to your project or group and select **Wiki**. 1. Go to the page you want to delete. @@ -355,3 +362,4 @@ For the status of the ongoing development for CommonMark and GitLab Flavored Mar - [Project wikis API](../../../api/wikis.md) - [Group repository storage moves API](../../../api/group_repository_storage_moves.md) - [Group wikis API](../../../api/group_wikis.md) +- [Wiki keyboard shortcuts](../../shortcuts.md#wiki-pages) diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 32bb202767a..11b570f19e5 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -1,6 +1,6 @@ --- -stage: Create -group: Source Code +stage: Manage +group: Workspace info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" --- @@ -165,7 +165,7 @@ To push a new project: As project creation permissions can have many factors, contact your GitLab administrator if you're unsure. -1. If you want to push using SSH, ensure you have [created a SSH key](../../ssh/README.md) and +1. If you want to push using SSH, ensure you have [created a SSH key](../../ssh/index.md) and [added it to your GitLab account](../../ssh/index.md#add-an-ssh-key-to-your-gitlab-account). 1. Push with one of the following methods. Replace `gitlab.example.com` with the domain name of the machine that hosts your Git repository, `namespace` with the name of @@ -187,6 +187,12 @@ You can view your new project at `https://gitlab.example.com/namespace/myproject Your project's visibility is set to **Private** by default, but you can change it in your [project's settings](../../public_access/public_access.md#change-project-visibility)). +This feature does not work for project paths that have previously been in use and +[renamed](settings/index.md#renaming-a-repository). A redirect exists over the previous project path +that causes push attempts to redirect requests to the renamed project location, instead of creating +a new project. To create a new project, use the [Web UI](#create-a-project) or the +[Projects API](../../api/projects.md#create-project). + ## Fork a project A fork is a copy of an original repository that you put in another namespace diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index bf9abcca640..203b1c9e93a 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Manage +group: Workspace info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index 37e89dd54db..f46c5428248 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -39,7 +39,7 @@ These shortcuts are available in most areas of GitLab: | <kbd>Shift</kbd> + <kbd>s</kbd> | Go to your Snippets page. | | <kbd>s</kbd> / <kbd>/</kbd> | Put cursor in the search bar. | | <kbd>Shift</kbd> + <kbd>i</kbd> | Go to your Issues page. | -| <kbd>Shift</kbd> + <kbd>m</kbd> | Go to your Merge requests page.| +| <kbd>Shift</kbd> + <kbd>m</kbd> | Go to your [Merge requests](project/merge_requests/index.md) page. | | <kbd>Shift</kbd> + <kbd>t</kbd> | Go to your To-Do List page. | | <kbd>p</kbd> + <kbd>b</kbd> | Show or hide the Performance Bar. | | <kbd>g</kbd> + <kbd>x</kbd> | Toggle between [GitLab](https://gitlab.com/) and [GitLab Next](https://next.gitlab.com/) (GitLab SaaS only). | @@ -77,17 +77,17 @@ relatively quickly to work, and they take you to another page in the project. | <kbd>g</kbd> + <kbd>i</kbd> | Go to the project issues list (**Issues > List**). | | <kbd>i</kbd> | Go to the New Issue page (**Issues**, select **New Issue** ). | | <kbd>g</kbd> + <kbd>b</kbd> | Go to the project issue boards list (**Issues > Boards**). | -| <kbd>g</kbd> + <kbd>m</kbd> | Go to the project merge requests list (**Merge Requests**). | +| <kbd>g</kbd> + <kbd>m</kbd> | Go to the project [merge requests](project/merge_requests/index.md) list (**Merge Requests**). | | <kbd>g</kbd> + <kbd>j</kbd> | Go to the CI/CD jobs list (**CI/CD > Jobs**). | | <kbd>g</kbd> + <kbd>l</kbd> | Go to the project metrics (**Monitor > Metrics**). | | <kbd>g</kbd> + <kbd>e</kbd> | Go to the project environments (**Deployments > Environments**). | | <kbd>g</kbd> + <kbd>k</kbd> | Go to the project Kubernetes cluster integration page (**Infrastructure > Kubernetes clusters**). Note that you must have at least [`maintainer` permissions](permissions.md) to access this page. | | <kbd>g</kbd> + <kbd>s</kbd> | Go to the project snippets list (**Snippets**). | -| <kbd>g</kbd> + <kbd>w</kbd> | Go to the project wiki (**Wiki**), if enabled. | +| <kbd>g</kbd> + <kbd>w</kbd> | Go to the [project wiki](project/wiki/index.md) (**Wiki**), if enabled. | ### Issues and merge requests -These shortcuts are available when viewing issues and merge requests: +These shortcuts are available when viewing issues and [merge requests](project/merge_requests/index.md): | Keyboard shortcut | Description | |------------------------------|-------------| diff --git a/doc/user/todos.md b/doc/user/todos.md index f0601db0300..9a985664ed1 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -7,8 +7,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # To-Do List **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/2817) in GitLab 8.5. - Your *To-Do List* is a chronological list of items waiting for your input. The items are known as *to-do items*. @@ -67,8 +65,6 @@ You can manually add an item to your To-Do List. ## Create a to-do item by directly addressing someone -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7926) in GitLab 9.0. - You can create a to-do item by directly addressing someone at the start of a line. For example, in the following comment: diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 07a5eda8cfb..5a48353c9d4 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -5,7 +5,7 @@ group: Utilization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Storage usage quota **(FREE SAAS)** +# Storage usage quota **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/13294) in GitLab 12.0. > - Moved to GitLab Free. @@ -18,29 +18,27 @@ you must purchase additional storage. For more details, see [Excess storage usag ## View storage usage -To help manage storage, a namespace's owner can view: +You can view storage usage for your project or [namespace](../user/group/#namespaces). -- Total storage used in the namespace -- Total storage used per project +1. Go to your project or namespace: + - For a project, on the top bar, select **Menu > Projects** and find your project. + - For a namespace, enter the URL in your browser's toolbar. +1. From the left sidebar, select **Settings > Usage Quotas**. +1. Select the **Storage** tab. -To view storage usage, from the namespace's page go to **Settings > Usage Quotas** and select the -**Storage** tab. The Usage Quotas statistics are updated every 90 minutes. +The statistics are displayed. Select any title to view details. The information on this page +is updated every 90 minutes. -If your namespace shows `N/A` as the total storage usage, push a commit to any project in that -namespace to trigger a recalculation. - -A stacked bar graph shows the proportional storage used for the namespace, including a total per -storage item. Click on each project's title to see a breakdown per storage item. +If your namespace shows `N/A`, push a commit to any project in the +namespace to recalculate the storage. ## Storage usage statistics -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247831) in GitLab 13.7. -> - It's [deployed behind a feature flag](../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab SaaS. -> - It's recommended for production use. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68898) project-level graph in GitLab 14.4 [with a flag](../administration/feature_flags.md) named `project_storage_ui`. Disabled by default. +> - Enabled on GitLab.com in GitLab 14.4. -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `project_storage_ui`. On GitLab.com, this feature is available. The following storage usage statistics are available to an owner: diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index 2ce30c645d5..d75d91f087d 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -6,24 +6,33 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Workspace -Workspace will be above the [top-level namespaces](../group/index.md#namespaces) for you to manage everything you can do as a GitLab administrator, including: +Workspace will be above the [top-level namespaces](../group/index.md#namespaces) for you to manage +everything you do as a GitLab administrator, including: - Defining and applying settings to all of your groups, subgroups, and projects. - Aggregating data from all your groups, subgroups, and projects. -The functionality in the [Admin Area](../admin_area/index.md) of self-managed installations will be split up and go to: +Our goal is to reach feature parity between SaaS and self-managed installations, with all +[Admin Area settings](/ee/user/admin_area/settings/) moving to either: -1. Groups (available in the Workspace, Top-level group namespaces, and Sub-groups) -1. Hardware Controls (for functionality that does not apply to groups) - -Our goal is to reach feature parity between SaaS and Self-Managed installations, with all [Admin Area settings](/ee/user/admin_area/settings/) moving to either: - -- Workspace (contains features relevant to both GitLab-managed and self-managed installations) with a dedicated Settings menu available within the left navigation bar. -- Hardware controls (only contains features relative to Self-Managed installations, with one per installation). +- Groups. Available in the Workspace, top-level group namespaces, and sub-groups. +- Hardware Controls. For functionality that does not apply to groups, Hardware Controls are only + applicable to self-managed installations. There is one Hardware Controls section per installation. NOTE: Workspace is currently in development. +## Demo: New hierarchy concept for groups and projects for epics + +The following demo introduces the new hierarchy concept for groups and projects for epics. + +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/embed/fE74lsG_8yM">Consolidating groups and projects update (2021-08-23)</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/fE74lsG_8yM" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + ## Concept previews The following provide a preview to the Workspace concept. |
